Deep Reinforcement Learning for Radio Resource Allocation and Management in Next Generation Heterogeneous Wireless Networks: A SurveyThanks: The authors are with the Division of Information and Computing Technology, College of Science and Engineering, Hamad Bin Khalifa University, Qatar (e-mail: aalwarafy@hbku.edu.qa; moabdallah@hbku.edu.qa; bciftler@hbku.edu.qa; aalfuqaha@hbku.edu.qa; mhamdi@hbku.edu.qa). This work has been submitted to the IEEE for possible publication. Copyright may be transferred without notice, after which this version may no longer be accessible.

The explosive growth in the number and types of modern smart devices, such as smartphones/tablets and wearable devices, has led to the emergence of disruptive wireless communications and networking technologies, such as 5G NR cellular networks, IoT networks, personal (or wireless body area networks), device-to-device (D2D) communications, holographic imaging and haptic communications, and vehicular networks [3, 7, 8, 4, 9, 10, 11]. Such networks are envisaged to meet the stringent requirements of the emerging applications and services via supporting high data rates, coverage, and connectivity with significant enhancements in reliability, reduction in latency, and mitigation of energy consumption.

However, achieving this goal in such large-scale, versatile, and complex wireless networks is quite challenging, as it requires a judicious allocation and management of the networks’ limited radio resources [12, 13]. In particular, efficient and more advanced RRAM solutions must be developed to balance the tradeoff between enhancing network performance while guaranteeing an efficient utilization of radio resources. Furthermore, efficient RRAM solutions must also strike and intelligent tradeoff between optimizing network radio resources and satisfying users’ QoE. For example, RRAM techniques must jointly enhance network spectral efficiency (SE), energy efficiency (EE), and throughput while mitigating interference, reducing latency, and enhancing rate for user devices.

Efficient and advanced RRAM schemes can considerably enhance the system’s SE compared to the traditional techniques by relying on the advanced channel and/or source coding methods. RRAM is essential in broadcast wireless networks covering wide geographical areas as well as in modern cellular communication networks comprised of several adjacent and dense access points (APs) that typically share and reuse the same radio frequencies.

From a cost point of view, the deployment of wireless APs and sites, e.g., base stations (BSs), including the real estate costs, planning, maintenance, and energy, is the most critical aspect alongside with the frequency license fees. Hence, the goal of RRAM is maximizing the network’s SE in terms of bits/sec/Hz/area unit or Erlang/MHz/site, under some constraints related to user fairness. For instance, the service grade must meet a minimum acceptable level of QoS, including the coverage of certain geographical areas while mitigating network outages caused by interference, noise, large-scale fading (due to path losses and shadowing), and small-scale fading (due to multi-path). The service grade also depends on blocking caused by admission control, scheduling errors, or inability to meet certain QoS demands of edge devices (EDs).

Future wireless communication networks are complex due to their large-scale, versatile, and heterogeneous nature. To optimally allocate and manage radio resources in such networks, we typically formulate RRAM as complex optimization problems. The objective of such problems is to achieve a particular goal, such as maximizing network sum-rate, SE, and EE, given the available radio resources and QoS requirements of user devices. Unfortunately, the massive heterogeneity nature of modern networks poses tremendous challenges during the process of formulating optimization problems as well as applying conventional techniques to solve them, such as optimization, heuristic, and game theory algorithms.

The large-scale nature of next generation networks makes it quite difficult to formulate RRAM optimization problems that are often intractable non-convex. Also, conventional techniques used to solve the RRAM problems require complete or quasi-complete knowledge of the wireless environment, including accurate channel models and real-time CSI. However, obtaining such information in a real-time fashion in these large-scale networks is quite difficult or even impossible. Furthermore, conventional techniques are often computationally-expensive and incur considerable timing overhead. This renders them inefficient for most emerging time-sensitive applications, such as autonomous vehicles and robotics.

Moreover, game theory-based techniques are unsuitable for future heterogeneous networks (HetNets) as such techniques are devised for homogeneous players. Also, the explosive number of network APs and user devices will create extra burdens on game theory-based techniques. In particular, network players, such as BSs, APs, and user devices, need to exchange a tremendous amount of data and signaling. This will induce unmanageable overhead that largely increases delay, computation, and energy/memory consumption of network elements.

Emerging artificial intelligence (AI) techniques, such as deep reinforcement learning (DRL), have shown efficient performance in addressing various issues in modern wireless communication networks, including solving complex RRAM optimization problems [14, 15, 16, 17, 18, 19, 20, 21, 22]. In the context of RRAM, DRL methods are mainly used as an alternative to overcome the shortcomings and limitations of the conventional RRAM techniques discussed above. In particular, DRL techniques can solve complex network RRAM optimization problems and take judicious control decisions with only limited information about the network statistics. They achieve this by enabling network entities, such as BSs, RAN’s APs, edge servers (ESs), gateways nodes, and user devices, to make intelligent and autonomous control decisions, such as RRAM, user association, and RAN’s selection, in order to achieve various network goals such as sum-rate maximization, reliability enhancement, delay reduction, and SE/EE maximization. In addition, DRL techniques are model-free that enable different network entities to learn optimal policies about the network, such as RRAM and user association, based on their continuous interactions with the wireless environment, without knowing the exact channel models or other network statistics a-priori. These appealing features make DRL methods one of the main key enabling technologies to address the RRAM issue in modern wireless communication networks [3, 2].

Static approaches are designed based on a priori statistical information and cannot adapt to wireless network parameters, such as traffic load, users’ mobility pattern, channel conditions/quality, network spectrum occupancy, and users’ QoS demands. These techniques are simple; however, they suffer from several degradations, such as severe underutilization of radio resources, increased network outage, reduced network throughput, and poor network QoS.

Static RRAM techniques are employed in several traditional wireless communication networks, such as cellular communication networks and WLANs. Examples of static RRAM techniques include circuit-mode communication using frequency division multiple access (FDMA) and time division multiple access (TDMA) schemes and fixed radio resource allocation, such as fixed power and channel allocation.

On the contrary, dynamic or adaptive RRAM approaches are more efficient as they can dynamically adjust the network radio resources to accurately track variations in propagation conditions and user QoS requirements.

Dynamic RRAM schemes are widely utilized in designing modern wireless systems. They have shown efficient results in reducing the expensive manual network planning and achieving tighter radio resource utilization, which will lead eventually to enhanced network SE. Some RRAMs schemes are centralized, where several BSs, ESs, wireless APs, and network gateways are controlled by a central Radio Network Controller (RNC). Others are distributed, either autonomous algorithms implemented in smart user devices, BSs, ESs, or APs, or coordinated by exchanging information among these network entities. Examples of dynamic RRAM schemes include power control algorithms, dynamic spectrum/channel allocation algorithms, multi-access control schemes, link adaptation algorithms, precoding schemes, traffic adaption algorithms, channel-dependent scheduling schemes, and cognitive radio approaches.

In dynamic RRAM, we typically formulate the wireless radio resource allocation problem as a complex optimization problem. The main objective of such a problem is maximizing/minimizing some utility/cost functions, e.g., network sum-rate, EE, and SE, while constraining the available network’s radio resources such as the available power and bandwidth. The state-of-the-art approaches to solve these wireless RRAM optimization problems are heuristic-based, optimization-based, and game theory-based approaches. Such approaches employ advanced algorithms to solve the RRAM problem either optimally or sub-optimally.

In RL, $Q$-learning is one of the most widely used algorithms to address MDPs. It obtains the optimal values of the $Q$-function iteratively using the following Bellman equation updating rule:

where $\alpha_{t}$ is the learning rate that defines how much the new information contributes to the existing $Q$-value. The main idea of this Bellman rule relies on finding the Temporal Difference (TD) between the current $Q$-value ($Q(s_{t},a_{t})$) and the predicted $Q$-value ($r_{t}(s_{t},a_{t})+\gamma\underset{a_{t+1}}{\text{max}}Q(s_{t+1},a_{t+1})-Q(s_{t},a_{t})$). The $Q$-learning algorithm uses this Bellman equation to construct a table of all possible $Q$ values for each stat-action pair. The algorithm terminates when we reach a certain number of iterations or when all Q-values have converged. In such a case, the optimal policy will determine the optimal action to take at each state such that $Q^{\pi^{*}}(s_{t},a_{t})$ is maximized for all states in the state space, i.e., $\pi^{*}=\underset{a_{t+1}}{\text{argmax}}~Q^{\pi^{*}}(s_{t},a_{t})$. This optimal policy says that at any state we take the action that will eventually obtain the highest cumulative reward.

However, the $Q$-learning algorithm has many limitations when applied for RRAM in modern wireless networks. First, it is applicable only to problems with low dimensionality of both state and action spaces, making it unscalable. Second, it is applicable only on RRAM with discrete state space and action space, such as channel access and RANs assignment. If, however, they are applied to problems with continuous action space nature, such as power allocation, the action space must be digitized. This renders them inaccurate due to the quantization error.

Since the $Q$-learning algorithm relies on building a table for the $Q$ values, it will fail to obtain the optimal policy when the state space and action space become prohibitively large. This case is commonly encountered in the RRAM problems of modern wireless systems. To overcome this issue, the DQN algorithm has been developed , which inherits the advantages of $Q$-learning and DL techniques. The main idea is to replace the table in the $Q$-learning algorithms with a DNN that tries to approximate the $Q$ values. Hence, the DNN is also called the function approximator and denoted as $Q(s_{t},a_{t}|\theta)$, where $\theta$ represents the training parameters (i.e., weights) of the DNN. Fig. 9 shows the DQN architecture. The replay memory is denoted by $\mathcal{D}$, and it is mainly used to break the correlation between the training samples or transitions, i.e., ($s_{t},a_{t},r_{t},s_{t+1}$), by making them independently and identically distributed i.i.d. During the learning process of the policy, we store the training transitions that are generated during the interaction with the wireless environment in $\mathcal{D}$. The DQN’s agent will then randomly select minibatch samples of transitions from $\mathcal{D}$ to train its DNN. To enhance the stability of the DQN model, the target $Q$ network is used, whose weights will be periodically updated to track those of the main $Q$ network.

Since the DQN algorithm is mainly used to learn the optimal policy, i.e., $\pi^{*}=\underset{a}{\text{argmax}}~Q^{\pi^{*}}(s_{t},a_{t})$, the optimal $Q$-function is derived from the following iterative Bellman equation:

and the DQN algorithm is then optimized by iteratively updating the weights $\theta$ of its DNN to minimize the following Bellman loss function;

where $\theta^{{}^{\prime}}$ is the weights of the target $Q$ network.

The DQN algorithm is applicable to a wide variety of RRAM problems, specifically for problems characterized by their discrete action space. As we will elaborate in-depth in Section IV, the DQN technique can be used efficiently for channel allocation, access control, spectrum access, user association, and RANs assignment. The DQN algorithm can also be used for RRAM problems with continuous action space, such as power control, by discretizing the action space. However, such a methodology makes DQN vulnerable to serious quantization error that may considerably deteriorate its accuracy. There are also other limitations in the basic DQN, and various $Q$-learning algorithms have been proposed to overcome them, as we discuss in the following sections.

The Double DQN technique has been proposed in [69] to enhance the basic DQN algorithm. The DQN algorithm tends to overestimate the $Q$ values, which can degrade the training process and lead to suboptimal policies. The overestimation results from the positive bias caused by the max operation employed in the Bellman equation. Specifically, the root cause is that the same training transitions are utilized in selecting and evaluating an action. As a solution to this problem, the authors in [69] propose to use two $Q$ value functions, one for selecting the best action and the other to evaluate the best action. The action selection is still based on the online weights parameters $\theta$, while the second weights parameters $\theta^{{}^{\prime}}$ are used to evaluate the value of this policy. So, as in the conventional $Q$ learning, the value of the policy is still estimated based on the current $Q$ values. The weights parameters $\theta^{{}^{\prime}}$ are updated via switching between $\theta$ and $\theta^{{}^{\prime}}$.

Hence, the target $Q$ values are derived from the following modified Bellman equation [69]:

and the Double DQN algorithm uses the following modified Bellman loss function to update its weights;

The Double DQN algorithm is also widely used in RRAM problems, as we will discuss in the next section. Although this algorithm has advantages over the basic DQN algorithm, they both share the same shortcomings.

This algorithm is another enhancement to the basic DQN proposed in [70]. Recall that the goal of the network is to estimate the $Q$ values, i.e., $Q(s_{t},a_{t})$. This function can be divided into two terms; the state-value function $V(s)$, which tells the importance of being in a particular state, and the action-value function (or the advantage function) $A(s,a)$, which tells the importance of selecting a particular action among all available actions. Hence, the $Q$ value function can be written as $Q(s,a)=V(s)+A(s,a)$. The authors in [70] utilized this concept and suggested having two main independent paths of fully-connected layers instead of having only a single path as the case in the basic DQN. One path will estimate $V(s)$, and the other path will estimate $A(s,a)$. The two paths will eventually be combined to produce one single output, which is $Q(s,a)$ as [70]:

where $\mathcal{A}$ is the number of actions in the action space, and $\beta$ and $\alpha$ are the weights of the fully-connected layers of the two paths $V(s_{t};\theta,\beta)$ and $A(s_{t},a_{t};\theta,\alpha)$, respectively. Here, the loss function is obtained similar to the DQN and Double DQN algorithms.

The main idea of this algorithm is to increase the probabilities of good actions and reduce the probabilities of bad actions. In general, the REINFORCE algorithm differs from the $Q$ learning methods in three core aspects. First, REINFORCE algorithm does not need a replay buffer $\mathcal{D}$ during training as it belongs to the on-policy family, which requires only fresh training transitions. Although this enhances its convergence speed, it needs much more interaction with the environment. Second, the REINFORCE algorithm implicitly performs the exploration process, as it depends on the probabilities returned by the network, which incorporate uniform random agent behavior. Third, no target network is required in the REINFORCE algorithm as the $Q$ values are obtained from the experiences in the environment.

The weights $\theta$ of the network in the REINFORCE algorithm are updated to minimize the following loss function:

The disadvantage of the REINFORCE algorithm is that it suffers from high variance, meaning that any small shift in the return leads to a different policy. This limitation motivated the actor-critic algorithms.

The actor-critic methods are mainly developed to enhance the convergence speed and stability (i.e., reducing the variance) of the policy-gradient method. Like the policy-based methods, it utilizes the accumulated discounted reward $J$ to obtain the gradient of the policy $\triangledown J$, which provides the direction that enhances the policy. This algorithm learns a critic to reduce the variance of gradient estimates since it utilizes various samples, whereas the REINFORCE algorithm utilizes only a single sample trajectory.

To select the best action in any state, the total discount reward of the action is used, i.e., $Q(s,a)$. The total reward can be decomposed into state-value function $V(s)$ and advantage function $A(s,a)$, i.e., as $Q(s,a)=V(s)+A(s,a)$. So, another DNN is utilized to estimate $V(s)$, which is trained based on the Bellman equation. The estimated $V(s)$ is then leveraged to obtain the policy gradient and update the policy network such that the probabilities of actions with good advantage values are increased. Hence, the actor is the policy network $\pi(a|s)$ that takes actions by returning the probability distribution of actions, while the critic network evaluates the quality of the taken actions, $V(s)$. This algorithm is also called the advantage actor-critic method (A2C).

In the A2C algorithm, the weights of the actor network $\theta_{\pi}$ and critic network $\theta_{\text{v}}$ are updated using the accumulated policy gradients $\partial\theta_{\pi}$ and value gradients $\partial\theta_{\text{v}}$, respectively, according to the following formulas [72]:

where $R\leftarrow r_{t}(s_{t},a_{t})+R$. The weights are updated to move in the direction of the policy gradients and in the opposite direction of the value gradients.

The asynchronous advantage actor-critic (A3C) algorithm is an extension of the basic A2C [72]. This algorithm is used to solve the high variance issue in gradients that results in non-optimal policies. A3C algorithm conducts a parallel implementation of the actor-critic algorithm, where the actor and critic share the network layers. A global NN is trained to output action probabilities and an estimate of the advantage function $A(s_{t},a_{t}|\theta_{\pi},\theta_{\text{v}})$ given by $\sum_{i=0}^{k-1}\gamma^{i}r_{t+1}+\gamma^{k}V(s_{t+k}|\theta_{\text{v}})-V(s_{t}|\theta_{\text{v}})$, where $k$ depends on the state and upper-bounded by the maximum number of time steps. The update rule in A3C is given by [72]:

where $\theta_{\pi}^{{}^{\prime}}$ is thread-specific weights.

Several parallel actor learners are instantiated with copies of both the environment and global NN weights. Each learner independently interacts with its environment and gathers training transitions to derive the gradients with respect to its NN weights. Learners will then propagate their gradients to the global NN to update its weights. This mechanism ensures a periodic update of the global model with diverse transitions from each learner.

DDPG is one of the most widely used DRL techniques in addressing RRAM problems for wireless networks characterized by their high dimensionality and continuous action space [73]. DDPG algorithm belongs to the actor-critic family, and it combines both $Q$-learning and policy gradients algorithms. It consists of actor and critic networks. The actor network takes the state as its input, and it outputs the exact ”deterministic” action, not probability distribution over actions as in the actor-critic algorithm. Whereas the critic is a $Q$-value network that takes both the state and action as inputs, and it outputs the $Q$-value as a single output.

The deterministic policy gradient (DPG) algorithm is proposed in [74] to overcome the limitation caused by the $max$ operator in the $Q$-learning algorithm, i.e., $\underset{a_{t+1}}{\text{max}}Q(s_{t+1},a_{t})$. It simultaneously learns both the $Q$-function and the policy. In particular, the DPG algorithm has a parameterized actor function $\mu(s|\theta^{\mu})$ with weights $\theta$, which learns the deterministic policy that gives the optimal action corresponding to $\underset{a_{t+1}}{\text{max}}Q(s_{t+1},a_{t})$. The critic $Q(s,a)$ is leaned via minimizing the Bellman loss function as in the $Q$-learning algorithm.

The learning process of the actor policy is updated using gradient ascent with respect to $\theta^{\mu}$ in order to solve the objective given by the following chain rule [74]:

The DDPG algorithm proposed in [73] is built based on the DPG algorithm, where both the policy and critic are DNNs, as shown in Fig. 10. The DDPG algorithm creates a copy of both the actor and critic networks, $Q^{{}^{\prime}}(s,a|\theta^{Q^{{}^{\prime}}})$ and $\mu^{{}^{\prime}}(s|\theta^{\mu^{{}^{\prime}}})$, respectively, to compute the target values. The weights of these two target networks, $\theta^{Q^{{}^{\prime}}}$ and $\theta^{\mu^{{}^{\prime}}}$, are then updated to slowly track the weight of the learned network to provide more stable training using $\theta^{{}^{\prime}}\leftarrow\tau\theta+(1-\tau)\theta^{{}^{\prime}}$ with $\tau\ll 1$. The critic network is updated to minimize the following Bellman loss function [73];

Note that the DDPG algorithm is off-policy, which means that we use a replay buffer $\mathcal{D}$ to store training transitions.

The exploration-exploitation issue is addressed by adding the Ornstein–Uhlenbeck (OU) process or some Gaussian noise $\mathcal{N}$ to the action selected by the policy, i.e., $\mu(s_{t}|\theta_{t}^{\mu})+\varepsilon\mathcal{N}$ [73].

In the following paragraphs, we review works that employ DRL algorithms to address the power allocation problem in cellular, cellular IoT, and wireless homogeneous networks (HomNets) depicted in Fig. 2.

In [88], the authors propose a multi-agent $Q$-learning-based RL model to address the problem of downlink power control and rate adaptation in cellular systems. The agent is a network entity that resides in the cell, such as an eNB. The action space is discrete, corresponding to allocating downlink transmit power of the cell. The state space comprises four elements; cell power, average reference signal received power, average interference, and cell reward. The reward function is continuous, defined based on the $\alpha$-fair resource allocation utility function. It is defined by the radio measurement or performance indicator associated with all users in the radio cell. System-level simulations show that their proposed method quickly learns the power control policy to provide significant energy savings and fairness across the network users.

The authors in [89] propose a single-agent DQN method to address the problem of power resource use in fuel cell powered data center power grids. In the model, the agent is the data center architecture whose state space is discrete and comprised of the following elements; 1) the number of data centers in the previous section at the current and next time slots, 2) the action at the current time slot, 3) and the reward function at the current time slot. The action space is choosing the data center, and the reward is a function of the sum of all the data centers’ power demands in each time slot. Their approach is compared to the state-of-the-art approaches and evaluated based on real-world traces, which shows that it reduces the energy gap and saves more energy at around 5%.

In [90], the authors propose single- and multi-agent actor-critic DRL methods to tackle the problem of downlink sum-rate maximization through power allocation in multi-cell, multi-user cellular networks. In their model, the agents are the base stations (BSs), whose state space is continuous and comprises network CSI and the transmit power allocation by previous BSs. The action space is continuous, which represents the power allocation, while the reward function is the cellular network sum SE. Experimental results demonstrate that their proposed DRL-based method can both achieve higher SE than conventional optimization algorithms, such as fractional programming (FP) and weighted minimum mean-squared error (WMMSE) while performing two times faster than these conventional methods.

The work in [91] proposes a distributive multi-agent DDPG-based DRL algorithm to address the problem of sum-rate maximization via continuous power control in wireless mobile networks. The agents are each transmitter (e.g., mobile devices, links, etc.) whose state is a combination of three feature groups; the local information, interfering neighbors, and interfered neighbors feature groups. Each agent’s action is choosing the transmit power level, while the reward is a function of the sum-rate maximization problem. Simulation results show that their proposed method gives better performance results than the conventional FP methods and comparable results with the WMMSE methods.

D2D communication has emerged as one of the main enabling technologies for modern wireless networks. In [92], the authors present a centralized multi-agent DQN-based DRL algorithm to address the problem of power allocation of D2D cellular communication in a time-varying environment. The agents are the D2D transmitters, whose state space is continuous, comprised of the SINR and channel gain of users. The action space is discrete, representing the transmit power of each D2D user, while the reward is a function of system throughput. Simulation results show that their proposed algorithm outperforms the traditional RL methods in terms of network capacity and user’s achieved QoS.

5G UDNs are characterized by their high vulnerability to inter-cell interference, which can be greatly reduced via judicious power management. Towards this, Saeidian et al. [93] propose a data-driven approach based on a multi-agent DQN algorithm to tackle the downlink power control in dense 5G cellular networks. The agents are the BSs, whose state space is continuous, comprised of path-gain, SINR, downlink rate, and downlink power. The action space is discrete, representing the downlink power, while the reward is a function of the network-wide harmonic-mean of throughput. Simulation results indicate that their approach can improve data rates at the cell edge while ensuring a reduced transmitted power compared to the baseline fixed power allocation approaches.

Non-orthogonal multiple access (NOMA) technology has recently emerged as an efficient tool to enhance the QoS and EE of millimeter-wave (mmWave) communication systems via enhancing the power level of received signals. In this context, the authors in [94] propose a multi-agent DQN-based DRL framework to optimize the EE in downlink full-duplex cooperative NOMA of mmWave UDNs. The agents are the relay near users, whose state space is continuous, consisting of information related to wireless environment and channel, the user’s battery capacity, energy power transfer coefficient, self-interference cancellation residue coefficient, and the buffer size of nearby relay users. The action space is to specify the required user pairing between the near relay user group and edge user group, along with the pre-processing of EE power allocation. The reward is a function of the EE of the mmWave network. Experimental results are compared with a conventional centralized iteration algorithm, which demonstrate both the superiority of their proposed algorithm in terms of the convergence speed and the efficiency to provide near-optimal results.

Due to the importance of the power allocation in multi-user cellular networks, the authors in [95] address this issue by building on their initial investigation in [96]. A multi-agent DQN-based DRL algorithm is proposed in which each BS-user link is considered as an agent. The state space is continuous, comprised of a logarithmic normalized interferer, the link’s corresponding downlink rate, and the transmitting power. The action space is discrete, corresponding to the downlink power allocation, while the reward is continuous, which is a function of the downlink data rate of the communication link. Experimental results indicate that their proposed DQN outperforms benchmark algorithms such as FP, WMMSE, random power allocation, and maximum power allocation in terms of achievable averaged sum-rate and the convergence time when considering different user densities.

In another interesting work in [97], the authors design a multi-agent DQN and DDPG-based DRL framework to address the problem of power control in HetNets. A centralized-training-distributed-execution algorithm is designed in which the APs are the agents, each of which implements a local DNN. The state space of each local DNN is continuous, representing the local state information, while the local action space is continuous, representing the transmit power. Then, multiple-actor-shared-critic method (MASC) is proposed to separately train each of these local DNN in an online fashion. The main idea is that the MASC training method is composed of multiple actor DNNs and a shared critic DNN. An actor DNN is first established in the core network for each local DNN, and the structure of each actor DNN is the same as the corresponding local DNN. Then, a shared critic DNN is established in the core network for these actor DNNs. Historical global information is provided into the critic DNN, and the output of the critic DNN will evaluate whether the output power of each actor DNN is optimal or not from a global view. The reward function is continuous, representing the data rate between each AP and its associated user. Simulation results show that their proposed algorithm outperforms the WMMSE and FP algorithms in terms of both convergence rate and computational complexity.

Managing radio resources in virtualized networks is attracting more attention lately. In this context, the authors in [98] address the problem of throughput maximization in C-RANs via power allocation in virtualized 5G networks. The authors propose a multi-agent DQN-based DRL algorithm to solve the problem in which the agents are each link between RRH and user. The action space is discrete, corresponding to the transmit power. The state space is continuous, representing the current partial CSI and the respective power set. The reward of each slice is discrete, defined as a function of the sum of its tenants’ rates. Via simulation, the authors show that their proposed scheme achieves a higher sum-rate compared to greedy search-based power allocation approaches.

In the following paragraphs, we review works that utilize DRL algorithms to address the power allocation issue in IoT and other emerging wireless networks shown in Fig. 2.

Developing efficient spectrum sharing schemes is regarded as one of the main persistent objectives and challenges in CRNs. In [99], the authors propose a non-cooperative single-agent DQN-based DRL scheme to address the problem of spectrum sharing via power control in CRNs. In their model, the agent is the SU, whose action space is discrete, corresponding to selecting the transmit power from a pre-defined power set. The state space is discrete, defined by four parts; the transmit power of PU and SU, the path loss between PU and a sensor that measures the RSS, the path loss between the SU and a sensor that measures the RSS, and some Gaussian random variable. The reward is a discrete function defined by the achieved SINR level and the minimum SINR requirements of both PU and SU. Simulation results show that their proposed algorithm is robust against random variation in state observations, and the SU interacts with PU efficiently until they reach a state in which both users successfully transmit their own data.

In other work in [13], the authors present a non-cooperative multi-agent algorithm to address the problem of power allocation in D2D communication networks based on three DQNs, namely, DQN, Double DQN, and Dueling DQN. The agents are the D2D transmitters in each D2D pair, whose state space is discrete, comprised of the level of the interference indicator function. The action space is discrete, representing the set of transmitting power levels, while the reward is a function of the system EE. Simulation results show the ability of such DQN-based models to provide energy-efficient power allocation for the underlying D2D network.

In [100], a DRL algorithm based on multi-agent DQN is proposed to tackle the problem of downlink power allocation in multi-cell cellular networks. The agents are the base stations (BSs), whose stat space is hybrid, comprised of the number of users connected to the BSs and the corresponding power allocation. The action space corresponds to selecting the adjustment value of the power on the subcarriers of the BSs, while the reward is a function of the overall network capacity. Experimental results demonstrate that their proposed algorithm is better than both the water-filling power allocation and $Q$-learning methods in terms of model stability and convergence speed.

UAV networks are attracting considerable attention recently due to their ability to provide enhanced QoS communication in harsh and vital environments. However, power management is one of the key challenges in such networks. In this context, the authors in [101] address the problem of downlink power control in ultra-dense UAV networks with the aim of maximizing the network’s EE. A multi-agent DQN-based DRL model is proposed in which the agents are the UAVs in the network. The state space is continuous, representing the remaining energy of the UAV and the interference caused by neighboring UAVs. The action space is discrete, representing the set of possible discrete transmit power values, while the reward function is the EE of the UAV network. Simulation results are compared with $Q$-learning and random algorithms, which show the superiority of their proposed scheme in terms of both the convergence speed and EE.

Other inserting work [102] proposes a multi-agent DQN-based DRL method to study the problem of transmit power control in wireless networks. The agents are the transmitters whose state space is continuous, consisting of three main feature groups; local information, interfering neighbors, and interfered neighbors. The action space is discrete corresponds to discrete power levels, while the reward is a function of the weighted sum-rate of the whole network. Experimental results demonstrate that the proposed distributed algorithm provides comparable and even better performance results to the state-of-the-art optimization-based algorithms available in the literature.

Another interesting work is proposed in [103]. The author proposes a single-agent smoothing DQN-network to control power allocation in cognitive radio-based wireless networks. The agent is the sensor, whose state space is discrete, comprising of a set of received signal strength (RSS) information. The action space is discrete, corresponding to selecting the transmit power, while the reward is a function of the SINR. Using simulation results, the author shows that the proposed smoothing DQN outperforms the conventional DQN in cognitive-based radio networks.

In [104], the authors propose a multi-agent DDPG-based DRL to address the problem of joint trajectory design and power allocation in multi-UAV wireless networks. In their scheme, the agents are the UAVs, whose state space is a discrete binary indicator function representing whether the QoS of the user ends (UEs) are satisfied or not. The action space is also discrete, corresponding to selecting UAVs’ trajectory and transmission power. The reward is a continuous function defined by the joint trajectory design and power allocation as well as the number of UEs covered by the UAVs. Simulation results show that the proposed algorithm achieves higher network utility and capacity than the other optimization methods in wireless UAV networks with reduced computational complexity.

High-speed railway (HSR) systems are one of the emerging IoT applications for next-generation wireless networks. Such systems are characterized by their rapid variations in the wireless environment, which mandate the development of light-weighted RRAM solutions. As a response to this, Xu et al. [105] propose a multi-agent DDPG-based DRL model to address the problem of sum-rate maximization via power allocation in hybrid beamforming-based mmWave HSR systems. In their approach, each mobile relay (MR) acts as an agent. The action space is continuous, corresponding to the transmit power level of each MR agent. Also, the state space is continuous, defined by; each MR own signal channel, local observation information of each MR, i.e., beamforming design, each MR achievable rate, and each MR transmit power in the last time step. The reward function is continuous, defined by the achievable sum-rate of the network. Simulation results demonstrate that the SE of their proposed algorithm is comparable to the full digital beamforming scheme, and it outperforms conventional approaches such as maximum power allocation, random power allocation, DQN, and FP.

Federated deep reinforcement learning (FDRL) is an emerging technique that integrates FD and DRL methods. FDRL can be utilized as an efficient tool to enhance the RRAM solutions in large-scale distributed systems). As an example, an interesting approach is proposed in [106], in which the authors propose a cooperative multi-agent actor-critic-based FDRL framework for distributed wireless networks. The authors particularly address the problem of energy/spectrum efficiency maximization via distributed power allocation for network edge users. In their proposed model, the agents are the edge users, whose action space is continuous, defined as the power allocation strategies. The state space is continuous, defined by the allocated transmit power, the SINR on the assigned RBs, and the reward of the previous training time step. The system is defined in terms of a local continuous cost function expressed in terms of SINR, power, path loss, and environmental noise. Using simulation results, the authors demonstrate that their proposed framework achieves better performance accuracy in terms of power allocation than other approaches such as greedy, non-cooperation power allocation, and traditional FL.

In the following paragraphs, we review works that employ DRL techniques to address the power allocation issue in satellite networks as well as emerging satellite IoT systems.

Managing downlink transmit power in satellite networks is also one of the major persistent challenges. To this end, the authors in [107] extended their work in [108] and present a single-agent Proximal Policy Optimization (PPO)-based DRL model to solve the problem of power allocation in multi-beam satellite systems. In their model, the agent is the processing engine that allocates power within the satellite, whose state space is continuous, comprises the set of demand requirements per beam, and the optimal power allocations for the two previous time steps. The action space is continuous, representing the allocation of the power for each beam, while the reward is a function of both the link data rate achieved by the beam and the power set of the agent. Experimental results demonstrate the robustness of their proposed DRL algorithm in dynamic power resource allocation for multi-beam satellite systems.

NOMA technique has shown efficient results in improving the performance of terrestrial mmWave cellular systems [109]. This has motivated the use of NOMA for satellite communication systems. However, managing the radio resources in such a system becomes an imperative issue. In this context, Yan et al. [110] study the problem of power allocation for NOMA-enabled SIoT using a single-agent DQN-based DRL scheme. In their system, the agent is the satellite, whose action space is discrete, corresponding to selecting the power allocation coefficient for each NOMA user. The state space is continuous, consisting of the average SNR, link budget, and delay-QoS requirements of NOMA users, while the reward is discrete, which is a function of the effective capacity of each NOMA user. Experimental results demonstrate that their proposed DRL-based power allocation scheme can produce optimal/near-optimal actions, and it provides superior performance to both the fixed power allocation strategies and OMA scheme.

Multi-RAT wireless HetNets is one of the main enabling technologies for modern wireless systems, including 6G networks [3]. In HetNets, several RATs with different operating characteristics coexist to enhance network coverage and reliability while providing enhanced QoE to users. The underlying RATs have non-overlapping radio resources; therefore, there would not be typically interference in the network.

Since a stand-alone network with a single RAT would not be able to support the stringent requirements of emerging disruptive applications, modern smart user devices are equipped with advanced capabilities that enable them to aggregate various radio resources to boost their QoE. Modern user devices can operate in a multi-mode scenario, in which each user device can be connected to a single RAT at any time. Alternatively, user devices can operate in a multi-homing scenario such that they can be connected simultaneously to various RATs to aggregate their radio resources, such as bandwidth and data rate. Multi-RAT networks include the coexistence of RATs, such as the licensed band networks, unlicensed bands networks, hybrid systems, and any combination of the wireless networks that are shown in Fig. 3.

Visible Light Communication (VLC) is a promising RAT that can support multi-Gbps of data rates over wireless links [111]. It is mainly developed for indoor applications; however, it is gaining considerable attention lately for outdoor applications as well [112]. This has motivated researchers to propose solutions that integrate VLC with conventional radio systems to boost aggregate data rates. Managing radio resources in these integrated systems, however, becomes a challenge. In this context, in [40], the authors propose a multi-agent $Q$-learning-based two-time scale scheme to address the power allocation issue for multi-Homing hybrid RF/VLC networks. In their technique, the agents are the RF and VLC APs, whose action space is discrete, corresponding to selecting the downlink power level that ensures the QoS’s satisfaction of the multi-homing users. The state space is discrete, which is a function of users’ achievable and target rates from the RF and VLC APs. The reward is also discrete, which is a function of the achieved and target rates from all RF and VLC APs. Experimental results demonstrate that not only the users’ target rates are satisfied, but also the ability of their algorithm to adapt to the network’s dynamics.

For the same network settings in [40], Ciftler et al. [42] propose a DRL-based scheme to enhance the results and overcome the shortcomings. In particular, the authors in [42] propose a non-cooperative multi-agent DQN-based algorithm to address the problem of power allocation in hybrid RF/VLC networks. The agents are the RF and VLC APs whose action space is discrete, representing the transmit power. The state space is continuous, comprised of the actual and target rates, while the reward function is continuous and is a function of target rate band, target rate, and actual rate. Using simulation results, the authors demonstrate that the DQN-based algorithm converges with the rate of 96.1% compared with the $Q$-learning-based algorithm’s convergence rate of 72.3%.

In the following paragraphs, we review works that employ DRL algorithms to address the spectrum and access control problem in cellular and HomNets depicted in Fig. 2.

RRAM in UAV-assisted cellular networks is also one of the main emerging challenges. Towards this end, in [113], a multi-agent DQN model addresses the joint user association, spectrum allocation, and content caching in an LTE network consisting of UAVs serving ground users. In their model, the agents are the UAVs, which have storage units and have the ability to cached contents in LTE-BSs. These UAV agents can access the licensed as well as the unlicensed spectrum bands, and a remote cloud-based server is used to control them. The licensed cellular spectrum band is used in the transmissions from the cloud to the UAVs. Each UAV agent has to obtain 1) its user association, 2) bandwidth assignment indicators in the licensed spectrum band, 3) time slot indicators in the unlicensed spectrum band, and 4) content that the users request. The input of the DQL is the other agents’ actions (the UAV-user association schemes), and the output is the set of users that the UAV can handle. Simulation results demonstrate that their proposed DQL strategy enhances the number of users up to 50% compared to the standard $Q$-learning strategy.

Based on their initial work in [114], the authors in [115] propose a multi-agent Dueling Double deep Q-network (D3QN)-based DRL model to handle the joint BS and channel selections in macro and femto BS networks sharing a set of radio channels. In their scheme, the agents are the UEs, whose state space is a discrete binary vector that shows whether UEs’ SINR higher than the minimum QoS requirement or not. The action space is discrete, corresponding to the BS and channel association. The reward function is discrete in which the UE agent will receive a utility as a reward if the QoS is met; otherwise, it will receive a negative value for the reward. Simulation results demonstrate that their proposed strategy outperforms the standard $Q$-learning strategy in terms of generalization, system capacity, and convergence speed.

The problem of user association problem in cellular IoT networks is studied in [116]. The goal is to assign IoT devices to particular cellular users to maximize the sum-rate of the IoT network. Two single-agent DQN DRL algorithms are proposed; the first one utilizes global information to makes decisions for all IoT devices at one time, while the other algorithm uses local information to make a distributed decision for only a single IoT device at one time. In their model, the BS acts as the agent whose state space is continuous, consisting of both historical CSI and interference information. The action space is discrete, representing both all possible association schemes of the network and the individual association for only a single IoT device. The reward function of the first DQN algorithm is the sum-rate of all IoT devices, while for the second DQN includes both the current transmission rate of IoT devices and the interference with other IoT devices. Experimental results demonstrate that their proposed DRL framework both scalable and achieves performance comparable to the optimal user association policy.

In another interesting work in [117], the authors study the problem of spectrum allocation in the emerging integrated access and backhaul (IAB) cellular networks characterized by their dynamic environment and large-scale deployment. The problem is formulated as a non-convex mix-integer and non-linear programming, and a DRL framework based on single-agent Double DQN and actor-critic algorithms is proposed to solve it. In their model, the agent is a center-located controller or distributed UE. The state space is discrete, indicating the status of UEs’ QoS, and the action space is discrete, corresponding to the allocation matrix for the donor BS and IAB nodes. The reward function is modeled to optimize the proportional fairness allocation of the network. Experimental results demonstrate that their framework has promising results compared to other conventional spectrum allocation policies.

The problem of load balancing in large-scale and dynamic wireless networks is also another important issue. In this context, the authors in [118] present a multi-agent $Q$-learning-based algorithm to address the problem of user association for load balancing in vehicular networks. In their scheme, the agents are the BSs, whose action space is discrete, representing the associations with the network’s vehicles. The state space is a hybrid (continuous and discrete), consisting of the service resources and its service demands, SINR matrix, and association matrix. The reward is a continuous function defined through the association and SINR matrices. Using experiments on real-field taxi movements, the authors evaluate the performance of their proposed algorithm, and they show that it provides higher quality load balancing compared to conventional association methods.

In other work in [119], the authors propose a multi-agent $Q$-learning-based algorithm for the cognitive RAT selection issue in 5G HetNets. In their model, the agents are the terminal devices whose action is discrete, corresponding to identifying the available RATs. The state space is also discrete that corresponds to the currently active RAT association, while the reward function is continuous, which is defined by the average measured throughput and handovers call drops. Using experimental results, the authors show that their proposed framework outperforms the random and max-SINR decision approaches.

Zheng et al. [120] propose a single agent actor-critic-based DRL algorithm to address the problem of channel assignment for hybrid NOMA-based 5G cellular networks. The agent is the BS, whose action space is discrete, corresponding to assigning channels for users. The state space is a hybrid (continuous and discrete) comprised of three elements; the CSI matrix, achieved users’ data rate in the previous time slot, and the assigned channels in the previous time slot. The reward is a discrete function defined in terms of users’ SE, the number of channels that use NOMA for transmission, and the number of users whose data rate is zero. Simulation results demonstrate that their proposed method outperforms some conventional approaches, such as greedy, random, match theory-based, and Genetic Algorithms, in terms of both network SE and sum-rate.

The problem of spectrum management in wireless DSA is addressed in [121] based on distributed multi-agent DQN. In their approach, the agents are each DSA user, whose action space is discrete, corresponding to the transmit power change for each channel. The state space is discrete, defined as the transmit power on wireless channels. The reward is a continuous function defined by the SE and the penalty caused by the interference to PUs. Experimental results show that their proposed model with echo state network-based DQN achieves a higher reward with both the achievable data rate and PU protections.

In the following paragraphs, we review works that employ DRL algorithms to address the spectrum and access control problem in IoT and emerging wireless networks illustrated in Fig. 2.

In [122, 123, 124], the authors propose a single-agent DQN-based DRL scheme to tackle the problem of dynamic channel access for IoT sensor networks. In their scheme, the agent is the sensor, and its action is discrete, corresponding to selecting one channel to transmit its packets at each time slot. The state space is discrete, which is a combination of rewards and actions in the previous time slots. The reward function is also discrete, which is ”+1” if the selected channel is in low interference in such case a successful transmission occurs; otherwise, the reward is ”-1” in such case the selected channel is in high interference, and a transmission failure occurs. Simulation results show that their proposed scheme achieves an average reward of 4.4 compared to 4.5 obtained using the conventional myopic policy [125], which needs a compact knowledge of the transition matrix of the system.

Other work in [126] develops a single-agent DQN-based DRL model to address the channel selection in energy harvesting-based IoT sensors. In that work, the agent is one BS, which controls the channel assignments for energy harvesting-enabled sensors. The problem of the agent is to predict the battery level of the sensors and to assign channels to sensors such that the total rate is maximized. The DQL model used to solve this problem has two long-short-term-memory (LSTM) neural network (NN) layers, one for predicting the sensor’s battery state and one for obtaining channel access policy based on the predicted states obtained from the first layer. The agent’s action is all the available sensors that require to access the channels. The state contains the history of channel access scheduling, true and predicted battery information history and the current sensor’s CSI. Simulation results show that the total rates obtained using the DQL scheme are 6.8 kbps compared to 7 kbps obtained from the optimal scheme rate.

A multi-agent DQN-based DRL model is proposed in [127] in order to address the cooperative spectrum sensing issue in CRNs. In their scheme, the agents are the SUs whose action is discrete, corresponding to sensing the spectrum for possible transmission without interfering with the PUs. The state space is discrete, and it is comprised of four elements representing cases when the spectrum is sensed as occupied, the spectrum is not sensed in a particular time slot, the spectrum is sensed as idle, and one of the other SUs broadcast the sensing result first. The reward function is the binary indicator, which is ”+1” if the spectrum is sensed as idle and ”0” otherwise. Simulation results show that their proposed algorithm has a faster convergence speed and better reward performance than the conventional $Q$-learning algorithm.

Xu et al. [128] propose a single-agent DQN and DDQN-based DRL approaches to address the problem of dynamic spectrum access in wireless networks. In their model, the agent is a wireless node (e.g., a user) whose action is discrete, corresponding to sensing the discrete frequency channel for possible data transmission. The state space is discrete, defining if the channel is occupied or idle at time slot $t$. The reward function is discrete, which is ranging from 0 to 100 for successful transmission; otherwise, the reward is ”-10” if the channel state is occupied and the user transmission fails. It is shown using simulation results that both DQN and DDQN can learn different nodes’ communication patterns and achieve near-optimal performance without prior knowledge of system dynamics.

Based on their initial work in [129], the authors in [130] propose a distributed single- and multi-agent DQN-based DRL schemes to address the spectrum sharing problem in V2X networks. In their proposed system, multiple V2V links reuse the frequency spectrum preoccupied with V2I links. The agents are the V2V links whose action space is discrete, corresponding to spectrum sub-band and power selection. Each agent’s local observation space includes local channel information (such as its own signal channel gain, interference channels from other V2V transmitters, interference channel from its own transmitter to the BS, and the interference channel from all V2I transmitters), the remaining V2V payload, and the remaining time budget. The reward is continuous, which is a function of both the instantaneous sum capacity of all V2I links and the effective V2V transmission rate until the payload is delivered. Experimental results show that the agents cooperatively learn a policy that enables them to simultaneously improve the sum capacity of V2I links and payload delivery rate of V2V links. The authors also show that their proposed models for the single-agent and multi-agent settings provide very close performance to the conventional exhaustive search.

The authors in [131] propose a single-agent DQN model to address the joint channel access and packet forwarding in a multi-sensor scenario. In the proposed scheme, one sensor is the agent, which acts as a relay to forward packets arriving from its surrounding sensors to the sink. The agent has a buffer to store arriving packets. The agent’s action is to choose channels for the packet forwarding, the packets transmitted on these channels, and a modulation scheme at each time slot to maximize its utility (defined as the ratio of the transmitted packets number to the transmit power). The state is the combination of the buffer and channel states. The input of the DQL model is the state, while the output is the corresponding action selection. Simulation results demonstrate that the proposed DQL scheme enhances system utility (i.e., 0.63) compared to the conventional random action selection scheme (i.e., 0.37).

Khan et al. [132] propose a multi-agent A3C-based DRL to address the problem of vehicle-cell association in mmWave V2X networks. The agents are the RSUs whose action is discrete, corresponding to determining the optimal vehicle-RSU association for RSU. The state space is a hybrid (discrete and continuous) defined in terms of the last channel observations, rate threshold violation indicator, and experienced data rate of vehicles. The reward function is continuous, which is defined in terms of the average rate per vehicle and threshold rate. Using experimental results, it is shown that their proposed algorithm achieves around 15% sum-rate gains and a 20% reduction in vehicular user outages compared to baseline approaches.

The problem of dynamic spectrum access in CRNs is investigated in [133] through combining DRL and evolutionary game theory. In particular, uncooperative multi-agent DQN is considered in which the agents are the SUs whose action is discrete, corresponding to selecting the access channel. The state space is discrete, which includes two main parts; the channel selected by the agent and the utility obtained after transmission on the selected channel. The reward function is defined in terms of evolutionary game theory. Simulation results indicate the performance enhancement of their proposed algorithm over the case without learning in terms of average system capacity.

Another interesting work is presented in [134] in which the authors propose a multi-agent DQN-based DRL algorithm to address the problem of optimum multi-user access control in Non-Terrestrial Networks (NTNs). In their model, UEs are the independent agents that report their experiences and local observations to a centralized trainer controller located at the backhaul network. The latter will then utilize the collected experiences to update the global DQN parameter. The agent’s state space is continuous, comprised of the connected NT-BS of UEs at the previous time slots, the RSS of UEs, the number of connected UEs of each NT-BS, and the transmission rate of UEs. The action space is discrete, representing the binary indicator functions of UEs, while the reward is a function of the transmission rate of UEs. Experimental performance results show that their proposed scheme is efficient in addressing the fundamental issues in the deployment of NTNs infrastructure, and it outperforms the traditional algorithms in terms of both the data rate and the number of handovers.

In [135], the authors propose a multi-agent DQN-based DRL scheme to address the problem of dynamic joint spectrum access and mode selection (SAMS) in cognitive radio networks (CRNs). The agents are the secondary users (SUs) whose action space is discrete, corresponding to selecting the access channel and access mode. The state space of each SU agent is discrete, comprised of the action taken by the $m$th SU agent, the ACKs of all SUs agents, and the ACK of the $m$th SU agent. The reward function is discrete, which is ”1” if the action selection process is successful; otherwise, there is a collision, and the agent will receive a ”0” reward.

Tomovic et al. [136] propose a single-agent DRL model based on the integration of Double deep $Q$-learning architecture and RNN to address the problem of DSA in multi-channel wireless networks. In particular, the agent is the SU node, whose action space is discrete, representing the selection of a channel for sensing. The state space is also discrete, comprised of a history of the binary observations and history of taken actions. The reward function is a discrete binary function, which is ”1” if the observation is ”1” and ”0” otherwise. Simulation results show that their proposed method is able to find a near-optimal policy in a smaller number of iterations, and it can support a wide range of communication environment conditions.

In other work in [137], the authors propose both a single-agent and multi-agent deep actor-critic DRL-based framework for dynamic multi-channel access in wireless networks. In their system, the agents are the users whose action space is discrete, corresponding to selecting a channel. The observation space is also discrete, which is defined based on the status of the channel and collision status. The reward function is discrete, which is ”+1” if the selected channel is good; otherwise, it is ”-1”. Using simulation results, the authors show that their proposed actor-critic framework outperforms the DQN-based algorithm, random access, and the optimal policy when there is full knowledge of the channel dynamics.

The problem of DSA for the CRN is studied in [138] based on an uncoordinated and distributed multi-agent DQN model. The agents are CRs, whose action is discrete, representing the possible transmit powers. The state space is discrete, reflecting whether the limits for DSA are being met or not, depending on the relative throughput change at all the primary network links. The reward is also discrete, which is a function of the throughput of the links and the environment’s state. Experimental results reveal that their proposed scheme finds policies that yield performance within 3% of an exhaustive search solution, and it finds the optimal policy in nearly 70% of cases.

Industrial IoT (IIoT) has emerged recently as an innovative networking ecosystem that facilitates data collection and exchange in order to improve network efficiency, productivity, and other economic benefits [139]. RRAM in such a sophisticated ecosystem is also a challenge that needs more investigation. In this context, recently in [140], the authors propose a solution for spectrum resource management for IIoT networks, with the goal of enabling spectrum sharing between different kinds of UEs. In particular, a single-agent DQN algorithm is proposed in which the agent is the BS. The action space is discrete, which corresponds to the allocation of spectrum resources for all UEs. The observation space is a hybrid (continuous and discrete) consisting of four elements; the current action (i.e., the allocation of spectrum resources), the data priority of type I UEs, the buffer length of type II UE, and the communication status of the first type of UEs. The reward function is continuous, defined to address their optimization problem. It is divided into four objectives; 1) maximizing the spectrum resource utilization; 2) quickly transmitting the high-priority data; 3) meeting the threshold rate requirement of the first type of UEs; 4) ensuring that the second type of UEs completes the transmission in time. Using simulation results, it is demonstrated that their proposed algorithm achieves better network performance with a faster convergence rate compared with other algorithms.

In [141], the authors propose a multi-agent Double DQN-based DRL model to address the problem of DSA in distributed wireless networks. In particular, they design a channel access scheme to maximize channel throughput regarding fair channel access. The agents in their scheme are the users. The action space is discrete, which is ”0” if the user does not attempt to transmit packets during the current time slot, and it is ”1” if it has attempted to transmit. The state space is discrete, consisting of four main elements; each user action taken on the current time slot, channel capacity (which could be negative, positive, or zero), a binary acknowledgment signal showing if the user transmits successfully or not, and a parameter that enables each user to estimate other users’ situations. The reward is a discrete binary function that takes the value of ”1” if the user transmits successfully; otherwise, it is ”0” meaning that the user transmitted with collision. It is shown using simulation results that their scheme can maximize the total throughput while trying to make fair resource allocation among users. Also, it is shown that their proposed scheme outperforms the conventional Slotted-Aloha scheme in terms of sum-throughput.

Wang et al. [142] address the problem of DSA in VANETs, by proposing an interesting scheme that combines multi-hop forwarding via vehicles and DSA. The optimal DSA policy is defined to be the joint maximization of channel utilization and minimization of the packet loss rate. A multi-agent DRL network structure is presented that combines RNN and DQN for learning the time-varying process of the system. In their scheme, each user acts as an agent whose action space is discrete, corresponding to choosing a channel for transmission at time slot $t$. The state space is discrete, composed of three components; a binary transmission condition $\eta$, which is ”1” if the transmission is successful and ”0” otherwise, the channel selection action, and the channel status indicator after each dynamic access process. The reward is a discrete binary function, which takes a positive value if $\eta=1$, otherwise it takes the value of ”0”. Simulation results show that their proposed scheme: 1) reduces the packet loss rate from 0.8 to around 0.1, 2) outperforms Slotted-Aloha and DQN in terms of reducing collision probability and channel idle probability by about 60%, and 3) enhances the transmission success rate by around 20%.

Most recently in [143], the authors propose multi-agent DQN-based DRL schemes to address the problem of joint cooperative spectrum sensing and channel access in clustered cognitive UAV (CUAV) communication Networks. Three algorithms are proposed: 1) a time slot multi-round revisit exhaustive search based on virtual controller (VC-EXH), 2) a $Q$-learning based on independent learner (IL-Q) and 3) a DQN based on independent learner (IL-DQN). The agents are the CUAVs in the network. The action space of any CUAV agent is a discrete function defined by the steps that this agent moves clockwise in time slot $t$ relative to the channel selected in time slot $t-1$ on the PU channel ring. The state space is a discrete set consisting of two main elements: 1) the number of CUAVs agents that have selected a particular channel to sense and access in the previous time slot and 2) a binary indicator function that shows the occupancy status of a particular channel in the previous time slot. The reward is a discrete function defined in terms of spectrum sensing, channel access, utility, and cost. Experimental results show that all the three algorithms proposed show efficient results in terms of convergence speed and the enhancement of utilization of idle spectrum resources.

The authors in [144] propose a multi-agent deep recurrent Q-network-based DRL model to address the problem of DSA in dynamic heterogeneous environments with partial observations. In their work, the authors consider a case-study with multiple heterogeneous PUs sharing multiple independent radio channels. The agents are the SUs, whose action space is discrete, corresponding to deciding whether to transmit in a particular band or wait during the next time slot. The state space is discrete, representing whether the channels are occupied, idle, or unknown. The reward function is discrete, represented by two values; 100 per channel for successful transmission and -50 per channel for collision. Using simulation results, the authors show that their proposed algorithm handles various dynamic communication environments, and its performance outperforms the myopic conventional methods and very close to the optimization-based approaches.

In the following paragraphs, we review works that employ DRL algorithms to address the spectrum and access control problem in satellite networks and emerging satellite IoT systems.

The work in [145] proposes a single-agent DQN-based DRL algorithm that considers the problem of channel assignment in multi-beam satellite systems. In their scheme, the agent is the satellite, whose action is discrete, including an index that indicates the channel that the newly arrived user has occupied. The agent’s reward is discrete, which contains a positive value if the service is satisfied, and a negative value if the service is not satisfied or blocked. The state space is also discrete, which comprises three elements; the current users, the current channel assignment matrix, and a list of the new user arrivals. Experimental results demonstrate that their proposed scheme decreases the blocking probability and improves the carried traffic up to 24.4% as well as enhances the spectrum efficiency compared to the conventional fixed channel assignment approach.

The authors in [146] propose a single-agent DQN-based DRL algorithm to address the problem of dynamic channel allocation in multi-beam satellite systems. In particular, an image-like tensor formulation on the system environments is considered in order to extract traffic spatial and temporal features. The agent in their model is the satellite, whose action space is discrete, corresponding to determining the resource allocation schemes. The state space is continuous, consisting of two elements; the system resource allocation state and the users’ service request state. The reward function is discrete, which is defined in terms of the optimization objective function.

Recently, the authors in [147] propose a single-agent DQN-based DRL approach for energy-efficient channel allocation in SIoT. The agent in their model is the LEO satellite, whose action space is discrete, corresponding to mapping from newly coming node tasks to channels to be allocated. The state space is discrete, including information about user tasks, such as the size and location of tasks. The reward is continuous, which is divided into two normalized reward function components; the power efficiency reward and the normalized value of the service blocking rate. Both of these reward components are functions of power set up by the agent, the optimal power decided by the location of the beam, and the number of served nodes. Experimental results demonstrate that their proposed algorithm saves energy consumption of around 67.86% compared to some conventional approaches.

In [148], the authors propose a centralized single-agent DQN-based scheme to address the problem of dynamic channel allocation in SIoT. The agent in their model is the satellite, whose action is discrete, corresponding to selecting which sensors to allocate channels to. The state space is discrete, comprised of three parts; the number of tasks in each time step, the bandwidth that a sensor node requires, and the duration of a new task. The reward is continuous, which is a function of the duration of data transmission for the sensor. Using simulation results, it is shown that their proposed algorithm both provides higher transmission success rates and reduces data transmission latency by at least 87.4% compared to the conventional channel allocation algorithms.

Zheng et al. [149] propose a single-agent $Q$-learning-based RL model to address the problem of combination allocation of fixed channel pre-allocation and dynamic channel scheduling in a network architecture of LEO satellites that utilizes a centralized resource pool. In their model, the satellite serves as an agent whose action is discrete, corresponding to assigning channels to users. The state space is discrete, defined by the channel assignment of users in each beam. The reward is continuous, which is a function of the user’s supply-demand ratio. Experimental results demonstrate that their proposed approach enhances system supply-demand ratio by 14% and 18% compared to the static channel allocation the Lagrange algorithm channel allocation method.

In the following paragraphs, we review works that employ DRL algorithms to address the spectrum and access control problem in multi-RAT HetNets. This includes the coexistence of various variants of the wireless networks shown in Fig. 2.

In [150], a multi-agent DQN-based model that jointly tackles the dynamic channel selection and interference management in Small Base Stations (SBSs) cellular networks that share a set of unlicensed channels in Long Term Evolution (LTE) networks. In the proposed scheme, the SBSs are the agents who choose one of the available channels for transmitting packets in each time slot. The agent’s action is channel access and channel selection probability. The DQL input includes the channels’ traffic history of both the SBSs and Wireless Local Area Networks (WLAN), while the output is the agent’s predicted action vectors. Simulation results reveal that their proposed DQL strategy enhances the average data rate by up to 28% when compared to the conventional $Q$-learning scheme.

In [151], a single-agent DQN-based model is proposed to tackle the dynamic spectrum allocation for multiple users that share a set of $K$ channels in the same network settings of [150]. In their scheme, the agent is the user whose action is either choosing a channel with a particular attempt probability or selecting not to transmit. The agent’s state includes the history of the actions of the agent and its current observations. The DQL model input is the previous actions along with their observations, while the output is the Q-values corresponding to the actions. Simulation results demonstrate that their proposed DQL strategy achieves a double data rate compared to the state-of-the-art Slotted-Aloha scheme.

Wang et al. [152] propose a single-agent prediction-DDPG-based DRL algorithm to study the problem of the dynamic multichannel access (MCA) for the hybrid long-term evolution and wireless local area network (LTE-WLAN) aggregation in dynamic HetNets. The agent is the central BS controller, whose state space is continuous, consisting of both the channels’ service rates and the users’ requirement rates. The action space, on the other hand, is discrete, representing the users’ index. Two reward functions are provided; online traffic real reward and online traffic prediction reward, each of which are functions of users’ requirements, channels’ supplies, degree of system fluctuation, the relative resource utilization, and the quality of user experience. Using simulation results, the authors demonstrate the efficiency of the proposed prediction-DDPG model in solving the dynamic MCA problem compared to conventional methods.

Other interesting work in [153], the authors investigate the joint allocation of the spectrum, computing, and storing resources in multi-access edge computing (MEC)-based vehicular networks. In particular, the authors propose multi-agent DDPG-based DRL algorithms to address the problem in a hierarchical fashion considering a network comprised of macro eNodeB (MeNB) and Wi-Fi APs. The agents are the controller installed at MEC servers. The agents’ action space is discrete including the spectrum slicing ratio set, spectrum allocation fraction sets for the MeNB and for each Wi-Fi AP, computing resource allocation fraction, and storing resource allocation fraction. The state space is discrete representing information of the vehicles within the coverage area of the MEC server, including vehicles’ number, x-y coordinates, moving state, position, and task information. The reward function is discrete, defined in terms of the delay requirement, and requested storing resources required to guarantee the QoS demands of an offloaded task. Provided experimental results reveal that their proposed schemes achieve high QoS satisfaction ratios compared with the random assignment techniques.

Recently in [154], the authors propose a single-agent DQN algorithm based on Monte Carlo Tree Search (MCTS) to address the problem of dynamic spectrum sharing between 4G LTE and 5G NR systems. In particular, the authors used the MuZero algorithm to enable a proactive BW split between 4G LTE and 5G NR. The agent is a controller located at the network core, whose action space is discrete, corresponding to a horizontal line splitting the BW to both 4G LTE and 5G NR. The state space is discrete, defined by five elements: 1) an indicator if the user is an NR user or not, 2) the number of bits in the user’s buffer, 3) an indicator of whether the user is configured with multimedia broadcast single frequency network (MBSFN) or not, 4) the number of bits that can be transmitted for the user in a given subframe, and 5) the number of bits that will arrive for each user in the future subframes. The reward function is a continuous function defined as a summation of the exponential of the delayed packet per user. Experimental results show that their proposed scheme provides comparable performance to the state-of-the-art optimal solutions.

In the following paragraphs, we review works that employ DRL algorithms to address the joint RRAM issue in cellular and HomNets shown in Fig. 2.

The work in [159] presents a single-agent DDPG-based DRL model to study the joint optimization of user mode selection, bandwidth allocation, power control, and channel selection with the aim for maximizing the EE in D2D-enabled cellular HetNets. The agent is a controller in the macrocell, whose state space is continuous, representing the user’s QoS satisfaction degree. The action space is continuous, which is to select power, channel, and bandwidth allocation factor, while the reward is a continuous function defined by the system EE. Experimental results demonstrate the efficiency of their proposed algorithm in terms of both the convergence speed and improving the EE in a D2D-enabled HetNets compared with other benchmark schemes.

In another interesting work in [160], the authors study the problem of joint optimization of transmission mode selection and resource allocation for CV2X. They propose single-agent settings in which DQN and federated learning (FL) models are integrated to improve the model’s robustness. The agent in their model is each V2V pair. The action space is discrete, representing the resource block (RB) allocation, communication mode selection, and transmit power level of the V2V transmitter. The state space is a hybrid (continuous and discrete) consisting of five main parts; the received interference power at the V2V receiver and the BS on each RB at the previous subframe, the number of selected neighbors on each RB at the previous subframe, the large-scale channel gains from the V2V transmitter to its corresponding V2V receiver and the BS, current load, and remaining time to meet the latency threshold. The reward is a continuous function defined in terms of the sum-capacity of vehicular UEs as well as the QoS requirements of both vehicular UEs and V2V pairs. Using experimental results, the authors show that their proposed two-timescale federated DRL algorithm outperforms other decentralized baselines.

Jang et al. [161] propose a multi-agent DQN-based algorithm to address the problem of sum-rate maximization via a joint optimization of resource allocation and power control in small cell wireless networks. The agents in their proposed model are the small cell BSs, whose action space is discrete, corresponding to selecting the resource allocation and power control of small BS on RB. The state space is continuous, including all the CSI that the small BS collects on RB, such as local CSI, local CSI at the transmitter, etc. The reward is a continuous function expressed by the average sum-rate of its own serving users and the other small BSs. Experimental results show that their proposed approach both outperforms the conventional algorithms under the same CSI assumptions and provides a flexible tradeoff between the amount of CSI and the achievable sum-rate.

In another interesting work in [162], the authors propose a model-driven multi-agent Double DQN-based framework for resource allocation in UDNs. In particular, They first develop a DNN-based optimization framework comprised of a series of ADMM iterative procedures that uses the CSI as the learned weights. Then, a channel information absent $Q$-learning resource allocation algorithm is presented to train the DNN-based optimization scheme without massive labeling data, where the EE, SE, and fairness are jointly optimized. The agents are each D2D transmitter, whose action space is discrete, corresponding to selecting a subcarrier and corresponding transmission power. The state space is a hybrid (continuous and discrete) consisting of two parts; user association information and interference power. The reward function is continuous, comprised of two components; the network EE and the fairness of service quality, which is expressed by the variance of throughput between authorized users. Using experimental results, it is demonstrated that their proposed algorithm has a rapid convergence speed, well characterizes the extent of optimization objective with partial CSI, and outperforms other existing resource allocation algorithms.

Liu et al. [163] propose a single-agent Dueling DQN-based DRL model to address the problem of joint energy efficiency (EE) and spectral efficiency (SE) in 5G ultra-dense networks (UDNs). In their approach, the agent is a macro gNodeB (MgNB) whose state space is continuous, comprised of the number and throughput of all small cells as well as the allocation of all resource blocks (RBs) in the network. The agent’s action space is to define which RBs are reused by the small cell in the network, while the reward is a function of the tradeoff between SE and EE. Experimental results demonstrate that their proposed algorithm performs better than both the conventional $Q$-learning and DQN.

The authors in [164] propose a multi-agent Double DQN-based scheme to address the problem of joint subcarrier assignment and power allocation in D2D underlying 5G cellular networks. The agents in their model are the D2D pairs, whose action space is discrete, corresponding to determining the transmit power allocation on the available subcarriers. The state space is a hybrid (continuous and discrete), comprised of four components: 1) local information (including the previous transmit power, previous SE achieved by transmitters, channel gain, and SINR), 2) the interference that each agent causes at the BS side, 3) the interference received from agent’s interfering neighbors and the SE achieved by agent’s neighbors, and 4) the interference that each agent causes to its neighbors. The reward is a continuous function comprised of three elements: 1) the SE achieved by each agent, 2) the SE degradation of the agent’s interfered neighbors, and 3) the penalty due to the interference generated at the BS. Experimental results show that their proposed algorithm outperforms both the exhaustive and random subcarrier and even power (RSEP) assignment methods in terms of SE of D2D pairs.

In previous work in [165], the authors propose a multi-agent $Q$-learning-based scheme to address the problem of joint channel and power level selection in autonomous D2D-based heterogeneous cellular networks. Two spectrum usage scenarios are considered; when the D2D pairs transmit over the dedicated spectrum bands and when they shared cellular/D2D channels. The agents are the D2D pairs whose action space is discrete, defined as the set of their possible channel and power level decisions. The state space is also discrete, which takes the value of ”1” if the SINR is greater than some predefined threshold; otherwise, the value of the state is ”0”. The agent’s reward is a continuous function, defined by the current agent’s state and its partially observable actions. Experimental results show that their proposed algorithm has relatively fast convergence and near-optimal performance after a few number of iterations.

Most recently in [166], the authors propose a multi-agent DQN-based DRL scheme to address the problem of spectrum allocation and power control for Mission-critical communication (MCC) in 5G networks. In MCC, multiple D2D users reuse non-orthogonal wireless resources of cellular users without BS in order to enhance the network’s reliability. The paper aims to help the D2D users autonomously select the channel and allocate power to maximize system capacity and SE while minimizing interference to cellular users. The agents are the D2D transmitters whose action space is discrete, corresponding to channel and power level selection. The state space is discrete, defined in a three-dimensional matrix, which includes information on the channel state of uses, the state of power level, and the number of the D2D pairs. The reward function is discrete, defined in terms of the total system capacity and constraints. Simulation results show that their proposed learning approach significantly improves spectrum allocation and power control compared to traditional methods.

The authors in [167] propose a multi-agent DQN-based model to address the problem of joint user association and power control in OFDMA-based wireless HetNet. The agents are the UEs, whose action space is discrete, corresponding to jointly associate with the BS and determine the transmit power. The state space is discrete, which is defined by the situation of all UEs association with BS and power control. The reward function is continuous, which is defined in terms of the sum-EE of all UEs. Using simulation results, it is shown that their proposed method outperforms the $Q$-learning method in terms of convergence and EE.

The authors in [168] propose a decentralized multi-agent DQN-based scheme to address the problem of joint channel resource and transmit power in D2D underlay cellular networks. In their scheme, the agents are each D2D transmitter. The action space is discrete, corresponding to the selection of each D2D’s channel resource and transmit power for data transmission. The state space is discrete consisting of six elements; the transmit power level of the cellular users, the maximum transmit power of each D2D transmitter, the distances between the cellular users and the D2D receiver devices, distances between D2D transmit devices and D2D receiver devices, selected channel of other agents, and selected transmit power level of other agents. The reward is a continuous function defined by the total effective throughput of D2D pairs in the network. Simulation results show that their proposed algorithm achieves better performance in terms of the total effective data rate than the random resource allocation method.

In [169], a single-agent DQN-based DRL model is proposed to address the problem of joint optimization of user association, resource allocation, and power allocation in HetNets. The agent is the BS, whose action is discrete, corresponding to power allocation to users. The state space is discrete, defined by the channel gain matrix and the set of users association. The reward function is continuous, defined by the utility function of users’ achieved data rate. Using simulation results, the authors show that their proposed algorithm outperforms some of the existing methods in terms of SE and convergence speed.

Huang et al. [170] propose a single-agent DQN model to address the problem of delay minimization via joint spectrum and power resource allocation in mmWave mobile hybrid access network. The agent is located in the roadside BS, whose action space is discrete, corresponding to allocating spectrum and power resources for data. The state space is discrete, consisting of information about the current power and spectrum of the resource pool, required spectrum and power, and the number of spectrum and power levels. The reward signal is a continuous function defined in terms of queueing delay and the resource length required for each data. Using simulation results, it is shown that their proposed model guarantees the URLLC delay constraint when the load does not exceed 130%, which outperforms other conventional methods such as random and greedy algorithms.

In the following paragraphs, we review works that employ DRL algorithms to address the joint RRAM issue in IoT and emerging wireless networks depicted in Fig. 2.

For the same system settings in [102], the authors in [64] extended their work and propose a multi-agent DDPG-based DRL framework to address the problem of the joint spectrum and power allocation in wireless networks. Two DRL-based algorithms are proposed, which are executed and trained simultaneously in two layers in order to jointly optimize the discrete subband selection and continuous power allocation. The agent in their approach is each transmitter. In the top layer, the action space of all agents is discrete, representing the discrete subband selection, while the bottom layer has a continuous action space corresponding to the transmit power allocation. The state space is a hybrid (continuous and discrete), containing information on achieved SE, transmit power, sub-band selection, rank, and downlink channel gain. The reward is shared by both layers, which is a continuous function defined in terms of the externality of agents to interference and the spectral efficiency. Using experimental results, the authors show that their proposed framework outperforms the conventional fractional programming algorithm.

Based on their initial work in [171], the authors in [172] extended their work and propose a distributed multi-agent DQN-based DRL scheme to address the problem of joint channel selection and power control in D2D networks. The agents in their model are the D2D pairs, whose action space is discrete, corresponding to selecting a channel and a transmit power. The state space of each agent is a hybrid (continuous and discrete) which contains three sets of information; local information, non-local information from the agent’s receiver-neighbor set, and non-local information from the agent’s transmitter-neighbor set. The reward function of each agent is continuous, which is decomposed into the following elements; its own received signal power, its own total received SINR, its interference caused to transmitter-neighbors, the received signal power, and the total received SINR of transmitter-neighbors. Using simulation results, it is shown that the performance of their scheme closely approaches that of the FP-based algorithm even without knowing the instantaneous global CSI.

In [173], the authors extended their previous work in [174, 175] and present a distributed multi-agent DQN-based model to address the problem of joint sub-band selection and power level control in V2V communication networks. Their proposed model is applicable to both unicast and broadcast scenarios. The agents are the V2V link or vehicles whose action space is discrete, corresponding to the selection of the frequency band and transmission power level that generate small interference to all V2V and V2I links while ensuring enough resources to meet latency constraints. The state space is continuous, containing the following information; the CSI of the V2I link, the received interference signal strength in the previous time slot, the channel indices selected by neighbors in the previous time slot, the remaining load for transmission, and the time left before violating the latency constraint. The reward function is continuous, consisting of three components; the capacity of V2I links, the capacity of V2V links, and the latency constraint. Using experimental results, it is shown that agents learn to satisfy the latency constraints on V2V links while minimizing the interference to V2I communications.

The authors in [176] propose a single-agent Double DQN-based DRL to address the problem of joint channel selection and power allocation with network slicing in CRNs. The aim of their study is to provide high SE and QoS for cognitive users. The agent is the overall CRN, whose action space is discrete, corresponding to the channel selection and power allocation of SUs. The state space is continuous, defined by the SINR of the PU. The reward function is continuous, which is a function of the system SE, user QoS, interference temperature, and the interference temperature threshold. Experimental results show that their proposed algorithm improves the SE and QoS and provides faster convergence and more stable performance compared to the $Q$-learning and DQN algorithms.

The problem of joint subcarrier assignment and power allocation in an uplink multi-user 5G-based NOMA systems is addressed in [177]. A multi-agent two-step DRL algorithm is proposed; the first step employs the DQN algorithm to output the optimum subcarrier assignment policy, while the second step employs the DDPG algorithm to dynamically allocate the transmit power for the network’s users. The agent is a controller located at the BS, whose action space is a hybrid (discrete and continuous), corresponding to the subcarrier assignment decisions and power allocation decisions. The state space is continuous, which is defined by the users’ channel gains at each subcarrier. The reward function is defined as the sum EE of the NOMA system. Experimental results show that their proposed algorithm provides better EE than the fixed and DQN-based power allocation schemes.

In [178], the authors propose a single-agent DQN-based algorithm to address the problem of joint channel assignment and power allocation in multi-carrier NOMA-based systems. The authors used an attention-based neural network (ANN) to perform the channel assignment process. The agent in their approach is the BS, whose action is discrete, representing the selection of a channel for a user. The state space is discrete, characterized by the channel information that is defined by user-channel assignment pairs. The reward function is continuous, defined by the user’s achieved data rate. Using simulation results, the authors show that their proposed algorithm achieves better system performance in terms of sum-rate, compared to the state-of-the-art exhaustive search approaches and the method reported in [182].

In another work [179], the authors address the problem of dynamic optimization of channel selection and power allocation in wireless networks based on distributed multi-agent DQN algorithm. The authors adopted a partially distributed framework with the aim of minimizing the long-term average data backlog of the network’s users. In their model, the agents are the users whose action space is discrete and finite, representing the users’ transmit power allocation for data blocks in the data buffer. The state space is also discrete and finite comprised of two parts; the users’ data size and data block remaining lifetime. The reward function is continuous, defined in terms of the size of the remaining data in the data block and the transmitted data taken from the data block. Experimental results demonstrate that their proposed solution outperforms the conventional exhaustive search and Block Coordinate Descent-based optimization algorithms in terms of EE and data backlog.

The authors in [180] propose a multi-agent DDPG-based model to address the problem of joint power and spectrum allocation in NOMA-based V2X networks. In particular, the authors are looking to maximize the sum-rate of V2I communications. The agents are the V2V communication links. The state space is discrete, defined by a set of actions performed by V2I and V2V communication links. The set includes the transmit power of both V2I and V2V links as well as the spectrum multiplexing factor of both V2V and V2V links. The state space is continuous, defined by five parts; the local channel gain information of each V2V link, interference channels from other V2V communication links, interference channel from each link’s own transmitter to the BS, interference channel from all V2I transmitters, and the state of queue length in the buffer of each V2V transmitter. The reward function is continuous, defined by the achieved sum-rate of V2I communication links and the delivery probability of V2V communication links. Compared with both the DQN algorithm and random resource allocation scheme, simulation results show that their proposed algorithm outperforms both of them in terms of maximizing the sum-rate of V2I communication links while meeting the latency and reliability requirements of V2V communications.

Munaye et al. [158] propose a multi-agent DQN-based DRL model to address the problem of joint radio resources of bandwidth, throughput, and power in UAV-assisted IoT networks. The agents are the UAVs, whose action space is discrete, corresponding to jointly selecting channel allocation of bandwidth, throughput, and power. The state space is discrete, comprising three components; the air-to-ground channel used by users, the rate of power consumption, and the interference. The reward is a discrete function, defined in terms of throughput, power allocation, bandwidth, and SINR levels. Simulation results show that their proposed algorithm outperforms other algorithms in terms of accuracy, convergence speed, and error.

In the following paragraphs, we review works that employ DRL algorithms to address the joint RRAM problem in multi-RAT HetNets. This includes the coexistence of various variants of the wireless networks as illustrated in Fig. 2.

Most recently, the authors in [181] present a multi-agent DQN-based algorithm to address the problem of joint optimization of bandwidth, power, and user association in hybrid RF/VLC systems. The APs are the agents whose action is discrete, representing the bandwidth, association function, and power level. The state space is discrete, which is a function of the problem constraints such as system bandwidth, association function, and power levels. The reward is discrete, which is a function of the rates delivered by the APs. Experimental results show that their algorithms improve the achievable sum-rate and number of iterations for convergence by 10% and 54% compared to that obtained using conventional optimization approaches.

Another interesting work is proposed by Alwarafy et al. in [39]. In that work, the authors propose a hierarchical multi-agent DQN and DDPG-based algorithm to address the problem of sum-rate maximization in multi-RAT multi-connectivity wireless HetNets. The authors addressed the problem of multi-RATs assignment and continuous power allocation that maximize the network sum rate. In their model, single and multi-agents are proposed. The edge server acts as a single agent employed by DQN, while RATs APs behave as multi-agents employed by DDPG. For the single-agent DQN model, the action space is discrete, corresponding to the RATs-EDs assigning process. The state space of the DQN is continuous, comprised of the link gains and the required data rates of users. The reward function of the DQN agent is continuous, defined by the difference between the achieved rate and the required rate by users. For the multi-agent DDPG models, the action space is continuous, representing the power allocation of each RAT AP agent. The state space is a hybrid (continuous and discrete) consisting of four elements: the RATs-EDs assignment process performed by the DQN agent, the minimum data rate of users, the gains of the links, and the achieved data rate. Experimental results show that their algorithm’s performance is approximately 98.1% and 95.6% compared to the conventional CVXPY solver that assumes full knowledge of the wireless environment.

An interesting work is presented in [157], in which the authors address the problem of network selection with the aim of optimizing medical data delivery over heterogeneous health systems. In particular, an optimization problem is formulated in which the network selection problem is integrated with adaptive compression to minimize network energy consumption and latency while meeting applications’ QoS requirements. A single-agent DDPG-based DRL model is proposed to solve it. The agent is a centralized entity that can access all radio access networks (RANs) information and Patient Edge Node (PEN) data running in the core network. The action space is discrete, corresponding to the joint selection of data split over the existing RANs and the adequate compression ratio. The state space is a hybrid (continuous and discrete) defined by two elements: the fraction of time that the PENs should use over a particular RAN and the PEN investigated in the current timestamp. The reward is a continuous function, which is defined in terms of: the fraction of data of PEN that will be transferred through RAN, the energy consumed by PEN to transfer bits over RAN, distortion, expected latency of RANs, the monetary cost of PENs to use RANs, the resource share, the fraction of time that the PENs should use over a particular, and the data rate. Simulation results demonstrate that their proposed scheme outperforms the greedy techniques in terms of minimizing energy consumption and latency while satisfying different PENs requirements.