On the Convergence of Decentralized Adaptive Gradient Methods

Decentralized optimization. Traditional decentralized optimization methods include well-know algorithms such as ADMM (Boyd et al., 2011), Dual Averaging (Duchi et al., 2012), Distributed Subgradient Descent (Nedic and Ozdaglar, 2009). More recent algorithms include Extra (Shi et al., 2015), Next (Lorenzo and Scutari, 2016), Prox-PDA (Hong et al., 2017), GNSD (Lu et al., 2019), and Choco-SGD (Koloskova et al., 2019). While these algorithms are commonly used in applications other than deep learning, recent algorithmic advances in the machine learning community have shown that decentralized optimization can also be useful for training deep models such as neural networks. Lian et al. (2017) demonstrate that a stochastic version of Decentralized Subgradient Descent can outperform parameter server-based algorithms when the communication cost is high. Tang et al. (2018) propose the D² algorithm improving the convergence rate over Stochastic Subgradient Descent. Assran et al. (2019) propose the Stochastic Gradient Push that is more robust to network failures for training neural networks. The study of decentralized training algorithms in the machine learning community is only at its initial stage. No existing work, to our knowledge, has seriously considered integrating adaptive gradient methods in the setting of decentralized learning. One noteworthy work (Nazari et al., 2019) proposes a decentralized version of AMSGrad (Reddi et al., 2018) and it is proven to satisfy some non-standard regret.

In distributed optimization (with $N$ nodes), we aim at solving the following problem

where $x$ is the vector of parameters and $f_{i}$ is only accessible by the $i$-th node. Through the prism of empirical risk minimization procedures, $f_{i}$ can be viewed as the average loss of the data samples located at node $i$, for $i\in[N]$. Throughout the paper, we make the following mild assumptions required for analyzing the convergence behavior of the different decentralized optimization algorithms introduced above:

Assumptions A1 and A3 are standard in the distributed optimization literature. A2 is slightly stronger than the traditional assumption stating that the estimator has bounded variance, yet, it is commonly used for the analysis of adaptive gradient methods (Chen et al., 2019; Ward et al., 2019). Note that the bounded gradient estimator assumption A2 implies the bounded variance assumption A3. In decentralized optimization, the nodes are connected as a graph and each node only communicates to its neighbors. Hence, one usually constructs an $N\times N$ matrix $W$ for information sharing when designing new training algorithms. We denote by $\lambda_{i}$ its $i$-th largest eigenvalue and define $\lambda\triangleq\max(|\lambda_{2}|,|\lambda_{N}|)$. The matrix $W$ cannot be arbitrary, its required key properties are listed in the following assumption:

We now present the convergence failure of current decentralized adaptive method before introducing our general framework for decentralized adaptive gradient methods.

Recently, Nazari et al. (2019) initiated an attempt to bring adaptive gradient methods into decentralized optimization with Decentralized ADAM (DADAM), shown in Algorithm 1. DADAM is essentially a decentralized version of ADAM and the key modification is the use of a consensus step on the optimization variable $x$ to transmit information across the network, encouraging its convergence.

The matrix $W$ is a doubly stochastic matrix (which satisfies A4) employed for achieving average consensus of $x$. Introducing such mixing matrix is standard while designing the extension of an algorithm to its decentralized variant, such as distributed gradient descent (Nedic and Ozdaglar, 2009; Yuan et al., 2016). It is proven in Nazari et al. (2019) that DADAM admits a non-standard regret bound in the online setting. Nevertheless, whether the algorithm can converge to stationary points in standard offline settings such training neural networks is still unknown. The next theorem shows that DADAM may fail to converge in the offline settings.

Theorem 1 shows that, even though DADAM is proven to satisfy some regret bounds (Nazari et al., 2019), it can fail to converge to stationary points in the nonconvex offline setting (common for training neural networks). We conjecture that this inconsistency in the convergence behavior of DADAM is due to the definition of the regret in Nazari et al. (2019). We want to remark that this is not the first time adaptive gradient methods are found to be divergent. For example, Reddi et al. (2018) constructs examples showing that ADAM is divergent and Chen et al. (2020) exhibits a naive application of adaptive gradient methods under the federated learning settings that can potentially fail to converge. All these examples contribute to our motivation to rigorously study the convergence of adaptive gradient methods in the decentralized setting. The next section presents decentralized adaptive gradient methods that are guaranteed to converge to stationary points under assumptions and provide a characterization of that convergence in finite-time and independently of the initialization.

The divergent example provided in the previous section implies that one should synchronize the adaptive learning rates on different nodes. This can easily be achieved in the parameter server setting where all the nodes are sending their gradients to a central server at each iteration. The parameter server can then exploit the received gradients to maintain a sequence of synchronized adaptive learning rates when updating the parameters, see Reddi et al. (2021) for further details. However, in our decentralized setting, every node can only communicate with its neighbors and such central server does not exist. Under that setting, the information for updating the adaptive learning rates can only be shared locally instead of broadcasted over the whole network. This makes it impossible to obtain, in a single iteration, a synchronized adaptive learning rate update using all the information in the network.

Systemic Approach: On a systemic level, one way to alleviate this bottleneck is to design communication protocols in order to give each node access to the same aggregated gradients over the whole network, at least periodically if not at every iteration. Therefore, the nodes can update their individual adaptive learning rates based on the same shared information. However, such solution may introduce an extra communication cost since it involves broadcasting the information over the whole network.

Algorithmic Approach: Our contributions being on an algorithmic level, another way to solve the aforementioned problem is by letting the sequences of adaptive learning rates, present on different nodes, to gradually consent, through the iterations. Intuitively, if the adaptive learning rates can consent fast enough, the difference among the adaptive learning rates on different nodes will not affect the convergence behavior of the algorithm. Consequently, no extra communication costs need to be introduced. We now develop this exact idea within the existing adaptive methods stressing on the need for a relatively low-cost and easy-to-implement consensus of adaptive learning rates.

Below is main archetype of the adaptive rates consensus mechanism within a decentralized framework that we propose in this paper.

While each node can have different $\hat{v}_{t,i}$ in DADAM (Algorithm 1), one can keep track of the min/max/average of these adaptive learning rates and use that latter quantity as the new adaptive learning rate. The upstream definition of some convergent lower and upper bounds may also lead to a gradual synchronization of the adaptive learning rates on different nodes as developed for AdaBound in Luo et al. (2019).

In this paper, we present an algorithm framework for decentralized adaptive gradient methods as Algorithm 2, which uses average consensus of $\hat{v}_{t,i}$ (see consensus update in line 8 and 11) to help convergence. Algorithm 2 can become different adaptive gradient methods by specifying $r_{t}$ as different functions. E.g., when we choose $\hat{v}_{t,i}={\frac{1}{t}\sum_{k=1}^{t}g_{k,i}^{2}}$ , Algorithm 2 becomes a decentralized version of AdaGrad. When one chooses $\hat{v}_{t,i}$ to be the adaptive learning rate for AMSGrad, we get decentralized AMSGrad (Algorithm 3). The intuition of using average consensus is that for adaptive gradient methods such as AdaGrad or Adam, $\hat{v}_{t,i}$ approximates the second moment of the gradient estimator, the average of the estimations of those second moments from different nodes is an estimation of second moment on the whole network. Also, this design will not introduce any extra hyperparameters that can potentially complicate the tuning process ($\epsilon$ in line 9 is important for numerical stability as in vanilla Adam). The following result gives a finite-time convergence rate for our framework described in Algorithm 2.

In addition, one can specify $\alpha$ to show convergence in terms of $T$, $d$, and $N$. An immediate result, shown in Corollary 2.1, is by setting $\alpha=\sqrt{N}/\sqrt{Td}$:

Corollary 2.1 indicates that if $\mathbb{E}[\mathcal{V}_{T}]=o(T)$ and $\bar{U}_{t}$ is bounded from above, then Algorithm 2 is guaranteed to converge to stationary points of the loss function. Intuitively, this means that if the adaptive learning rates on different nodes do not change too fast, the algorithm can converge. In convergence analysis, the term $\mathbb{E}[\mathcal{V}_{T}]$ upper bounds the total bias in update direction caused by the correlation between $m_{t,i}$ and $\hat{v}_{t,i}$. It is shown in Chen et al. (2019) that when $N=1$, $\mathbb{E}[\mathcal{V}_{T}]=\tilde{O}(d)$ for AdaGrad and AMSGrad. Besides, $\mathbb{E}[\mathcal{V}_{T}]=\tilde{O}(Td)$ for Adam which do not converge. Later, we will show convergence of decentralized versions of AMSGrad and AdaGrad by bounding this term as $O(Nd)$ and $O(Nd\log(T))$, respectively. Corollary 2.1 also conveys the benefits of using more nodes in the graph employed. When $T$ is large enough such that the term $O(\sqrt{d}/\sqrt{TN})$ dominates the right hand side of (3), then linear speedup can be achieved by increasing the number of nodes $N$.

Another point worth discussion is the choice of $W$ since the convergence rate depends on $\lambda$ which is depedent on $W$. A common way to set $W$ for undirected graph is the maximum-degree method (MDM) in Boyd et al. (2004). Denote $d_{i}$ as degree of vertex $i$ and $d_{\max}=\max_{i}d_{i}$, MDM sets $W_{i,i}=1-d_{i}/d_{\max}$, $W_{i,j}=1/d_{\max}$ if $i\neq j$ and $(i,j)$ is an edge, and $W_{i,j}=0$ otherwise. This $W$ ensures Assumption A4 for many common connected graph types, so does the variant $\gamma I+(1-\gamma)W$ for any $\gamma\in[0,1)$. A more refined choice of $W$ coupled with a comprehensive discussion on $\lambda$ in our Theorem 2 can be found in Boyd et al. (2009), e.g., $1-\lambda=O(1/N^{2})$ for cycle graphs, $1-\lambda=O(1/\log(N))$ for hypercube graphs, $\lambda=0$ for fully connected graph. Intuitively, $\lambda$ can be close to 1 for sparse graphs and to 0 for dense graphs. This is consistent (2), whose RHS is large for $\lambda$ close to 1 and small for $\lambda$ close to 0 since average consensus on sparser graphs is expected to take longer time.

We now present, in Algorithm 3, a notable special case of our algorithmic framework, namely Decentralized AMSGrad, which is a decentralized variant of AMSGrad. Compared with DADAM, the above algorithm exhibits a dynamic average consensus mechanism to keep track of the average of $\{\hat{v}_{t,i}\}_{i=1}^{N}$, stored as $\tilde{u}_{t,i}$ on $i$-th node, and uses $u_{t,i}:=\max(\tilde{u}_{t,i},\epsilon)$ for updating the adaptive learning rate for $i$-th node. As the number of iteration grows, even though $\hat{v}_{t,i}$ on different nodes can converge to different constants, the $u_{t,i}$ will converge to the same number $\lim\limits_{t\rightarrow\infty}\frac{1}{N}\sum_{i=1}^{N}\hat{v}_{t,i}$ if the limit exists.

This average consensus mechanism enables the consensus of adaptive learning rates on different nodes, which accordingly guarantees the convergence of the method to stationary points. The consensus of adaptive learning rates is the key difference between decentralized AMSGrad and DADAM and is the reason why decentralized AMSGrad is convergent while DADAM is not.

One may notice that decentralized AMSGrad does not reduce to AMSGrad for $N=1$ since the quantity $u_{t,i}$ in line 10 is calculated based on $v_{t-1,i}$ instead of $v_{t,i}$. This design encourages the execution of gradient computation and communication in a parallel manner. Specifically, line 4-7 (line 4-6) in Algorithm 3 (Algorithm 2) can be executed in parallel with line 8-9 (line 7-8) to overlap communication and computation time. If $u_{t,i}$ depends on $v_{t,i}$ which in turn depends on $g_{t,i}$, the gradient computation must finish before the consensus step of the adaptive learning rate in line 9. This can slow down the running time per-iteration of the algorithm. To avoid such delayed adaptive learning, adding $\tilde{u}_{t-\frac{1}{2},i}=\tilde{u}_{t,i}-\hat{v}_{t-1,i}+\hat{v}_{t,i}$ before line 9 and getting rid of line 12 in Algorithm 2 is an option. Similar convergence guarantees will hold since one can easily modify our proof of Theorem 2 for such update rule. As stated above, Algorithm 3 converges, with the following rate:

Theorem 3 shows that Algorithm 3 converges with a rate of $\mathcal{O}(\sqrt{d}/\sqrt{T})$ when $T$ is large, which is the best known convergence rate under the given assumptions. Note that in some related works, SGD admits a convergence rate of $\mathcal{O}(1/\sqrt{T})$ without any dependence on the dimension of the problem. Such improved convergence rate is derived under the assumption that the gradient estimator have a bounded $L_{2}$ norm, which can thus hide a dependency of $\sqrt{d}$ in the final convergence rate. Another remark is the convergence measure can be converted to $\frac{1}{T}\sum_{t=1}^{T}\mathbb{E}\left[\left\|{\nabla f(\overline{X}_{t})}\right\|^{2}\right]$ using the fact that $\|\overline{U}_{t}\|_{\infty}\leq G_{\infty}^{2}$ (by update rule of Algorithm 3), for the ease of comparison with existing literature.

Step 1: Reparameterization. Similarly to Yan et al. (2018); Chen et al. (2019) with SGD (with momentum) and centralized adaptive gradient methods, define the following auxiliary sequence: $Z_{t}=\overline{X}_{t}+\frac{\beta_{1}}{1-\beta_{1}}(\overline{X}_{t}-\overline{X}_{t-1})\,,$ with $\overline{X}_{0}\triangleq\overline{X}_{1}$. Such an auxiliary sequence can help us deal with the bias brought by the momentum and simplifies the convergence analysis.

Step 2: Bounding gradient. With the help of $Z_{t}$, we can remove the complicated update dependence on $m_{t}$, and perform convergence analysis to bound gradient of $Z_{t}$. Then bound gradient of $\overline{X}_{t}$ by smoothness of gradient, which yields:

where $\Delta_{f}:=\mathbb{E}[f(Z_{1})]-\mathbb{E}[f(Z_{T+1})]$ $D_{1},D_{2}$ and $D_{3}$ are three terms, defined in Appendix B, and can be tightly bounded from above. We first bound $D_{3}$ using the following quantities of interest:

where $\lambda=\max(|\lambda_{2}|,|\lambda_{N}|)$ and recall that $\lambda_{i}$ is $i$-th largest eigenvalue of $W$.

Then, bounding $D_{1}$ and $D_{2}$ give rise to the terms related to $\mathbb{E}\left[\sum_{t=1}^{T}\|(-\hat{V}_{t-2}+\hat{V}_{t-1})\|_{abs}\right]$.

Step 3: Bounding the drift term variance. An important term that needs upper bounding in our proof is the variance of the gradients multiplied (element-wise) by the adaptive learning rate, $\mathbb{E}\left[\left\|\frac{1}{N}\sum_{i=1}^{N}\frac{g_{t,i}}{\sqrt{u_{t,i}}}\right\|^{2}\right]\leq\mathbb{E}[\|\Gamma_{u}^{f}\|^{2}]+\frac{d}{N}\frac{\sigma^{2}}{\epsilon}$, where $\Gamma_{u}^{f}:=1/N\sum_{i=1}^{N}\nabla f_{i}(x_{t,i})/\sqrt{u_{t,i}}$. We can then transform $\mathbb{E}[\|\Gamma_{u}^{f}\|^{2}]$ into $\mathbb{E}[\|\Gamma_{\overline{U}}^{f}\|^{2}]$ by splitting out two error terms, then bounding the error terms as operated for $D_{2}$ and $D_{3}$. Then, by plugging it into (4), we obtain the desired bound in Theorem 2.

In this section, we provide a decentralized version of AdaGrad (Duchi et al., 2011) (optionally with momentum) converted by Algorithm 2, further supporting the usefulness of our decentralization framework. The required modification for decentralized AdaGrad is to specify line 4 of Algorithm 2 as follows: $\hat{v}_{t,i}=\frac{t-1}{t}\hat{v}_{t-1,i}+\frac{1}{t}g_{t,i}^{2}$, which is equivalent to $\hat{v}_{t,i}={\frac{1}{t}\sum_{k=1}^{t}g_{k,i}^{2}}$. In this section, we call this algorithm decentralized AdaGrad.

The pseudo code of the algorithm is shown in Algorithm 4. There are two details in Algorithm 4 worth mentioning. The first one is that the introduced framework leverages momentum $m_{t,i}$ in updates, while original AdaGrad does not use momentum. The momentum can be turned off by setting $\beta_{1}=0$ and the convergence results will still hold. The other one is that in Decentralized AdaGrad, we use the average instead of the sum in the term $\hat{v}_{t,i}$. In other words, we write $\hat{v}_{t,i}={\frac{1}{t}\sum_{k=1}^{t}g_{k,i}^{2}}$. This latter point is different from the original AdaGrad which actually uses $\hat{v}_{t,i}={\sum_{k=1}^{t}g_{k,i}^{2}}$.

The reason is that in the original AdaGrad, a constant stepsize ($\alpha$ independent of $t$ or $T$) is used with $\hat{v}_{t,i}={\sum_{k=1}^{t}g_{k,i}^{2}}$. This is equivalent to using a well-known decreasing stepsize sequence $\alpha_{t}=\frac{1}{\sqrt{t}}$ with $\hat{v}_{t,i}={\frac{1}{t}\sum_{k=1}^{t}g_{k,i}^{2}}$. In our convergence analysis, which can be found below, we use a constant stepsize $\alpha=O(\frac{1}{\sqrt{T}})$ to replace the decreasing stepsize sequence $\alpha_{t}=O(\frac{1}{\sqrt{t}})$. Such a replacement is popularly used in Stochastic Gradient Descent analysis for the sake of simplicity and to achieve a better convergence rate. In addition, it is easy to modify our theoretical framework to include decreasing stepsize sequences such as $\alpha_{t}=O(\frac{1}{\sqrt{t}})$. The convergence analysis for decentralized AdaGrad is shown in Theorem 4.

We compare the training loss and testing accuracies of different D-PSGD, DADAM, and our proposed decentralized AMSGrad, with different stepsizes on heterogeneous data distribution. We use 5 nodes and the heterogeneous data distribution is created by assigning each node with data of only two labels. Note that there are no overlapping labels between different nodes.

We observe Figure2(a) and (d) that the stepsize $10^{-3}$ works best for D-PSGD in terms of test accuracy and $10^{-1}$ works best in terms of training loss. This difference is caused by the inconsistency among the model parameters on different nodes when the stepsize is large.

Figure 2(b) and (e) shows the performance of decentralized AMSGrad with different stepsizes. We see that its best performance is better than the one of D-PSGD and the performance is more stable (the test performance is less sensitive to stepsize tuning). As expected, the performance of DADAM is not as good as D-PSGD or decentralized AMSGrad, see Figure 2(c) and (f). Its divergence characteristic, highlighted Section 2.3, coupled with the heterogeneity in the data amplify its non-convergence issue in our experiments. From the experiments above, we can see the advantages of decentralized AMSGrad in terms of both performance and ease of parameter tuning, and the importance of ensuring the theoretical convergence of any newly proposed methods in the presented setting.