Coded computation can speed up distributed learning in the presence of straggling workers. Partial recovery of the gradient vector can further reduce the computation time at each iteration; however, this can result in biased estimators, which may slow down convergence, or even cause divergence. Estimator bias is particularly prevalent when the straggling behavior is correlated over time, which results in the gradient estimators being dominated by a few fast servers. To mitigate biased estimators, we design a timely dynamic encoding framework for partial recovery that includes an ordering operator that changes the codewords and computation orders at workers over time. To regulate the recovery frequencies, we adopt an age metric in the design of the dynamic encoding scheme. The proposed age-based scheme prioritizes the recovery of computations with relatively large age. We show through numerical results that the proposed dynamic encoding strategy increases the timeliness of the recovered computations, which, as a result, reduces the bias in model updates, and accelerates the convergence compared to conventional static partial recovery schemes.

Age-Based Coded Computation for Bias Reduction in Distributed Learning / Ozfatura, E.; Buyukates, B.; Gunduz, D.; Ulukus, S.. - (2020), pp. 1-6. (Intervento presentato al convegno 2020 IEEE Global Communications Conference, GLOBECOM 2020 tenutosi a twn nel 2020) [10.1109/GLOBECOM42002.2020.9322412].

Age-Based Coded Computation for Bias Reduction in Distributed Learning

Gunduz D.;
2020

Abstract

Coded computation can speed up distributed learning in the presence of straggling workers. Partial recovery of the gradient vector can further reduce the computation time at each iteration; however, this can result in biased estimators, which may slow down convergence, or even cause divergence. Estimator bias is particularly prevalent when the straggling behavior is correlated over time, which results in the gradient estimators being dominated by a few fast servers. To mitigate biased estimators, we design a timely dynamic encoding framework for partial recovery that includes an ordering operator that changes the codewords and computation orders at workers over time. To regulate the recovery frequencies, we adopt an age metric in the design of the dynamic encoding scheme. The proposed age-based scheme prioritizes the recovery of computations with relatively large age. We show through numerical results that the proposed dynamic encoding strategy increases the timeliness of the recovered computations, which, as a result, reduces the bias in model updates, and accelerates the convergence compared to conventional static partial recovery schemes.
2020
2020 IEEE Global Communications Conference, GLOBECOM 2020
twn
2020
1
6
Ozfatura, E.; Buyukates, B.; Gunduz, D.; Ulukus, S.
Age-Based Coded Computation for Bias Reduction in Distributed Learning / Ozfatura, E.; Buyukates, B.; Gunduz, D.; Ulukus, S.. - (2020), pp. 1-6. (Intervento presentato al convegno 2020 IEEE Global Communications Conference, GLOBECOM 2020 tenutosi a twn nel 2020) [10.1109/GLOBECOM42002.2020.9322412].
File in questo prodotto:
File Dimensione Formato  
Age-Based_Coded_Computation_for_Bias_Reduction_in_Distributed_Learning.pdf

Accesso riservato

Tipologia: Versione pubblicata dall'editore
Dimensione 699.14 kB
Formato Adobe PDF
699.14 kB Adobe PDF   Visualizza/Apri   Richiedi una copia
Pubblicazioni consigliate

Licenza Creative Commons
I metadati presenti in IRIS UNIMORE sono rilasciati con licenza Creative Commons CC0 1.0 Universal, mentre i file delle pubblicazioni sono rilasciati con licenza Attribuzione 4.0 Internazionale (CC BY 4.0), salvo diversa indicazione.
In caso di violazione di copyright, contattare Supporto Iris

Utilizza questo identificativo per citare o creare un link a questo documento: https://hdl.handle.net/11380/1247332
Citazioni
  • ???jsp.display-item.citation.pmc??? ND
  • Scopus 24
  • ???jsp.display-item.citation.isi??? 20
social impact