In the study of time evolution of the parameters in Deep Learning systems, subject to optimization via SGD (stochastic gradient descent), temperature, entropy and other thermodynamic notions are commonly employed to exploit the Boltzmann formalism. We show that, in simulations on popular databases (CIFAR10, MNIST), such simplified models appear inadequate: different regions in the parameter space exhibit significantly different temperatures and no elementary function expresses the temperature in terms of learning rate and batch size, as commonly assumed. This suggests a more conceptual approach involving contact dynamics and Lie Group Thermodynamics.
On the Thermodynamic Interpretation of Deep Learning Systems / Fioresi, R.; Faglioni, F.; Morri, F.; Squadrani, L.. - 12829:(2021), pp. 909-917. (Intervento presentato al convegno 5th International Conference on Geometric Science of Information, GSI 2021 tenutosi a fra nel 2021) [10.1007/978-3-030-80209-7_97].
On the Thermodynamic Interpretation of Deep Learning Systems
Faglioni F.;
2021
Abstract
In the study of time evolution of the parameters in Deep Learning systems, subject to optimization via SGD (stochastic gradient descent), temperature, entropy and other thermodynamic notions are commonly employed to exploit the Boltzmann formalism. We show that, in simulations on popular databases (CIFAR10, MNIST), such simplified models appear inadequate: different regions in the parameter space exhibit significantly different temperatures and no elementary function expresses the temperature in terms of learning rate and batch size, as commonly assumed. This suggests a more conceptual approach involving contact dynamics and Lie Group Thermodynamics.File | Dimensione | Formato | |
---|---|---|---|
Articolo.pdf
Accesso riservato
Descrizione: Articolo
Tipologia:
Versione pubblicata dall'editore
Dimensione
2.09 MB
Formato
Adobe PDF
|
2.09 MB | Adobe PDF | Visualizza/Apri Richiedi una copia |
Pubblicazioni consigliate
I metadati presenti in IRIS UNIMORE sono rilasciati con licenza Creative Commons CC0 1.0 Universal, mentre i file delle pubblicazioni sono rilasciati con licenza Attribuzione 4.0 Internazionale (CC BY 4.0), salvo diversa indicazione.
In caso di violazione di copyright, contattare Supporto Iris