In the study of time evolution of the parameters in Deep Learning systems, subject to optimization via SGD (stochastic gradient descent), temperature, entropy and other thermodynamic notions are commonly employed to exploit the Boltzmann formalism. We show that, in simulations on popular databases (CIFAR10, MNIST), such simplified models appear inadequate: different regions in the parameter space exhibit significantly different temperatures and no elementary function expresses the temperature in terms of learning rate and batch size, as commonly assumed. This suggests a more conceptual approach involving contact dynamics and Lie Group Thermodynamics.

On the Thermodynamic Interpretation of Deep Learning Systems / Fioresi, R.; Faglioni, F.; Morri, F.; Squadrani, L.. - 12829:(2021), pp. 909-917. (Intervento presentato al convegno 5th International Conference on Geometric Science of Information, GSI 2021 tenutosi a fra nel 2021) [10.1007/978-3-030-80209-7_97].

On the Thermodynamic Interpretation of Deep Learning Systems

Faglioni F.;
2021

Abstract

In the study of time evolution of the parameters in Deep Learning systems, subject to optimization via SGD (stochastic gradient descent), temperature, entropy and other thermodynamic notions are commonly employed to exploit the Boltzmann formalism. We show that, in simulations on popular databases (CIFAR10, MNIST), such simplified models appear inadequate: different regions in the parameter space exhibit significantly different temperatures and no elementary function expresses the temperature in terms of learning rate and batch size, as commonly assumed. This suggests a more conceptual approach involving contact dynamics and Lie Group Thermodynamics.
2021
5th International Conference on Geometric Science of Information, GSI 2021
fra
2021
12829
909
917
Fioresi, R.; Faglioni, F.; Morri, F.; Squadrani, L.
On the Thermodynamic Interpretation of Deep Learning Systems / Fioresi, R.; Faglioni, F.; Morri, F.; Squadrani, L.. - 12829:(2021), pp. 909-917. (Intervento presentato al convegno 5th International Conference on Geometric Science of Information, GSI 2021 tenutosi a fra nel 2021) [10.1007/978-3-030-80209-7_97].
File in questo prodotto:
File Dimensione Formato  
Articolo.pdf

Accesso riservato

Descrizione: Articolo
Tipologia: Versione pubblicata dall'editore
Dimensione 2.09 MB
Formato Adobe PDF
2.09 MB Adobe PDF   Visualizza/Apri   Richiedi una copia
Pubblicazioni consigliate

Licenza Creative Commons
I metadati presenti in IRIS UNIMORE sono rilasciati con licenza Creative Commons CC0 1.0 Universal, mentre i file delle pubblicazioni sono rilasciati con licenza Attribuzione 4.0 Internazionale (CC BY 4.0), salvo diversa indicazione.
In caso di violazione di copyright, contattare Supporto Iris

Utilizza questo identificativo per citare o creare un link a questo documento: https://hdl.handle.net/11380/1254442
Citazioni
  • ???jsp.display-item.citation.pmc??? ND
  • Scopus 1
  • ???jsp.display-item.citation.isi??? 0
social impact