Continual Learning has inspired a plethora of approaches and evaluation settings; however, the majority of them overlooks the properties of a practical scenario, where the data stream cannot be shaped as a sequence of tasks and offline training is not viable. We work towards General Continual Learning (GCL), where task boundaries blur and the domain and class distributions shift either gradually or suddenly. We address it through mixing rehearsal with knowledge distillation and regularization; our simple baseline, Dark Experience Replay, matches the network's logits sampled throughout the optimization trajectory, thus promoting consistency with its past. By conducting an extensive analysis on both standard benchmarks and a novel GCL evaluation setting (MNIST-360), we show that such a seemingly simple baseline outperforms consolidated approaches and leverages limited resources. We further explore the generalization capabilities of our objective, showing its regularization being beneficial beyond mere performance.

Dark Experience for General Continual Learning: a Strong, Simple Baseline / Buzzega, Pietro; Boschini, Matteo; Porrello, Angelo; Abati, Davide; Calderara, Simone. - (2020). (Intervento presentato al convegno 34th Conference on Neural Information Processing Systems (NeurIPS 2020) tenutosi a Vancouver, Canada nel 6-12 December 2020).

Dark Experience for General Continual Learning: a Strong, Simple Baseline

Pietro Buzzega;Matteo Boschini;Angelo Porrello;Davide Abati;Simone Calderara
2020

Abstract

Continual Learning has inspired a plethora of approaches and evaluation settings; however, the majority of them overlooks the properties of a practical scenario, where the data stream cannot be shaped as a sequence of tasks and offline training is not viable. We work towards General Continual Learning (GCL), where task boundaries blur and the domain and class distributions shift either gradually or suddenly. We address it through mixing rehearsal with knowledge distillation and regularization; our simple baseline, Dark Experience Replay, matches the network's logits sampled throughout the optimization trajectory, thus promoting consistency with its past. By conducting an extensive analysis on both standard benchmarks and a novel GCL evaluation setting (MNIST-360), we show that such a seemingly simple baseline outperforms consolidated approaches and leverages limited resources. We further explore the generalization capabilities of our objective, showing its regularization being beneficial beyond mere performance.
2020
34th Conference on Neural Information Processing Systems (NeurIPS 2020)
Vancouver, Canada
6-12 December 2020
Buzzega, Pietro; Boschini, Matteo; Porrello, Angelo; Abati, Davide; Calderara, Simone
Dark Experience for General Continual Learning: a Strong, Simple Baseline / Buzzega, Pietro; Boschini, Matteo; Porrello, Angelo; Abati, Davide; Calderara, Simone. - (2020). (Intervento presentato al convegno 34th Conference on Neural Information Processing Systems (NeurIPS 2020) tenutosi a Vancouver, Canada nel 6-12 December 2020).
File in questo prodotto:
Non ci sono file associati a questo prodotto.
Pubblicazioni consigliate

Licenza Creative Commons
I metadati presenti in IRIS UNIMORE sono rilasciati con licenza Creative Commons CC0 1.0 Universal, mentre i file delle pubblicazioni sono rilasciati con licenza Attribuzione 4.0 Internazionale (CC BY 4.0), salvo diversa indicazione.
In caso di violazione di copyright, contattare Supporto Iris

Utilizza questo identificativo per citare o creare un link a questo documento: https://hdl.handle.net/11380/1211823
Citazioni
  • ???jsp.display-item.citation.pmc??? ND
  • Scopus 233
  • ???jsp.display-item.citation.isi??? ND
social impact