In this paper we study the properties of the quenched pressure of a multi-layer spin-glass model (a deep Boltzmann Machine in artificial intelligence jargon) whose pairwise interactions are allowed between spins lying in adjacent layers and not inside the same layer nor among layers at distance larger than one. We prove a theorem that bounds the quenched pressure of such a K-layer machine in terms of K Sherrington–Kirkpatrick spin glasses and use it to investigate its annealed region. The replica-symmetric approximation of the quenched pressure is identified and its relation to the annealed one is considered. The paper also presents some observation on the model’s architectural structure related to machine learning. Since escaping the annealed region is mandatory for a meaningful training, by squeezing such region we obtain thermodynamical constraints on the form factors. Remarkably, its optimal escape is achieved by requiring the last layer to scale sub-linearly in the network size.

Annealing and Replica-Symmetry in Deep Boltzmann Machines

Alberici D.;
2020

Abstract

In this paper we study the properties of the quenched pressure of a multi-layer spin-glass model (a deep Boltzmann Machine in artificial intelligence jargon) whose pairwise interactions are allowed between spins lying in adjacent layers and not inside the same layer nor among layers at distance larger than one. We prove a theorem that bounds the quenched pressure of such a K-layer machine in terms of K Sherrington–Kirkpatrick spin glasses and use it to investigate its annealed region. The replica-symmetric approximation of the quenched pressure is identified and its relation to the annealed one is considered. The paper also presents some observation on the model’s architectural structure related to machine learning. Since escaping the annealed region is mandatory for a meaningful training, by squeezing such region we obtain thermodynamical constraints on the form factors. Remarkably, its optimal escape is achieved by requiring the last layer to scale sub-linearly in the network size.
File in questo prodotto:
File Dimensione Formato  
12 Alberici, Barra, Contucci, Mingione - JSP20 - Annealing and replica-symmetry in deep Boltzmann machines.pdf

non disponibili

Tipologia: Documento in Versione Editoriale
Licenza: Creative commons
Dimensione 269.81 kB
Formato Adobe PDF
269.81 kB Adobe PDF   Visualizza/Apri   Richiedi una copia

I documenti in IRIS sono protetti da copyright e tutti i diritti sono riservati, salvo diversa indicazione.

Utilizza questo identificativo per citare o creare un link a questo documento: http://hdl.handle.net/11697/176487
Citazioni
  • ???jsp.display-item.citation.pmc??? ND
  • Scopus 13
  • ???jsp.display-item.citation.isi??? ND
social impact