Foundations of Unsupervised Deep Learning (Ruslan Salakhutdinov, CMU)

A type of neural network model that has shown recent successes in generating remarkable images.

A class of tractable probabilistic models that have shown recent successes in generating remarkable images.

Graphical models with stochastic binary visible and hidden variables, used to learn latent representations and modeling complex data like images and documents.

An extension of Restricted Boltzmann Machines that can model more complicated data through deeper architectures.

A problem formulation that arises when solving for coefficients in sparse coding given fixed bases, with many available solvers.

A common practitioner's tool for dimensionality reduction. Autoencoders can be seen as nonlinear extensions of PCA, particularly when the hidden layer is linear.

A technique mentioned in the context of text representation, potentially used to initialize models or sum word representations for input into simpler networks.

A text representation method mentioned as an alternative to bidirectional GRUs for embedding documents into a semantic space.

Models mentioned as a comparison point for unsupervised learning techniques, particularly in image classification.

A model used to extract latent codes for representation learning, completely in an unsupervised way. It serves as a dimensionality reduction technique and can be seen as a nonlinear extension of PCA.

Intractable probabilistic models used in unsupervised learning, forming a basis for more complex architectures.

A subclass of Helmholtz machines that have seen significant development and are used for learning latent representations, particularly in generative modeling.

A class of models that learns to generate data without explicitly specifying the density, by playing a game between a generator and a discriminator.

A type of generative model that works pixel by pixel, capable of generating remarkable images, although its representational power for other tasks is still under investigation.

A preferred method for text embedding in semantic spaces, capable of capturing context from both past and future words in a sequence.

Models developed in 1995 with a generative process and an approximate inference step, which initially struggled to work but have seen recent improvements.

A data preprocessing technique that can sometimes help in training models like VAEs, although not always necessary.

A specific type of Helmholtz machine that defines a generative process through cascades of stochastic layers, capable of modeling complex nonlinear relationships.

3156 Co-developer of Helmholtz Machines in 1995.

Mentioned in the context of work done at Stanford that is related to sparse coding and representation learning.

Co-developer of Helmholtz Machines in 1995.

Introduced the reparameterization trick in 2014, which significantly improved the training efficiency of Variational Autoencoders by enabling backpropagation through stochastic layers.

A method for building deep models by stacking layers and optimizing them sequentially, often useful when dealing with large amounts of unlabeled data and limited labeled data.

A mathematical principle that allows optimization of the variational lower bound, enabling learning in variational methods where direct likelihood optimization is intractable.

A clever algorithm developed by Hinton that approximates learning for Boltzmann Machines by running Markov chains for only one step, significantly improving efficiency over running to infinity.

An early algorithm associated with Helmholtz Machines that was found to not work effectively.

A common loss function used in VAEs that penalizes errors heavily, potentially leading to less sharp images compared to GANs.

A class of non-probabilistic models where data is represented as a sparse linear combination of bases. It was originally developed to explain early visual processing in the brain and is useful for feature representation.

A class of models within unsupervised learning, including both tractable (e.g., belief networks, autoregressive models) and intractable (e.g., Boltzmann machines, VAEs) types.

A model that combines an encoder and decoder with a sparsity constraint on the latent representation, similar to sparse coding but with an explicit encoder.

A technique for compressing data into a binary representation, enabling efficient searching through large databases. It's useful in computer vision for retrieving images quickly.

A distribution used as a conditional probability in models dealing with count data, such as documents, similar to what is seen in previous classes for predicting possible words.

Used in variational learning to measure the difference between an approximating distribution (recognition model) and the true posterior.

A key innovation that allows gradients to be computed through stochastic layers in VAEs by expressing the sampling process deterministically using an auxiliary variable, effectively separating the stochastic and deterministic parts.

Models discussed in relation to Restricted Boltzmann Machines, particularly concerning the estimation of the partition function and its computational complexity.

Mentioned for recent papers proposing sophisticated RNNs and convolutional models for pixel-based generative tasks.

Mentioned in the context of developing Variational Autoencoders that can generate nice-looking images.

Institution where work on sparse coding and representation learning was developed, particularly related to Andrew Ng's group.

Gated Recurrent Units, mentioned as a powerful choice for text representation, especially bidirectional GRUs, for embedding sentences or documents into a semantic space.

A dataset used as an example for clustering and visualization with unsupervised learning techniques, showing how models discover structure.