Sequence to Sequence Deep Learning (Quoc Le, Google)

Early models used a 'bag-of-words' approach, converting text into fixed-size vectors by counting word occurrences. This method, however, lost information about word order, leading to limitations in understanding sentence structure.

RNNs can process input sequences while preserving ordering information. They use hidden states that are updated at each step, allowing the model to maintain context and dependencies between sequential data points.

The encoder-decoder architecture consists of an encoder RNN that processes the input sequence into a fixed-length context vector, and a decoder RNN that uses this vector to generate the output sequence, one element at a time.

Beam search is a more sophisticated decoding strategy than greedy decoding. Instead of choosing the single most likely word at each step, it keeps track of multiple high-probability candidate sequences (beams), exploring more possibilities to find a better overall output.

The attention mechanism allows the decoder to dynamically focus on different parts of the input sequence when generating each output element. It calculates weights for each input hidden state, creating a context vector that emphasizes the most relevant information for the current prediction.

Speech recognition often involves very long input sequences, making backpropagation difficult. Additionally, while sequence-to-sequence models with attention work well for translation, CTC (Connectionist Temporal Classification) has shown stronger performance in speech recognition.

Memory networks enhance RNNs by providing an explicit memory component. The model can read from and write to this memory, allowing it to store and retrieve information over longer contexts, which is beneficial for tasks like question answering.

Neural programmers augment neural networks with explicit operations like addition, subtraction, and stack manipulation. This allows them to learn and execute algorithms, making them suitable for tasks requiring symbolic reasoning or computation.

Techniques include pre-training word vectors on large language models, reducing vocabulary size, or using dropout for regularization. Multi-task learning, where related tasks share data and parameters, can also improve performance with less data for each task.