Novel Architectures

Emerging architectural innovations beyond transformer-based models.

Overview

Research into new model architectures that could overcome transformer limitations.

Key Areas

State Space Models

• Mamba and variants
• Linear attention mechanisms

Hybrid Architectures

• Combinations of different attention patterns
• Mixture of experts innovations

Recurrent Approaches

• RWKV
• RetNet