ENGRAM
// Conditional Memory via Scalable Lookup: A New Axis of Sparsity for Large Language Models
Engram
Conditional Memory via Scalable Lookup: A New Axis of Sparsity for Large Language Models
13EmergingUnknown
What it does
This repository contains the official implementation for the paper: Conditional Memory via Scalable Lookup: A New Axis of Sparsity for Large Language Models. > Abstract: While Mixture-of-Experts (MoE) scales capacity via conditional computation, Transformers lack a native primitive for knowledge lookup. To address this, we explore conditional memory as a complementary sparsity axis, instantiated
Getting Started
git
git clone https://github.com/deepseek-ai/Engram