Retrieval-augmented generation (RAG) has emerged as a popular method for enhancing large language models (LLMs) by connecting them to external knowledge sources. However, RAG introduces challenges such as latency from real-time retrieval, potential errors in document selection, and increased system complexity.

A new paper, released in December 2024 and already cited twice, proposes a novel alternative: cache-augmented generation (CAG). This approach leverages the extended context windows of modern LLMs to preload all relevant knowledge, eliminating the need for real-time retrieval.

How CAG Works

CAG involves three phases:

1. Preloading: A curated collection of documents relevant to the target application is preprocessed and loaded into the LLM's extended context window. The LLM then processes these documents, transforming them into a key-value (KV) cache. This cache, which encapsulates the model's inference state, is stored for future use. The KV cache is a crucial component of transformer-based LLMs, storing the intermediate activations of the model's attention layers. By precomputing the KV cache for the preloaded knowledge, CAG eliminates the need to recompute it during inference, resulting in significant speed improvements.
2. Inference: When a user submits a query, the precomputed KV cache is loaded alongside the query. The LLM utilizes this cached context to generate a response, eliminating retrieval latency and errors.
3. Cache Reset: The KV cache can be efficiently reset for subsequent inference sessions by truncating the tokens corresponding to previous queries. This ensures sustained speed and responsiveness across multiple interactions.

Advantages of CAG

• Reduced inference time: Eliminating real-time retrieval makes the process faster, leading to quicker responses.
• Unified context: Preloading all knowledge provides a holistic and coherent understanding, improving response quality and consistency.
• Simplified architecture: Removing the retriever simplifies the system and reduces development overhead.

When is CAG Useful?

CAG is particularly advantageous when:

• The knowledge base is of a limited and manageable size, such as internal company documentation, FAQs, or customer support logs.
• Real-time retrieval is impractical or undesirable due to latency constraints or limited network connectivity.
• A simpler, more efficient system is preferred, reducing development and maintenance overhead.

The Future of CAG

CAG is a new and evolving approach, but it holds great promise for the future of knowledge integration in LLMs. As LLMs continue to advance, with longer context windows and improved information extraction capabilities, CAG will become even more powerful and versatile.

In conclusion, CAG offers a compelling alternative to RAG, particularly for applications with manageable knowledge bases. By leveraging the capabilities of long-context LLMs and precomputed KV caches, CAG provides a faster, simpler, and more efficient way to integrate knowledge into language models. This new paradigm has the potential to revolutionize how we build and deploy knowledge-intensive LLM applications.

For the full paper, see below.

Disclaimer: Generative AI was used in-part to write this summary. All contributions are from the oritinal authors.