Beyond Prompt Caching: Architecting Codebase Perception

An exploration on how coding agents are optimizing the retrieval pipeline for high-fidelity code context

In my previous post on Prompt Caching and Context Optimization, I focused on the economics of the context window. But once you’ve optimised the cost of delivery, you hit a harder technical wall: retrieval quality.

One of the key engineering challenges has shifted from code generation to codebase perception—the mechanical process of isolating the specific 1% of a repository required for a task. While modern LLMs have massive context windows, saturating them with unrefined data leads to significant performance degradation, increased “lost-in-the-middle” reasoning errors, and bloated operational costs.

---

The Unit of Retrieval: Why Boundaries Matter

Before indexing, you must define the “unit” of code the AI will consume. This is the foundation of your retrieval quality.

• Window-based Chunking: Splitting code every N characters.

  • Pros: Computationally cheap; uniform vector sizes.

  • Cons: “Syntax-blind.” It frequently severs logical blocks, resulting in fragmented embeddings that lose the developer’s intent.

• Symbol-level Indexing: Using an AST (Abstract Syntax Tree) parser—like Tree-sitter—to identify logical boundaries.

  • Pros: You index “complete thoughts.” A chunk corresponds to a specific function, class, or interface.

  • Cons: Requires language-specific grammar support.

By using a parser to ensure the index is aware of these boundaries, we can capture the intent of the code before it is vectorized:

def get_nodes(tree, source_code):
    nodes = []
    # Targeted queries for specific logical blocks
    query = language.query("""
        (function_definition) @function
        (class_definition) @class
    """)
    captures = query.captures(tree.root_node)
    
    # Each capture represents a 'complete thought' for the index
    for node, tag in captures:
        nodes.append({
            "type": tag,
            "content": source_code[node.start_byte:node.end_byte]
        })
    return nodes

Example: Consider an auth_service.py file. A window-based chunker might split the validate_session function in half, leaving the LLM with the variable declarations but none of the actual validation logic. By using symbol-level indexing, the agent retrieves the entire validate_session function as a single, coherent “node,” ensuring the AI sees the complete logic of the security check.

---

Multi-Modal Indexing

A production agent doesn’t store code once; it creates multiple indices to support different query types:

1. Lexical (Keyword): A literal text index (e.g., BM25). This is non-negotiable for exact-match scenarios like searching for a specific error constant (ERR_AUTH_001).

2. Dense Retrieval (Semantic): Uses encoders to map code into a high-dimensional vector space. This allows the system to retrieve implementation logic based on functional intent.

3. Relational (Structural): A graph-based index that tracks which symbols are contained within which files and their upstream/downstream dependencies.

   

   Multi-modal indexing on top of AST retrieved components allows AI Coding Agents to support multiple ways to look-up code

Example: If you ask the agent to “fix the login error,” the system uses these indices in parallel:

• Lexical: It finds the exact string LOGIN_FAILURE_EXCEEDED in your constants file.

• Semantic: It finds a function named handle_user_login because the embedding recognizes the intent, even if the word “error” isn’t in the function name.

• Structural: It identifies that handle_user_login relies on the UserDatabase class.

---

The Retrieval Pipeline

The retrieval pipeline acts as the active bridge to the multi-modal index, querying these stores in concert to reconstruct the “reality” of the codebase.

1. Candidate Generation & Fusion

The system runs lexical and semantic searches in parallel. These disparate lists are merged using Reciprocal Rank Fusion (RRF)—a scale-independent algorithm that prioritizes documents appearing high in multiple search results. By merging the literal and the conceptual, the pipeline ensures that a function named process_payment (Lexical hit) and a snippet describing “transaction handling” (Semantic hit) both rise to the top.

2. The Reranking Layer

Fusion provides candidates, but reranking provides precision.

• Cross-Encoders: Unlike retrieval embeddings, a Cross-Encoder jointly evaluates the query and candidate snippet in the same attention pass. This helps the system distinguish between similar-looking helpers to find the one relevant to the current scope.

• Heuristic Scoring: Injecting “engineering common sense,” such as prioritizing results in the src/ directory over tests/ or ignoring node_modules.

3. Dependency Hopping: Relational Context

Context building often requires more than just the initial “hit.” If the top-ranked snippet is a function implementing a specific interface, the system must “hop” to that definition. By tapping into the Structural Index, the pipeline navigates the dependency graph:

• Contract Retrieval: If the code uses a specific type or interface, the system pulls in those definitions so the LLM has the “contract,” not just the implementation.

• Context Extension: If a retrieved snippet is a method, the pipeline looks at its parent class and neighboring methods to provide necessary scope.

Example: When the search hits a method called authenticate_user(token: JWTToken), the LLM needs to know what a JWTToken actually looks like. The pipeline sees this dependency in the Structural Index and automatically “hops” to the models/auth.py file to pull in the JWTToken interface. This prevents the agent from hallucinating properties on the token that don’t exist.

Retrieval Workflow to complement a multi-index setup for efficient code retrieval

---

Maintaining Index Parity

A static index is a liability in a live coding environment. However, re-indexing an entire 2,000-file repository every time a developer hits Cmd+S is computationally expensive and introduces unacceptable latency.

The solution is to move from file-watching to Symbol-Level Synchronization. Instead of treating a file as a single block of text, the system treats it as a container for individual symbols (functions, classes, methods).

Increment Re-indexing workflow to keep the indexes fresh and the context relevant

The Incrementality Logic

When a file change is detected, the engine doesn’t just re-embed the whole file. It follows a “diff and update” flow:

1. AST Re-parse: The system immediately generates a new Abstract Syntax Tree for the changed file.

2. Symbol Hashing: It extracts each symbol and computes a unique hash based on the implementation logic.

3. State Comparison:

   • Unchanged: If a function’s hash matches the version already in the index, the system only updates the line numbers or file path metadata.

   • Changed: If the hash differs, only that specific snippet is sent to the embedding model.

   • Removed: Any symbol IDs no longer present in the AST are evicted from the vector store to prevent “ghost” results.

This approach ensures the index remains near-real-time while maintaining a high Embedding Hit Rate, significantly reducing both API costs and the time it takes for the agent to “perceive” your latest changes.

def update_index(file_path):
    new_symbols = parse_file(file_path)
    for symbol in new_symbols:
        symbol_hash = calculate_hash(symbol.content)
        if symbol_hash != get_stored_hash(symbol.id):
            # Only re-embed if the implementation logic has changed
            embed_and_upsert(symbol)

Example: Imagine you refactor your AuthService by adding a single comment or changing one line in verify_password. Instead of re-uploading and re-embedding every function in the authentication module, the Delta Update strategy sees that only the verify_password symbol hash has changed. It updates that single vector, keeping your index near-real-time without a massive spike in API latency.

---

Final Thought

An emergent frontier of AI coding is now context engineering. By shifting from naive text-splitting to AST-aware, multi-modal retrieval, we move from an LLM that guesses based on partial context to one that reasons over a more faithful reconstruction of the codebase.

This article was heavily inspired by the engineering challenges being solved by the teams at the forefront of this space. If you want to go deeper into the low-level optimizations of codebase indexing and discovery, I highly recommend these technical deep-dives from the Cursor team:

• Fast Regex Search: On building a custom engine for sub-millisecond keyword lookups across massive repos.

• Secure Codebase Indexing: How to handle the privacy and security trade-offs of local vs. remote indexing.

• Dynamic Context Discovery: Exploring how agents decide what is “relevant” in real-time as you type.

• The Path to Semantic Search: A look at the evolution of embeddings in the context of large-scale repositories.

---

---

Explore the Implementation

If you want to see these concepts in action, I’ve built a proof-of-concept called ASTRAG to explore these concepts as I was going through them. It demonstrates how to move beyond naive chunking by using Tree-sitter for AST-aware symbol extraction and relationship mapping.

Check out the repository here: github.com/susheem-k/ASTRAG. There is a simple readme that should get you started.

---