Large language models are increasingly used in settings that require reasoning over long documents, structured inputs, and multiple sources of evidence. In Transformer-based models, however, longer contexts remain limited by three coupled bottlenecks: the quadratic cost of self-attention during prefill, the linear growth of the Key-Value (KV) cache during generation, and the degradation of reasoning quality when relevant evidence is distributed across long or heterogeneous inputs. This thesis studies these limitations through three complementary principles: compression, amortization, and decomposition.
First, the thesis addresses the memory cost of long-context inference through query-aware KV cache compression. It introduces Finch, a training-free method that uses the model's own attention patterns to retain the cache entries most relevant to a given query. The same attention-guided principle is then extended from natural-language contexts to structured tabular inputs, showing that KV cache compression can preserve both textual and structural information in linearized tables.
Second, the thesis studies how to amortize the cost of compression when several queries are issued over the same corpus. It introduces CacheNotes, a task-aware offline compression method that constructs reusable compressed KV caches from a corpus and a natural-language task description. This shifts part of the processing cost away from query time while preserving task-relevant information for later use.
Third, the thesis investigates multi-document reasoning without concatenating all evidence into a single shared context. It introduces PCED, Parallel Context-of-Experts Decoding, which encodes retrieved documents independently and combines their contributions during generation through retrieval-aware contrastive decoding. This decomposed formulation reduces cross-document interference and improves the efficiency of evidence aggregation.
Together, these contributions show that long-context reasoning can be made more efficient not only by extending context windows, but also by selecting, reusing, and modularly combining contextual representations.
