Cross-technology co-optimization of the memory sub-system for Multi-Agentic LLM Serving

The compute and memory requirements for agentic AI systems are growing rapidly. Production deployments increasingly feature heterogeneous model ensembles—large LLMs for planning and reasoning alongside smaller task-specific models—all serving multiple concurrent users. In these multi-agentic serving systems, the KV cache dominates memory consumption, growing linearly with context length, model depth, and batch size. With agentic workflows exhibiting input-to-output token ratios exceeding 100:1 and cached-token costs being 10× lower than uncached tokens in production APIs, efficient memory management is the single most critical factor for both latency and cost.

Current inference frameworks employ sophisticated software optimizations—paged attention, automatic prefix caching, and LRU eviction—but treat the underlying memory as a uniform resource. This technology-agnostic approach misses a fundamental co-optimization opportunity: different memory technologies (high-density DRAM, 3D-stacked memories, ternary content-addressable memories) offer distinct characteristics in bandwidth, latency, density, and associative lookup capability that can be exploited to dramatically improve agentic serving efficiency.

Possible Research Phases

This proposal presents three broad research lanes for System-Technology Co-Optimization (STCO) of memory subsystems tailored to multi-agentic LLM serving:

P1 Memory Characterization of Multi-Agentic Workloads: This phase will focus on profiling and modelling the memory access patterns, capacity demands, and temporal locality of KV cache operations across diverse agentic serving scenarios. This characterization will establish technology requirements (capacity, bandwidth, associativity) that drive next phase. 

P2: Technology-Aware Memory Subsystem Design: The goal of this phase would be to investigate the properties of memory technology characteristics that match the workload demands and develop an innovative memory subsystem, considering multiple compute units within a single node. For example, the requirement for prefix caching memory would be significantly different from the requirement of serving model weights. The memory subsystem would have to be modelled by means of frameworks such as Gem5, SST, AstraSim, etc.

WP3: Evaluation on Synthetic and Trace-Driven Workloads: The proposed memory subsystem from phase 2 would be evaluated using (a) synthetic multi-tenant workloads modeling realistic agentic scenarios (multiple enterprise customers with distinct system prompts, concurrent users, and heterogeneous agent types) and (b) trace-driven simulations from production-representative agentic workloads. Key metrics include time-to-first-token (TTFT), throughput, memory utilization efficiency, and cost-per-served-token compared to baseline uniform-memory designs.[DB4] 

Expected Contributions

1. Comprehensive memory-technology characterization of multi-agentic LLM serving workloads

2. Novel TCAM-accelerated prefix cache index architecture reducing lookup latency and enabling scalable cache management[FG5] 

3. Open evaluation framework with relevant benchmarks