Optimizing Small LLMs for On-Device Inference: Overcoming Memory and Latency Bottlenecks on Edge Hardware
Practical techniques to run small LLMs on edge devices: quantization, memory layout, kernel optimizations, and runtime tricks to cut memory and latency.