Accelerating LLM Inference with Speculative Decoding - LinkedIn Engineering Blog

In this LinkedIn Engineering Blog post, Dhyey Mavani and the Hiring Assistant team reveal how they leveraged speculative decoding—a technique that drafts multiple tokens ahead and verifies them in parallel—to dramatically accelerate LLM inference without sacrificing quality. The article provides a comprehensive technical breakdown of how n-gram speculation works, why it’s particularly effective for structured outputs like rubric-style summaries, and how LinkedIn achieved nearly 4× throughput gains and 66% P90 latency reduction. The team discusses key configuration parameters (num_speculative_tokens, prompt_lookup_max, prompt_lookup_min) and explains when n-gram speculation is most effective versus draft-model approaches. This post offers valuable insights for teams building real-time AI agents at scale, demonstrating how operational simplicity and performance can be balanced in production GenAI systems.