As large language models (LLMs) grow in size and complexity, maximizing inference throughput while minimizing latency remains a critical challenge for enterprise production deployments. Speculative decoding is one effective strategy to address this, utilizing a lightweight draft model to guess future tokens which are then verified by the target LLM in a single forward pass. While state-of-the-art frameworks like Extrapolation Algorithm for Greater Language-model Efficiency (EAGLE) have achieved impressive speedups, they encounter a hidden architectural ceiling: their draft tokens are generated autoregressively. Because each draft token depends on the output of the previous one, producing K candidates requires K sequential forward passes through the draft head, creating a latency cost that grows linearly with speculation depth. EAGLE-3, the latest iteration, improved upon earlier versions by predicting tokens directly rather than features and by combining representations from multiple layers of the target model, boosting draft accuracy and allowing the method to benefit from larger training datasets. However, even with these gains, the fundamental sequential drafting constraint remains. The deeper you speculate, the more drafting overhead you accumulate, eventually eating into your performance gains.

To overcome this bottleneck, AWS invented Parallel-EAGLE (P-EAGLE) and contributed it to open source, a breakthrough method that transforms speculative decoding from an iterative process into a fully parallelized operation. P-EAGLE completely eliminates the nested sequential drafting phase by predicting all speculative draft tokens simultaneously in a single forward pass. To illustrate: if the target model generates the token “Paris,” EAGLE needs four sequential drafter passes to propose the next four tokens (“, known for its”). P-EAGLE instead fills positions 2–4 with learnable placeholders and predicts all four tokens at once (see Figure in Solution Overview). By decoupling the draft token count from the number of sequential forward passes, P-EAGLE allows for deeper speculation without scaling up latency overhead. On real-world benchmarks running on advanced high-performance hardware, this highly parallelized approach delivers up to a 1.69x throughput speedup over vanilla EAGLE frameworks.

Today, Amazon SageMaker JumpStart now natively supports P-EAGLE for an array of popular foundation models. SageMaker JumpStart provides a curated hub of state-of-the-art open-weight models that can be deployed with a single click or a few lines of code. By combining the model optimization of P-EAGLE with the fully managed environment of Amazon SageMaker AI, developers can now deploy P-EAGLE-accelerated inference endpoints that are up to 1.69x faster than EAGLE-3, without managing complex underlying CUDA kernels or distributed serving setups.

This post walks you through how to use P-EAGLE directly within Amazon SageMaker AI. It will demonstrate how to select a compatible model from the SageMaker JumpStart catalog, configure the parallel drafting specifications, and deploy a highly optimized real-time SageMaker AI endpoint to accelerate your generative AI applications.

The following benchmarks compare P-EAGLE, EAGLE-3, and standard inference (no speculation) on Qwen3-Coder-30B-A3B-Instruct running on NVIDIA B200 GPUs with FP8 quantization. Results are measured in estimated total output tokens per second (OTPS).

Output tokens per second comparison across concurrency levels. P-EAGLE (best K) consistently outperforms EAGLE-3 and baseline across both benchmarks.