SGLang-Neuron Brings Open-Weight Serving to AWS Trainium
By @sharedot · · 7 pages
- AI Frontier
- Open Weight Models
- Inference Infrastructure
AWS and the SGLang team released SGLang-Neuron v0.1.0 on Nov 12, serving 17 open-weight models on Trainium chips with the same SGLang interface.
What happened: a first public release
On November 12, 2026, AWS announced SGLang-Neuron, a serving runtime built with the SGLang team at LMSYS and RadixArk in collaboration with Amazon's Annapurna Labs chip unit, letting developers run open-weight models and reinforcement-learning rollouts on Trainium accelerators.
The SGLang team calls v0.1.0 'the first public release of SGLang's backend for AWS Trainium,' following a closed beta announced in October. AWS says developers keep 'the same SGLang interface you use today,' accessed through official Deep Learning Containers, with the code open source on GitHub.
Why it is surprising
Frontier-class open-weight models have historically been hard to serve on non-GPU silicon, requiring custom compilation pipelines and code forks.
SGLang-Neuron instead uses a decoupled out-of-tree plugin on top of TorchNeuron: the SGLang core keeps request handling, OpenAI-compatible endpoints, dynamic batching and RadixAttention prefix caching, while the plugin handles Trainium-specific attention, mixture-of-experts, and the graph runner.
According to FourWeekMBA, neither AWS nor the SGLang team compares Trainium serving speed or cost with Nvidia GPUs — the release is about software reach, not chip speed claims.
The evidence: Kimi-K3 on three nodes
FourWeekMBA counts 17 models in the release post's tables: 12 language models including Kimi-K3, GLM-5.3, Qwen3.5, GPT-OSS-120B and DeepSeek-V2-Lite, plus 5 diffusion models including FLUX.1-dev.
The post calls Kimi-K3 'the largest deployment in this release,' running across three Trainium3 nodes — one for prefill and two for decode — while GLM-5.2 serves a 1M-token context.
The one throughput figure given is a cache setting: on GPT-OSS-120B at TP8 and 32K context, HiCache L2 delivers 72% to 75% more output throughput, a developer-measured number FourWeekMBA notes was not independently verified.
The stakes: cheaper inference, trust questions
The release lands amid GPU cost pressure for research computing. According to CASRAI, academic grants have been drained by H100 and B200 rental premiums, and Trainium instances offer lower cost-per-token economics for sustained inference.
On the market side, Morph's Nick Khami argues open models will reach frontier capability within months, making inference pricing and provable model identity newly urgent — his $34-per-GPU-hour DeepSeek benchmark against $12–15 actual GB300 rentals frames why cheap, verifiable open-weight serving matters.
OpenAI-compatible APIs also mean labs need not rewrite client code, per CASRAI.
What comes next
The SGLang team validated a full reinforcement-learning loop end to end on Qwen3-30B-A3B, with rollouts served by SGLang-Neuron and training on TorchTitan, and the runtime also supports PD disaggregation across multi-node clusters via NIXL transfers over 3.2 Tbps EFA networking, per CASRAI.
Watchers should note what remains unestablished, as FourWeekMBA flags: no GPU comparison, no Trainium pricing, and an unverified throughput figure.
With AWS planning to sell Trainium beyond its own cloud, the question now is whether builders shift serving stacks toward Trainium — and whether verifiable inference matures fast enough to keep pace.