Many people. Full context.

NVIDIA GB10, 128 GB unified memory. Memory figures are whole-host unless marked.

People at once × filled context

Room for the whole team.

Qwen3.6-35B-A3B · NVFP4 variant · NVIDIA NGC vLLM 0.29KTS-1 adaptive capacity · 25 measured · 5 limits confirmed · 18 above a limit

Qwen3.6-35B-A3B · NVFP4 variant · NVIDIA NGC vLLM 0.29. KTS-1 adaptive capacity grid. Measured cells give writing speed per person and the slowest first word. Other cells are labelled capacity limit confirmed, or above confirmed limit and not attempted, and carry no numbers. Each cell links to its run.
People at once	4Knew	40Knew	64Knew	128Knew	250Knew	250Kalready read
1	104.0tok/s each First word 2.1 s Measured · A Run →	93.7tok/s each First word 11.4 s Measured · A Run →	88.6tok/s each First word 19.9 s Measured · A Run →	73.5tok/s each First word 53.3 s Measured · A Run →	62.0tok/s each First word 2.6 min Measured · A Run →	61.5tok/s each First word 3.5 s Measured · A Run →
2	86.4tok/s each First word 2.9 s Measured · A Run →	73.6tok/s each First word 18.9 s Measured · A Run →	59.9tok/s each First word 34.3 s Measured · A Run →	45.2tok/s each First word 95.4 s Measured · A Run →	31.0tok/s each First word 4.8 min Measured · A Run →	50.0tok/s each First word 6.3 s Measured · A Run →
5	65.8tok/s each First word 3.9 s Measured · A Run →	49.7tok/s each First word 41.4 s Measured · A Run →	44.6tok/s each First word 79.4 s Measured · A Run →	25.0tok/s each First word 3.7 min Measured · A Run →	18.8tok/s each First word 11.4 min Measured · A Run →	35.9tok/s each First word 14.7 s Measured · A Run →
8	43.2tok/s each First word 6.1 s Measured · A Run →	7.6tok/s each First word 67.0 s Measured · A Run →	6.9tok/s each First word 2.1 min Measured · A Run →	Capacity limit — confirmed By throughput Run →	Capacity limit — confirmed By throughput Run →	Capacity limit — confirmed By latency Run →
16	23.5tok/s each First word 11.1 s Measured · A Run →	Capacity limit — confirmed By throughput Run →	Capacity limit — confirmed By throughput Run →	Above confirmed limit — not attempted Limit: 8 people Run →	Above confirmed limit — not attempted Limit: 8 people Run →	Above confirmed limit — not attempted Limit: 8 people Run →
20	16.7tok/s each First word 12.8 s Measured · A Run →	Above confirmed limit — not attempted Limit: 16 people Run →	Above confirmed limit — not attempted Limit: 16 people Run →	Above confirmed limit — not attempted Limit: 8 people Run →	Above confirmed limit — not attempted Limit: 8 people Run →	Above confirmed limit — not attempted Limit: 8 people Run →
32	9.8tok/s each First word 19.4 s Measured · B Run →	Above confirmed limit — not attempted Limit: 16 people Run →	Above confirmed limit — not attempted Limit: 16 people Run →	Above confirmed limit — not attempted Limit: 8 people Run →	Above confirmed limit — not attempted Limit: 8 people Run →	Above confirmed limit — not attempted Limit: 8 people Run →
64	4.3tok/s each First word 40.7 s Measured · B Run →	Above confirmed limit — not attempted Limit: 16 people Run →	Above confirmed limit — not attempted Limit: 16 people Run →	Above confirmed limit — not attempted Limit: 8 people Run →	Above confirmed limit — not attempted Limit: 8 people Run →	Above confirmed limit — not attempted Limit: 8 people Run →

Speed: per-person median over the cell’s repeats. First word: slowest person in any repeat. A = configuration 415588da, B = configuration ce7a1141; no cell mixes configurations. A capacity limit — confirmed was observed on two different configurations and has no numbers; cells above a confirmed limit were not attempted. Method and terms · Export grid →

Required KTS-1 blocks

The same setup’s required blocks, one run each. Per-repeat values are on each run page.

13 blocks recorded · 11 passed · 2 did not pass

Required KTS-1 blocks for this setup
Block	Result	Recorded numbers	What happened	Run
Gate	Passed	cold decode tps: 105.56 tok/s idle power w: 5.2 W	Result recorded	Run 3cbd3228 →
Block 1 · Speed	Passed	No aggregate numbers published	Result recorded	Run b39ef8e9 →
Block 2 · People × context	Passed	No aggregate numbers published	Result recorded	Run 703e5797 →
Block 3 · Long chats	Failed	No aggregate numbers published	Standard check failed; the retained internal record contains the failure reason.	Run bc1a11e1 →
Block 4 · Answer quality	Failed	No aggregate numbers published	Standard check failed; the retained internal record contains the failure reason.	Run fac9a72d →
Block 5 · Documents	Passed	No aggregate numbers published	Result recorded	Run 192f0255 →
Block 6 · 30-minute stability	Passed	No aggregate numbers published	Result recorded	Run b9a60fe5 →
Block 7 · Resources / cleanup	Passed	No aggregate numbers published	Result recorded	Run 24192130 →
Block 7b · Short chat	Passed	No aggregate numbers published	Result recorded	Run 10af912e →
Block 7c · Capacity search	Passed	No aggregate numbers published	Result recorded	Run 473e4da5 →
Block 7d · Power	Passed	No aggregate numbers published	Result recorded	Run cdc70e34 →
Block 7e · Startup / model swap	Passed	n: 3 count	Result recorded	Run d201dcb8 →
Block 7f · RAG ingest	Passed	pages per hour: 8,906.19 count bytes per 1000 pages: 312,316,695 count	Result recorded	Run 190a3897 →

Headline results

One job, one measured run. Text jobs use the current default where one is published.

This hardware’s KTS-1 results are the grid above; every cell links to its run. Capacity limits are labels, never headlines. Historical runs remain in the archive below.

Five agents at once

A separate lab workload, not part of KTS-1. Five AI agents work at the same time, each in its own copy of a large open-source code base and a fictional shop’s data. They fix bugs, add features, answer questions about the code and produce data reports using tools. Hidden checks grade the final files and answers, never the agent’s own claim.

Met the bar

38/40 tasks solved5 agents served at once2.2 s first word95% of turns42.9 tok/s eachper reply · median

Qwen3.6-35B-A3B · NVFP4 variant · NVIDIA NGC vLLM 0.29 · with multi-token prediction

Every task, the full bar and method →

Bug fixes: 10/12
Data jobs: 12/12
New features: 6/6
Questions about the code: 8/8
Multi-step job: 2/2

Historical archive

Historical pre-standard observations and earlier exploratory sweeps on this hardware, with original measurements and setup history. Each run page contains its workload shape, request rows and provenance links.

Charts and answer checks

KTS-1 runs in progress. Comparison charts and answer checks will appear here as standard test suites complete.

Chat and code17 setups · 1 passed

Chat and code, current setups
Setup	Workload and settings	Measured results	Result	Run
Qwen3.8-27B · INT4 variant · NVIDIA NGC vLLM 0.29 · with multi-token prediction	concurrency: 1 · configured window: 8192	single user decode tps: 23.3 tok/sper person · single · sample count not recorded	Not graded	Run 6d39ad79 →
Qwen3.8-27B · INT4 variant · NVIDIA NGC vLLM 0.29	concurrency: 1 · configured window: 8192 · kv dtype: bf16	workload total: 16 countmachine total · single · sample count not recorded workload passed: 16 countmachine total · single · sample count not recorded single user decode tps: 13.6 tok/sper person · single · sample count not recorded	Passed	Run c8b7c076 →
Qwen3.8-27B · GSQ-RCO IQ3_S GGUF variant · llama.cpp	concurrency: 1 · configured window: 65536	chat decode tps: 16.05 tok/sper person · mean · sample count not recorded code ttft mean s: 0.13 sper person · mean · sample count not recorded code decode tps: 15.96 tok/sper person · mean · sample count not recorded Whole-host unavailable peak: 20.37 GiBmachine total · max · sample count not recorded	Not graded	Run 878d422a →
Qwen3.6-35B-A3B · NVFP4 variant · NVIDIA NGC vLLM · with multi-token prediction	configured window: 262144 · kv dtype: fp8	workload total: 32 countmachine total · single · sample count not recorded workload passed: 25 countmachine total · single · sample count not recorded	Partial	Run 1abbd6f0 →
Qwen3.8-27B · INT4 variant · vLLM 0.26	configured window: 8192		Failed	Run 290d4246 →
Qwen3.8-27B · NVFP4 variant with BF16 head · SGLang · speculative decoding	concurrency: 1 · configured window: 262144 · kv dtype: fp8_e4m3	chat decode tps: 19.73 tok/sper person · mean · sample count not recorded code decode tps: 24.14 tok/sper person · mean · sample count not recorded Whole-host unavailable peak: 110.51 GiBmachine total · max · sample count not recorded	Not graded	Run 75a65a84 →
Qwen3.8-27B · NVFP4 variant · SGLang · speculative decoding	concurrency: 2 · configured window: 262144 · kv dtype: fp8_e4m3	workload total: 16 countmachine total · single · sample count not recorded workload passed: 16 countmachine total · single · sample count not recorded tuning candidates: 9 countmachine total · single · sample count not recorded rejected candidates: 4 countmachine total · single · sample count not recorded Machine total: 59.16 tok/smachine total · single · sample count not recorded	Partial	Run d732a4ff →
Qwen3.6-35B-A3B · NVFP4 variant · NVIDIA NGC vLLM	configured window: 262144 · kv dtype: fp8	workload total: 32 countmachine total · single · sample count not recorded workload passed: 26 countmachine total · single · sample count not recorded	Partial	Run a8379842 →
Qwen3.8-27B · GSQ-RCO IQ3_S GGUF variant · llama.cpp · with multi-token prediction	concurrency: 2 · configured window: 65536	workload total: 16 countmachine total · single · sample count not recorded chat decode tps: 26.99 tok/sper person · mean · sample count not recorded workload passed: 15 countmachine total · single · sample count not recorded code decode tps: 33.64 tok/sper person · mean · sample count not recorded	Partial	Run d92bd079 →
Ornith-1.5-35B-A3B · NVFP4 variant · NVIDIA NGC vLLM	configured window: 262144 · kv dtype: fp8	workload total: 32 countmachine total · single · sample count not recorded workload passed: 10 countmachine total · single · sample count not recorded	Partial	Run 27103643 →
Qwen3.8-27B · NVFP4 variant with BF16 head · SGLang · speculative decoding	concurrency: 1 · configured window: 262144 · kv dtype: fp8_e4m3	chat decode tps: 26.44 tok/sper person · mean · sample count not recorded code decode tps: 40.56 tok/sper person · mean · sample count not recorded Whole-host unavailable peak: 112.66 GiBmachine total · max · sample count not recorded	Not graded	Run 62294ed0 →
Qwen3.8-27B · NVFP4 variant · SGLang · speculative decoding	concurrency: 1 · configured window: 262144 · kv dtype: fp8_e4m3	chat decode tps: 29.69 tok/sper person · mean · sample count not recorded code decode tps: 57.95 tok/sper person · mean · sample count not recorded Whole-host unavailable peak: 91.6 GiBmachine total · max · sample count not recorded	Not graded	Run d806de80 →
Ornith-1.5-35B-A3B · NVFP4 variant · NVIDIA NGC vLLM · with multi-token prediction	configured window: 262144 · kv dtype: fp8	workload total: 32 countmachine total · single · sample count not recorded workload passed: 9 countmachine total · single · sample count not recorded	Partial	Run 2f65fb17 →
Qwen3.8-27B · NVFP4 variant with BF16 head · SGLang · speculative decoding	concurrency: 1 · configured window: 262144 · kv dtype: fp8_e4m3	chat decode tps: 21.24 tok/sper person · mean · sample count not recorded code decode tps: 31.99 tok/sper person · mean · sample count not recorded Whole-host unavailable peak: 115.1 GiBmachine total · max · sample count not recorded	Not graded	Run d43d3d29 →
Qwen3.8-27B · NVFP4 variant · SGLang dev · speculative decoding	concurrency: 1 · configured window: 262144 · kv dtype: fp8_e4m3	workload total: 16 countmachine total · single · sample count not recorded chat decode tps: 34.33 tok/sper person · mean of 3 workload passed: 15 countmachine total · single · sample count not recorded code decode tps: 57.53 tok/sper person · mean of 3	Partial	Run 4d30da78 →
Qwen3.8-27B · NVFP4 variant with BF16 head · vLLM · speculative decoding	concurrency: 2 · configured window: 262144 · kv dtype: fp8	chat decode tps: 25.42 tok/sper person · mean · sample count not recorded code decode tps: 42.75 tok/sper person · mean · sample count not recorded Whole-host unavailable peak: 92.14 GiBmachine total · max · sample count not recorded	Not graded	Run 85224459 →
Qwen3.8-27B · NVFP4 variant · vLLM · speculative decoding	concurrency: 1 · configured window: 262144 · kv dtype: fp8	chat decode tps: 27.76 tok/sper person · mean · sample count not recorded code decode tps: 43.45 tok/sper person · mean · sample count not recorded Whole-host unavailable peak: 120.3 GiBmachine total · max · sample count not recorded	Not graded	Run 452eb52c →

Team load4 setups · 2 passed

Team load, current setups
Setup	Workload and settings	Measured results	Result	Run
Ornith-1.5-35B-A3B · NVFP4 variant · NVIDIA NGC vLLM 0.29	concurrency: 20 · configured window: 262144 · kv dtype: fp8 · thinking: false	First word after: 113.84 sper person · median of 20 hidden facts total: 60 countmachine total · single · sample count not recorded Slowest first word: 161.93 sper person · max of 20 hidden facts correct: 39 countmachine total · single · sample count not recorded correctness probes passed: 13 countmachine total · single · sample count not recorded Machine total: 71.65 tok/smachine total · single · sample count not recorded Machine total including first-word wait: 50.81 tok/smachine total · single · sample count not recorded correctness probes total: 20 countmachine total · single · sample count not recorded Writing speed each person: 5.95 tok/sper person · median of 20	Failed	Run e51d5cbb →
Qwen3.6-35B-A3B · NVFP4 variant · NVIDIA NGC vLLM 0.29	concurrency: 20 · configured window: 262144 · kv dtype: fp8 · thinking: false	First word after: 114.35 sper person · median of 20 hidden facts total: 60 countmachine total · single · sample count not recorded Slowest first word: 162.46 sper person · max of 20 hidden facts correct: 60 countmachine total · single · sample count not recorded correctness probes passed: 20 countmachine total · single · sample count not recorded Machine total: 71.52 tok/smachine total · single · sample count not recorded Machine total including first-word wait: 50.68 tok/smachine total · single · sample count not recorded correctness probes total: 20 countmachine total · single · sample count not recorded Writing speed each person: 5.95 tok/sper person · median of 20	Passed	Run d65163d3 →
Qwen3.6-35B-A3B · NVFP4 variant · NVIDIA NGC vLLM 0.29 · with multi-token prediction	concurrency: 20 · configured window: 262144 · kv dtype: fp8 · thinking: false	First word after: 115.25 sper person · median of 20 hidden facts total: 60 countmachine total · single · sample count not recorded Slowest first word: 165.8 sper person · max of 20 hidden facts correct: 60 countmachine total · single · sample count not recorded mtp acceptance rate: 0.87 ratiomachine total · single · sample count not recorded correctness probes passed: 20 countmachine total · single · sample count not recorded Machine total: 76.29 tok/smachine total · single · sample count not recorded Machine total including first-word wait: 53.57 tok/smachine total · single · sample count not recorded correctness probes total: 20 countmachine total · single · sample count not recorded Writing speed each person: 6.98 tok/sper person · median of 20	Passed	Run 071dcd0b →
Ornith-1.5-35B-A3B · NVFP4 variant · NVIDIA NGC vLLM 0.29 · with multi-token prediction	concurrency: 20 · configured window: 262144 · kv dtype: fp8 · thinking: false	First word after: 118.86 sper person · median of 20 hidden facts total: 60 countmachine total · single · sample count not recorded Slowest first word: 169.94 sper person · max of 20 hidden facts correct: 39 countmachine total · single · sample count not recorded mtp acceptance rate: 0.71 ratiomachine total · single · sample count not recorded correctness probes passed: 13 countmachine total · single · sample count not recorded Machine total: 73.35 tok/smachine total · single · sample count not recorded Machine total including first-word wait: 51.6 tok/smachine total · single · sample count not recorded correctness probes total: 20 countmachine total · single · sample count not recorded Writing speed each person: 6.71 tok/sper person · median of 20	Failed	Run 64ac3c82 →

Long documents9 setups · 6 passed

Long documents, current setups
Setup	Workload and settings	Measured results	Result	Run
Qwen3.8-27B · NVFP4 variant · SGLang dev · speculative decoding	5 people · distinct · cold · 64,000 filled content tokens	First word: 144.94 / 235.63 sper-person median / maximum 15/15 Hidden facts found 5/5 completed · 4 overlapping decodes 300 s reference: met 116.20 GiB whole-host peak Software thermal slowdown: not observed in samples	Failed	Run bf96ae39 →
Qwen3.8-27B · NVFP4 variant · SGLang dev · speculative decoding	5 people · distinct · cold · 128,000 filled content tokens	First word: 377.19 / 626.96 sper-person median / maximum 15/15 Hidden facts found 5/5 completed · 4 overlapping decodes 300 s reference: missed 116.34 GiB whole-host peak Software thermal slowdown: not observed in samples	Failed	Run 275ce298 →
Qwen3.8-27B · NVFP4 variant · SGLang dev · speculative decoding	1 people · distinct · cold · 250,000 filled content tokens	First word: 361.40 / 361.40 sper-person median / maximum 3/3 Hidden facts found 1/1 completed · 1 overlapping decodes 300 s reference: missed 116.08 GiB whole-host peak Software thermal slowdown: not observed in samples	Passed	Run 8ed191b4 →
Qwen3.8-27B · NVFP4 variant · SGLang dev · speculative decoding	2 people · distinct · cold · 250,000 filled content tokens	First word: 546.45 / 724.19 sper-person median / maximum 6/6 Hidden facts found 2/2 completed · 2 overlapping decodes 300 s reference: missed 116.31 GiB whole-host peak Software thermal slowdown: not observed in samples	Passed	Run 041ce71d →
Qwen3.8-27B · NVFP4 variant · SGLang dev · speculative decoding	3 people · distinct · cold · 250,000 filled content tokens	First word: 733.62 / 1087.17 sper-person median / maximum 9/9 Hidden facts found 3/3 completed · 3 overlapping decodes 300 s reference: missed 116.21 GiB whole-host peak Software thermal slowdown: not observed in samples	Passed	Run 8061ab66 →
Qwen3.8-27B · NVFP4 variant · SGLang dev · speculative decoding	4 people · distinct · cold · 250,000 filled content tokens	First word: 911.69 / 1446.39 sper-person median / maximum 12/12 Hidden facts found 4/4 completed · 4 overlapping decodes 300 s reference: missed 116.28 GiB whole-host peak Software thermal slowdown: not observed in samples	Passed	Run 055e55e7 →
Qwen3.8-27B · NVFP4 variant · SGLang dev · speculative decoding	5 people · distinct · cold · 250,000 filled content tokens	First word: 1090.27 / 1817.38 sper-person median / maximum 15/15 Hidden facts found 5/5 completed · 4 overlapping decodes 300 s reference: missed 116.26 GiB whole-host peak Software thermal slowdown: not observed in samples	Failed	Run f08bccbb →
Qwen3.8-27B · NVFP4 variant · SGLang dev · speculative decoding	5 people · distinct · warm · 250,000 filled content tokens	First word: 368.02 / 1093.74 sper-person median / maximum 15/15 Hidden facts found 5/5 completed · 4 overlapping decodes 300 s reference: missed 116.25 GiB whole-host peak Software thermal slowdown: not observed in samples	Failed	Run 1572c5a8 →
Qwen3.8-27B · NVFP4 variant · SGLang dev · speculative decoding	5 people · shared · cold · 250,000 filled content tokens	First word: 373.25 / 384.65 sper-person median / maximum 15/15 Hidden facts found 5/5 completed · 4 overlapping decodes 300 s reference: missed 116.11 GiB whole-host peak Software thermal slowdown: not observed in samples	Failed	Run 6b3b5039 →
Qwen3.8-27B · NVFP4 variant · SGLang dev · speculative decoding	5 people · shared · warm · 250,000 filled content tokens	First word: 6.83 / 7.79 sper-person median / maximum 15/15 Hidden facts found 5/5 completed · 5 overlapping decodes 300 s reference: met 115.99 GiB whole-host peak Software thermal slowdown: not observed in samples	Passed	Run ba811d67 →
Qwen3.6-35B-A3B · NVFP4 variant · NVIDIA NGC vLLM 0.29	1 people · distinct · cold · 250,000 filled content tokens	First word: 146.82 / 146.82 sper-person median / maximum 3/3 Hidden facts found 1/1 completed · 1 overlapping decodes 300 s reference: met 99.41 GiB whole-host peak Software thermal slowdown: not observed in samples	Passed	Run 46082918 →
Qwen3.6-35B-A3B · NVFP4 variant · NVIDIA NGC vLLM 0.29	5 people · distinct · cold · 250,000 filled content tokens	First word: 633.74 / 636.86 sper-person median / maximum 15/15 Hidden facts found 5/5 completed · 5 overlapping decodes 300 s reference: missed 100.89 GiB whole-host peak Software thermal slowdown: not observed in samples	Passed	Run a3fe8bb1 →
Qwen3.6-35B-A3B · NVFP4 variant · NVIDIA NGC vLLM 0.29	5 people · shared · cold · 250,000 filled content tokens	First word: 632.52 / 635.17 sper-person median / maximum 14/15 Hidden facts found 5/5 completed · 5 overlapping decodes 300 s reference: missed 100.59 GiB whole-host peak Software thermal slowdown: not observed in samples	Failed	Run 1b41e8c8 →
Qwen3.6-35B-A3B · NVFP4 variant · NVIDIA NGC vLLM 0.29	5 people · shared · warm · 250,000 filled content tokens	First word: 3.37 / 4.24 sper-person median / maximum 14/15 Hidden facts found 5/5 completed · 5 overlapping decodes 300 s reference: met 100.55 GiB whole-host peak Software thermal slowdown: not observed in samples	Failed	Run 5eaafaf6 →
Qwen3.8-27B · INT4 variant · NVIDIA NGC vLLM 0.29	5 people · distinct · cold · 64,000 filled content tokens	First word: 243.79 / 379.62 sper-person median / maximum 15/15 Hidden facts found 5/5 completed · 5 overlapping decodes 300 s reference: missed 103.63 GiB whole-host peak Software thermal slowdown: observed	Passed	Run ffc9ce5a →
Qwen3.8-27B · INT4 variant · NVIDIA NGC vLLM 0.29	5 people · distinct · cold · 128,000 filled content tokens	First word: 552.72 / 903.15 sper-person median / maximum 15/15 Hidden facts found 5/5 completed · 5 overlapping decodes 300 s reference: missed 103.38 GiB whole-host peak Software thermal slowdown: observed	Passed	Run 8a460b5c →
Qwen3.8-27B · INT4 variant · NVIDIA NGC vLLM 0.29	1 people · distinct · cold · 250,000 filled content tokens	First word: 457.98 / 457.98 sper-person median / maximum 3/3 Hidden facts found 1/1 completed · 1 overlapping decodes 300 s reference: missed 103.30 GiB whole-host peak Software thermal slowdown: not observed in samples	Passed	Run 19df494d →
Qwen3.8-27B · INT4 variant · NVIDIA NGC vLLM 0.29	2 people · distinct · cold · 250,000 filled content tokens	First word: 695.60 / 919.28 sper-person median / maximum 6/6 Hidden facts found 2/2 completed · 2 overlapping decodes 300 s reference: missed 103.38 GiB whole-host peak Software thermal slowdown: observed	Passed	Run 3b0fd2f8 →
Qwen3.8-27B · INT4 variant · NVIDIA NGC vLLM 0.29	3 people · distinct · cold · 250,000 filled content tokens	First word: 932.03 / 1379.84 sper-person median / maximum 9/9 Hidden facts found 3/3 completed · 3 overlapping decodes 300 s reference: missed 103.39 GiB whole-host peak Software thermal slowdown: observed	Passed	Run 063d452c →
Qwen3.8-27B · INT4 variant · NVIDIA NGC vLLM 0.29	4 people · distinct · cold · 250,000 filled content tokens	First word: 1162.91 / 1842.91 sper-person median / maximum 12/12 Hidden facts found 4/4 completed · 4 overlapping decodes 300 s reference: missed 103.46 GiB whole-host peak Software thermal slowdown: observed	Passed	Run 60aa0818 →
Qwen3.8-27B · INT4 variant · NVIDIA NGC vLLM 0.29	5 people · distinct · cold · 250,000 filled content tokens	First word: 1395.27 / 2307.50 sper-person median / maximum 15/15 Hidden facts found 5/5 completed · 4 overlapping decodes 300 s reference: missed 103.49 GiB whole-host peak Software thermal slowdown: observed	Failed	Run 0fef1bb0 →
Qwen3.8-27B · INT4 variant · NVIDIA NGC vLLM 0.29	5 people · shared · cold · 250,000 filled content tokens	First word: 467.26 / 467.26 sper-person median / maximum 15/15 Hidden facts found 5/5 completed · 5 overlapping decodes 300 s reference: missed 103.59 GiB whole-host peak Software thermal slowdown: observed	Passed	Run 28597e23 →
Qwen3.8-27B · INT4 variant · NVIDIA NGC vLLM 0.29	5 people · shared · cold · 250,000 filled content tokens	First word: 467.14 / 467.14 sper-person median / maximum 15/15 Hidden facts found 5/5 completed · 5 overlapping decodes 300 s reference: missed 103.46 GiB whole-host peak Software thermal slowdown: not observed in samples	Passed	Run 7f54527e →
Qwen3.8-27B · INT4 variant · NVIDIA NGC vLLM 0.29	5 people · shared · warm · 250,000 filled content tokens	First word: 8.91 / 10.68 sper-person median / maximum 15/15 Hidden facts found 5/5 completed · 5 overlapping decodes 300 s reference: met 103.50 GiB whole-host peak Software thermal slowdown: not observed in samples	Passed	Run f494ed66 →
Qwen3.8-27B · INT4 variant · NVIDIA NGC vLLM 0.29	5 people · shared · warm · 250,000 filled content tokens	First word: 10.88 / 10.89 sper-person median / maximum 15/15 Hidden facts found 5/5 completed · 5 overlapping decodes 300 s reference: met 103.38 GiB whole-host peak Software thermal slowdown: not observed in samples	Passed	Run 5c550512 →
Qwen3.8-27B · NVFP4 variant · SGLang dev	5 people · distinct · cold · 64,000 filled content tokens	First word: 140.65 / 239.41 sper-person median / maximum 15/15 Hidden facts found 5/5 completed · 4 overlapping decodes 300 s reference: met 114.20 GiB whole-host peak Software thermal slowdown: not observed in samples	Failed	Run 1281f46a →
Qwen3.8-27B · NVFP4 variant · SGLang dev	5 people · distinct · cold · 128,000 filled content tokens	First word: 372.46 / 632.84 sper-person median / maximum 15/15 Hidden facts found 5/5 completed · 4 overlapping decodes 300 s reference: missed 114.19 GiB whole-host peak Software thermal slowdown: not observed in samples	Failed	Run 4c073dc4 →
Qwen3.8-27B · NVFP4 variant · SGLang dev	1 people · distinct · cold · 250,000 filled content tokens	First word: 360.53 / 360.53 sper-person median / maximum 3/3 Hidden facts found 1/1 completed · 1 overlapping decodes 300 s reference: missed 114.08 GiB whole-host peak Software thermal slowdown: not observed in samples	Passed	Run 6963b324 →
Qwen3.8-27B · NVFP4 variant · SGLang dev	2 people · distinct · cold · 250,000 filled content tokens	First word: 542.39 / 719.99 sper-person median / maximum 6/6 Hidden facts found 2/2 completed · 2 overlapping decodes 300 s reference: missed 114.13 GiB whole-host peak Software thermal slowdown: not observed in samples	Passed	Run 5f884458 →
Qwen3.8-27B · NVFP4 variant · SGLang dev	3 people · distinct · cold · 250,000 filled content tokens	First word: 732.45 / 1088.78 sper-person median / maximum 9/9 Hidden facts found 3/3 completed · 3 overlapping decodes 300 s reference: missed 114.22 GiB whole-host peak Software thermal slowdown: not observed in samples	Passed	Run 14d21ed0 →
Qwen3.8-27B · NVFP4 variant · SGLang dev	4 people · distinct · cold · 250,000 filled content tokens	First word: 914.14 / 1453.74 sper-person median / maximum 12/12 Hidden facts found 4/4 completed · 4 overlapping decodes 300 s reference: missed 114.21 GiB whole-host peak Software thermal slowdown: not observed in samples	Passed	Run b5dc5ee1 →
Qwen3.8-27B · NVFP4 variant · SGLang dev	5 people · distinct · cold · 250,000 filled content tokens	First word: 1082.95 / 1821.66 sper-person median / maximum 15/15 Hidden facts found 5/5 completed · 4 overlapping decodes 300 s reference: missed 114.32 GiB whole-host peak Software thermal slowdown: not observed in samples	Failed	Run b9fd006b →
Qwen3.8-27B · NVFP4 variant · SGLang dev	5 people · distinct · warm · 250,000 filled content tokens	First word: 726.13 / 1462.36 sper-person median / maximum 15/15 Hidden facts found 5/5 completed · 4 overlapping decodes 300 s reference: missed 114.23 GiB whole-host peak Software thermal slowdown: not observed in samples	Failed	Run 449efa9b →
Qwen3.8-27B · NVFP4 variant · SGLang dev	5 people · shared · cold · 250,000 filled content tokens	First word: 373.65 / 388.36 sper-person median / maximum 15/15 Hidden facts found 5/5 completed · 4 overlapping decodes 300 s reference: missed 114.29 GiB whole-host peak Software thermal slowdown: not observed in samples	Failed	Run 8092de5e →
Qwen3.8-27B · NVFP4 variant · SGLang dev	5 people · shared · warm · 250,000 filled content tokens	First word: 10.03 / 10.42 sper-person median / maximum 15/15 Hidden facts found 5/5 completed · 5 overlapping decodes 300 s reference: met 114.14 GiB whole-host peak Software thermal slowdown: not observed in samples	Passed	Run 20d9c54b →
Qwen3.8-27B · INT4 variant · NVIDIA NGC vLLM 0.29	5 people · distinct · cold · 250,000 filled content tokens	First word: 2287.02 / 2288.09 sper-person median / maximum 15/15 Hidden facts found 5/5 completed · 5 overlapping decodes 300 s reference: missed 104.52 GiB whole-host peak Software thermal slowdown: observed	Passed	Run 143509cb →
Qwen3.8-27B · INT4 variant · NVIDIA NGC vLLM 0.29	5 people · distinct · warm · 250,000 filled content tokens	First word: 8.84 / 10.70 sper-person median / maximum 15/15 Hidden facts found 5/5 completed · 5 overlapping decodes 300 s reference: met 104.31 GiB whole-host peak Software thermal slowdown: not observed in samples	Passed	Run 641d1e53 →
NVIDIA-Nemotron-3.5-Lightning-30B-A3B · Q4_K_M GGUF variant · llama.cpp	1 people · distinct · cold · 250,000 filled content tokens	First word: 118.04 / 118.04 sper-person median / maximum 3/3 Hidden facts found 1/1 completed · 1 overlapping decodes 300 s reference: met 35.80 GiB whole-host peak Software thermal slowdown: not observed in samples	Passed	Run 6fc815a1 →
NVIDIA-Nemotron-3.5-Lightning-30B-A3B · Q4_K_M GGUF variant · llama.cpp	5 people · distinct · cold · 250,000 filled content tokens	First word: 375.66 / 624.68 sper-person median / maximum 15/15 Hidden facts found 5/5 completed · 3 overlapping decodes 300 s reference: missed 36.17 GiB whole-host peak Software thermal slowdown: observed	Failed	Run b911e8d5 →
NVIDIA-Nemotron-3.5-Lightning-30B-A3B · Q4_K_M GGUF variant · llama.cpp	5 people · shared · cold · 250,000 filled content tokens	First word: 373.11 / 611.98 sper-person median / maximum 15/15 Hidden facts found 5/5 completed · 4 overlapping decodes 300 s reference: missed 36.21 GiB whole-host peak Software thermal slowdown: observed	Failed	Run 2e4b0560 →
NVIDIA-Nemotron-3.5-Lightning-30B-A3B · Q4_K_M GGUF variant · llama.cpp	5 people · shared · warm · 250,000 filled content tokens	First word: 356.62 / 595.54 sper-person median / maximum 15/15 Hidden facts found 5/5 completed · 4 overlapping decodes 300 s reference: missed 35.27 GiB whole-host peak Software thermal slowdown: observed	Failed	Run 73fcdc42 →
Ornith-1.5-35B-A3B · NVFP4 variant · NVIDIA NGC vLLM 0.29 · with multi-token prediction	5 people · distinct · cold · 250,000 filled content tokens	First word: 673.54 / 676.91 sper-person median / maximum 12/15 Hidden facts found 5/5 completed · 5 overlapping decodes 300 s reference: missed 94.36 GiB whole-host peak Software thermal slowdown: unassessed	Failed	Run fcdbf0c4 →
Ornith-1.5-35B-A3B · NVFP4 variant · NVIDIA NGC vLLM 0.29 · with multi-token prediction	5 people · shared · cold · 250,000 filled content tokens	First word: 672.79 / 673.09 sper-person median / maximum 15/15 Hidden facts found 5/5 completed · 5 overlapping decodes 300 s reference: missed 94.35 GiB whole-host peak Software thermal slowdown: unassessed	Passed	Run ec6866ac →
Ornith-1.5-35B-A3B · NVFP4 variant · NVIDIA NGC vLLM 0.29 · with multi-token prediction	5 people · shared · warm · 250,000 filled content tokens	First word: 14.44 / 14.45 sper-person median / maximum 15/15 Hidden facts found 5/5 completed · 5 overlapping decodes 300 s reference: met 94.23 GiB whole-host peak Software thermal slowdown: unassessed	Passed	Run d0edaadb →
Qwen3.6-35B-A3B · NVFP4 variant · NVIDIA NGC vLLM 0.29 · with multi-token prediction	5 people · distinct · cold · 250,000 filled content tokens	First word: 676.47 / 679.32 sper-person median / maximum 15/15 Hidden facts found 5/5 completed · 5 overlapping decodes 300 s reference: missed 93.88 GiB whole-host peak Software thermal slowdown: unassessed	Passed	Run 107303a0 →
Qwen3.6-35B-A3B · NVFP4 variant · NVIDIA NGC vLLM 0.29 · with multi-token prediction	5 people · shared · cold · 250,000 filled content tokens	First word: 676.94 / 676.94 sper-person median / maximum 13/15 Hidden facts found 5/5 completed · 5 overlapping decodes 300 s reference: missed 93.95 GiB whole-host peak Software thermal slowdown: unassessed	Failed	Run 24e87573 →
Qwen3.6-35B-A3B · NVFP4 variant · NVIDIA NGC vLLM 0.29 · with multi-token prediction	5 people · shared · warm · 250,000 filled content tokens	First word: 14.47 / 14.47 sper-person median / maximum 13/15 Hidden facts found 5/5 completed · 5 overlapping decodes 300 s reference: met 93.87 GiB whole-host peak Software thermal slowdown: unassessed	Failed	Run 68146b5a →
Qwen3.8-27B · IQ3_S GGUF variant · llama.cpp · with multi-token prediction	5 people · distinct · cold · 64,000 filled content tokens	First word: 306.97 / 508.62 sper-person median / maximum 15/15 Hidden facts found 5/5 completed · 2 overlapping decodes 300 s reference: missed 78.26 GiB whole-host peak Software thermal slowdown: observed	Failed	Run bd428b7f →
Qwen3.8-27B · IQ3_S GGUF variant · llama.cpp · with multi-token prediction	5 people · distinct · cold · 128,000 filled content tokens	First word: 700.47 / 1163.41 sper-person median / maximum 15/15 Hidden facts found 5/5 completed · 1 overlapping decodes 300 s reference: missed 80.63 GiB whole-host peak Software thermal slowdown: observed	Failed	Run 4bfc3fa5 →
Qwen3.8-27B · IQ3_S GGUF variant · llama.cpp · with multi-token prediction	1 people · distinct · cold · 250,000 filled content tokens	First word: 562.34 / 562.34 sper-person median / maximum 3/3 Hidden facts found 1/1 completed · 1 overlapping decodes 300 s reference: missed 75.58 GiB whole-host peak Software thermal slowdown: observed	Passed	Run a17c165b →
Qwen3.8-27B · IQ3_S GGUF variant · llama.cpp · with multi-token prediction	2 people · distinct · cold · 250,000 filled content tokens	First word: 857.60 / 1143.42 sper-person median / maximum 6/6 Hidden facts found 2/2 completed · 1 overlapping decodes 300 s reference: missed 78.00 GiB whole-host peak Software thermal slowdown: observed	Failed	Run d00aaae5 →
Qwen3.8-27B · IQ3_S GGUF variant · llama.cpp · with multi-token prediction	3 people · distinct · cold · 250,000 filled content tokens	First word: 1147.20 / 1714.08 sper-person median / maximum 9/9 Hidden facts found 3/3 completed · 1 overlapping decodes 300 s reference: missed 80.51 GiB whole-host peak Software thermal slowdown: observed	Failed	Run f06367aa →
Qwen3.8-27B · IQ3_S GGUF variant · llama.cpp · with multi-token prediction	4 people · distinct · cold · 250,000 filled content tokens	First word: 1437.85 / 2300.67 sper-person median / maximum 12/12 Hidden facts found 4/4 completed · 1 overlapping decodes 300 s reference: missed 82.89 GiB whole-host peak Software thermal slowdown: observed	Failed	Run 23cac8cd →
Qwen3.8-27B · IQ3_S GGUF variant · llama.cpp · with multi-token prediction	5 people · distinct · cold · 250,000 filled content tokens	First word: 1733.38 / 2885.06 sper-person median / maximum 15/15 Hidden facts found 5/5 completed · 1 overlapping decodes 300 s reference: missed 85.23 GiB whole-host peak Software thermal slowdown: observed	Failed	Run a4cda7bc →
Qwen3.8-27B · IQ3_S GGUF variant · llama.cpp · with multi-token prediction	5 people · distinct · warm · 250,000 filled content tokens	First word: 3.42 / 3.42 sper-person median / maximum 15/15 Hidden facts found 5/5 completed · 5 overlapping decodes 300 s reference: met 85.47 GiB whole-host peak Software thermal slowdown: not observed in samples	Passed	Run 48e8f271 →
Qwen3.8-27B · IQ3_S GGUF variant · llama.cpp · with multi-token prediction	5 people · shared · cold · 250,000 filled content tokens	First word: 1729.64 / 2879.65 sper-person median / maximum 15/15 Hidden facts found 5/5 completed · 1 overlapping decodes 300 s reference: missed 85.52 GiB whole-host peak Software thermal slowdown: observed	Failed	Run 6d004e21 →
Qwen3.8-27B · IQ3_S GGUF variant · llama.cpp · with multi-token prediction	5 people · shared · warm · 250,000 filled content tokens	First word: 3.39 / 3.39 sper-person median / maximum 15/15 Hidden facts found 5/5 completed · 5 overlapping decodes 300 s reference: met 85.53 GiB whole-host peak Software thermal slowdown: not observed in samples	Passed	Run ef86f90e →

Rolling chat9 setups · 2 passed

Rolling chat, current setups
Setup	Workload and settings	Measured results	Result	Run
Qwen3.6-35B-A3B · NVFP4 variant · NVIDIA NGC vLLM 0.29	concurrency: 5 · configured window: 262144	roll ttft max seconds: 4.09 sper person · max · sample count not recorded requests ok: 42 countmachine total · single · sample count not recorded roll ttft p50 seconds: 4.06 sper person · median · sample count not recorded rolled off invalid: 7 countmachine total · single · sample count not recorded rolled off wrong: 1 countmachine total · single · sample count not recorded normal ttft max seconds: 3.97 sper person · max · sample count not recorded rolled off correct: 37 countmachine total · single · sample count not recorded normal ttft p50 seconds: 3.97 sper person · median · sample count not recorded rolled off total: 45 countmachine total · single · sample count not recorded requests total: 50 countmachine total · single · sample count not recorded	Failed	Run b043ed1b →
Qwen3.8-27B · INT4 variant · NVIDIA NGC vLLM 0.29	concurrency: 5 · configured window: 262144	roll ttft max seconds: 190.31 sper person · max · sample count not recorded requests ok: 50 countmachine total · single · sample count not recorded roll ttft p50 seconds: 154.99 sper person · median · sample count not recorded rolled off invalid: 0 countmachine total · single · sample count not recorded rolled off wrong: 0 countmachine total · single · sample count not recorded normal ttft max seconds: 136.46 sper person · max · sample count not recorded rolled off correct: 35 countmachine total · single · sample count not recorded normal ttft p50 seconds: 109.02 sper person · median · sample count not recorded rolled off total: 35 countmachine total · single · sample count not recorded requests total: 50 countmachine total · single · sample count not recorded	Failed	Run 933bd14a →
Qwen3.6-35B-A3B · NVFP4 variant · NVIDIA NGC vLLM 0.29	concurrency: 5 · configured window: 262144	roll ttft max seconds: 34.95 sper person · max · sample count not recorded requests ok: 49 countmachine total · single · sample count not recorded roll ttft p50 seconds: 27.79 sper person · median · sample count not recorded rolled off invalid: 0 countmachine total · single · sample count not recorded rolled off wrong: 0 countmachine total · single · sample count not recorded normal ttft max seconds: 19.11 sper person · max · sample count not recorded rolled off correct: 20 countmachine total · single · sample count not recorded normal ttft p50 seconds: 10.43 sper person · median · sample count not recorded rolled off total: 20 countmachine total · single · sample count not recorded requests total: 50 countmachine total · single · sample count not recorded	Failed	Run 7cde5180 →
Qwen3.6-35B-A3B · NVFP4 variant · NVIDIA NGC vLLM 0.29	concurrency: 5 · configured window: 262144	requests ok: 0 countmachine total · single · sample count not recorded rolled off invalid: 0 countmachine total · single · sample count not recorded rolled off wrong: 0 countmachine total · single · sample count not recorded rolled off correct: 0 countmachine total · single · sample count not recorded rolled off total: 0 countmachine total · single · sample count not recorded requests total: 50 countmachine total · single · sample count not recorded	Failed	Run b2acdc4b →
Qwen3.8-27B · INT4 variant · NVIDIA NGC vLLM 0.29	concurrency: 5 · configured window: 262144	roll ttft max seconds: 83.86 sper person · max · sample count not recorded requests ok: 105 countmachine total · single · sample count not recorded roll ttft p50 seconds: 42.64 sper person · median · sample count not recorded rolled off invalid: 0 countmachine total · single · sample count not recorded rolled off wrong: 0 countmachine total · single · sample count not recorded normal ttft max seconds: 58.59 sper person · max · sample count not recorded rolled off correct: 100 countmachine total · single · sample count not recorded normal ttft p50 seconds: 36.15 sper person · median · sample count not recorded rolled off total: 100 countmachine total · single · sample count not recorded requests total: 105 countmachine total · single · sample count not recorded	Passed	Run 861767d1 →
Qwen3.8-27B · INT4 variant · NVIDIA NGC vLLM 0.29	concurrency: 5 · configured window: 262144	roll ttft max seconds: 275.18 sper person · max · sample count not recorded requests ok: 50 countmachine total · single · sample count not recorded roll ttft p50 seconds: 192.55 sper person · median · sample count not recorded rolled off invalid: 0 countmachine total · single · sample count not recorded rolled off wrong: 0 countmachine total · single · sample count not recorded normal ttft max seconds: 127.11 sper person · max · sample count not recorded rolled off correct: 20 countmachine total · single · sample count not recorded normal ttft p50 seconds: 85.78 sper person · median · sample count not recorded rolled off total: 20 countmachine total · single · sample count not recorded requests total: 50 countmachine total · single · sample count not recorded	Failed	Run 8a22159d →
Qwen3.6-35B-A3B · NVFP4 variant · NVIDIA NGC vLLM 0.29	concurrency: 5 · configured window: 262144	roll ttft max seconds: 276.94 sper person · max · sample count not recorded requests ok: 59 countmachine total · single · sample count not recorded roll ttft p50 seconds: 215.31 sper person · median · sample count not recorded normal ttft max seconds: 69.03 sper person · max · sample count not recorded normal ttft p50 seconds: 39.16 sper person · median · sample count not recorded requests total: 105 countmachine total · single · sample count not recorded	Failed	Run 5f1b8f8e →
Qwen3.8-27B · INT4 variant · NVIDIA NGC vLLM 0.29	concurrency: 5 · configured window: 262144	roll ttft max seconds: 879.48 sper person · max · sample count not recorded requests ok: 60 countmachine total · single · sample count not recorded roll ttft p50 seconds: 675.79 sper person · median · sample count not recorded normal ttft max seconds: 237.34 sper person · max · sample count not recorded normal ttft p50 seconds: 173.58 sper person · median · sample count not recorded requests total: 105 countmachine total · single · sample count not recorded	Failed	Run c4e68314 →
Qwen3.6-35B-A3B · NVFP4 variant · NVIDIA NGC vLLM 0.29	concurrency: 5 · configured window: 262144	roll ttft max seconds: 29.24 sper person · max · sample count not recorded requests ok: 105 countmachine total · single · sample count not recorded roll ttft p50 seconds: 21.28 sper person · median · sample count not recorded rolled off invalid: 0 countmachine total · single · sample count not recorded rolled off wrong: 0 countmachine total · single · sample count not recorded normal ttft max seconds: 20.32 sper person · max · sample count not recorded rolled off correct: 90 countmachine total · single · sample count not recorded normal ttft p50 seconds: 14.05 sper person · median · sample count not recorded rolled off total: 90 countmachine total · single · sample count not recorded requests total: 105 countmachine total · single · sample count not recorded	Passed	Run 272292f2 →

Answer quality5 setups · 0 passed

Answer quality, current setups
Setup	Workload and settings	Measured results	Result	Run
Ornith-1.5-35B-A3B · NVFP4 variant · NVIDIA NGC vLLM 0.29	concurrency: 1 · configured window: 262144 · kv dtype: fp8	skipped cases: 4 countmachine total · single · sample count not recorded correctness failures: 50 countmachine total · single · sample count not recorded api cases passed: 71 countmachine total · single · sample count not recorded api cases total: 124 countmachine total · single · sample count not recorded completeness failures: 11 countmachine total · single · sample count not recorded	Partial	Run 70d5791f →
Qwen3.6-35B-A3B · NVFP4 variant · NVIDIA NGC vLLM 0.29 · with multi-token prediction	concurrency: 1 · configured window: 262144 · kv dtype: fp8	skipped cases: 4 countmachine total · single · sample count not recorded correctness failures: 18 countmachine total · single · sample count not recorded api cases passed: 101 countmachine total · single · sample count not recorded api cases total: 124 countmachine total · single · sample count not recorded completeness failures: 9 countmachine total · single · sample count not recorded	Partial	Run ffd38b7a →
Qwen3.6-35B-A3B · NVFP4 variant · NVIDIA NGC vLLM 0.29 · with multi-token prediction	concurrency: 1 · configured window: 262144 · kv dtype: fp8	skipped cases: 16 countmachine total · single · sample count not recorded correctness failures: 2 countmachine total · single · sample count not recorded api cases passed: 109 countmachine total · single · sample count not recorded api cases total: 112 countmachine total · single · sample count not recorded completeness failures: 1 countmachine total · single · sample count not recorded	Partial	Run 525c0716 →
Qwen3.6-35B-A3B · NVFP4 variant · NVIDIA NGC vLLM 0.29	concurrency: 1 · configured window: 262144 · kv dtype: fp8	skipped cases: 4 countmachine total · single · sample count not recorded correctness failures: 19 countmachine total · single · sample count not recorded api cases passed: 100 countmachine total · single · sample count not recorded api cases total: 124 countmachine total · single · sample count not recorded completeness failures: 9 countmachine total · single · sample count not recorded	Partial	Run d490228d →
Ornith-1.5-35B-A3B · NVFP4 variant · NVIDIA NGC vLLM 0.29 · with multi-token prediction	concurrency: 1 · configured window: 262144 · kv dtype: fp8	skipped cases: 4 countmachine total · single · sample count not recorded correctness failures: 48 countmachine total · single · sample count not recorded api cases passed: 72 countmachine total · single · sample count not recorded api cases total: 124 countmachine total · single · sample count not recorded completeness failures: 9 countmachine total · single · sample count not recorded	Partial	Run 18057fc2 →

Documents (RAG)11 setups · 0 passed

Documents (RAG), current setups
Setup	Workload and settings	Measured results	Result	Run
EmbeddingGemma 2 · float32 variant · Transformers text CPU 5.19	answer format: json-object-guided · dimensions: 512 · thinking: false	ingest seconds: 325.03 smachine total · single · sample count not recorded Reply time: 4.68 sper person · median · sample count not recorded recall at 5: 0.69 ratiomachine total · single · sample count not recorded Reply time: 7.41 sper person · p95 · sample count not recorded mrr: 0.69 ratiomachine total · single · sample count not recorded answer and citation correct: 30 countmachine total · single · sample count not recorded questions: 40 countmachine total · single · sample count not recorded pages: 800 countmachine total · single · sample count not recorded recall at 10: 0.75 ratiomachine total · single · sample count not recorded	Not graded	Run 32df0b19 →
EmbeddingGemma 2 · float32 variant · Transformers text CPU 5.19	dimensions: 512	ingest seconds: 323.04 smachine total · single · sample count not recorded Reply time: 26.89 sper person · median · sample count not recorded recall at 5: 0.69 ratiomachine total · single · sample count not recorded Reply time: 74.42 sper person · p95 · sample count not recorded mrr: 0.69 ratiomachine total · single · sample count not recorded answer and citation correct: 28 countmachine total · single · sample count not recorded questions: 40 countmachine total · single · sample count not recorded pages: 800 countmachine total · single · sample count not recorded recall at 10: 0.75 ratiomachine total · single · sample count not recorded	Not graded	Run e2b8dc39 →
all-MiniLM-L6-v2 · float32 variant · Transformers text CPU 5.19	dimensions: 384	ingest seconds: 82.64 smachine total · single · sample count not recorded Reply time: 20.67 sper person · median · sample count not recorded recall at 5: 0.31 ratiomachine total · single · sample count not recorded Reply time: 65.27 sper person · p95 · sample count not recorded mrr: 0.2 ratiomachine total · single · sample count not recorded answer and citation correct: 13 countmachine total · single · sample count not recorded questions: 40 countmachine total · single · sample count not recorded pages: 800 countmachine total · single · sample count not recorded recall at 10: 0.63 ratiomachine total · single · sample count not recorded	Not graded	Run 8938e1a8 →
EmbeddingGemma 2 · float32 variant · Transformers text CPU 5.19	dimensions: 512 · thinking: false	ingest seconds: 325.03 smachine total · single · sample count not recorded Reply time: 4.69 sper person · median · sample count not recorded recall at 5: 0.69 ratiomachine total · single · sample count not recorded Reply time: 7.58 sper person · p95 · sample count not recorded mrr: 0.69 ratiomachine total · single · sample count not recorded answer and citation correct: 1 countmachine total · single · sample count not recorded questions: 40 countmachine total · single · sample count not recorded pages: 800 countmachine total · single · sample count not recorded recall at 10: 0.75 ratiomachine total · single · sample count not recorded	Not graded	Run 16664715 →
EmbeddingGemma 2 · float32 variant · Transformers text CPU 5.19	dimensions: 512 · thinking: false	ingest seconds: 322.24 smachine total · single · sample count not recorded Reply time: 31.31 sper person · median · sample count not recorded recall at 5: 0.69 ratiomachine total · single · sample count not recorded Reply time: 66.64 sper person · p95 · sample count not recorded mrr: 0.69 ratiomachine total · single · sample count not recorded answer and citation correct: 27 countmachine total · single · sample count not recorded questions: 40 countmachine total · single · sample count not recorded pages: 800 countmachine total · single · sample count not recorded recall at 10: 0.75 ratiomachine total · single · sample count not recorded	Not graded	Run 740e0beb →
EmbeddingGemma 2 · float32 variant · Transformers text CPU 5.19	dimensions: 512	ingest seconds: 321.14 smachine total · single · sample count not recorded Reply time: 36.46 sper person · median · sample count not recorded recall at 5: 0.66 ratiomachine total · single · sample count not recorded Reply time: 98.68 sper person · p95 · sample count not recorded mrr: 0.56 ratiomachine total · single · sample count not recorded answer and citation correct: 23 countmachine total · single · sample count not recorded questions: 40 countmachine total · single · sample count not recorded pages: 800 countmachine total · single · sample count not recorded recall at 10: 0.72 ratiomachine total · single · sample count not recorded	Not graded	Run 1c681a2b →
EmbeddingGemma 2 · float32 variant · Transformers text CPU 5.19	dimensions: 512	ingest seconds: 316.99 smachine total · single · sample count not recorded Reply time: 22.22 sper person · median · sample count not recorded recall at 5: 0.53 ratiomachine total · single · sample count not recorded Reply time: 66.3 sper person · p95 · sample count not recorded mrr: 0.47 ratiomachine total · single · sample count not recorded answer and citation correct: 26 countmachine total · single · sample count not recorded questions: 40 countmachine total · single · sample count not recorded pages: 800 countmachine total · single · sample count not recorded recall at 10: 0.59 ratiomachine total · single · sample count not recorded	Not graded	Run 7d583e38 →
EmbeddingGemma 2 · float32 variant · Transformers text CPU 5.19	dimensions: 512	ingest seconds: 326.19 smachine total · single · sample count not recorded Reply time: 25.61 sper person · median · sample count not recorded recall at 5: 0.66 ratiomachine total · single · sample count not recorded Reply time: 78.08 sper person · p95 · sample count not recorded mrr: 0.6 ratiomachine total · single · sample count not recorded answer and citation correct: 25 countmachine total · single · sample count not recorded questions: 40 countmachine total · single · sample count not recorded pages: 800 countmachine total · single · sample count not recorded recall at 10: 0.78 ratiomachine total · single · sample count not recorded	Not graded	Run 641fc07e →
EmbeddingGemma 2 · float32 variant · Transformers text CPU 5.19	dimensions: 512	ingest seconds: 322.3 smachine total · single · sample count not recorded Reply time: 35.69 sper person · median · sample count not recorded recall at 5: 0.66 ratiomachine total · single · sample count not recorded Reply time: 97.27 sper person · p95 · sample count not recorded mrr: 0.56 ratiomachine total · single · sample count not recorded answer and citation correct: 23 countmachine total · single · sample count not recorded questions: 40 countmachine total · single · sample count not recorded pages: 800 countmachine total · single · sample count not recorded recall at 10: 0.72 ratiomachine total · single · sample count not recorded	Not graded	Run bb291997 →
all-MiniLM-L6-v2 · float32 variant · SentenceTransformers inside Open WebUI 5.5	dimensions: 384	ingest seconds: 85.12 smachine total · single · sample count not recorded Reply time: 17.76 sper person · median · sample count not recorded recall at 5: 0.31 ratiomachine total · single · sample count not recorded Reply time: 40.65 sper person · p95 · sample count not recorded mrr: 0.2 ratiomachine total · single · sample count not recorded answer and citation correct: 7 countmachine total · single · sample count not recorded questions: 40 countmachine total · single · sample count not recorded pages: 800 countmachine total · single · sample count not recorded recall at 10: 0.63 ratiomachine total · single · sample count not recorded	Not graded	Run 45b9fcd9 →
all-MiniLM-L6-v2 · float32 variant · SentenceTransformers inside Open WebUI 5.5	dimensions: 384 · thinking: false	ingest seconds: 85.65 smachine total · single · sample count not recorded Reply time: 17.52 sper person · median · sample count not recorded recall at 5: 0.31 ratiomachine total · single · sample count not recorded Reply time: 73.6 sper person · p95 · sample count not recorded mrr: 0.2 ratiomachine total · single · sample count not recorded answer and citation correct: 8 countmachine total · single · sample count not recorded questions: 40 countmachine total · single · sample count not recorded pages: 800 countmachine total · single · sample count not recorded recall at 10: 0.63 ratiomachine total · single · sample count not recorded	Not graded	Run 0934d05e →

Images2 setups · 2 passed

Images, current setups
Setup	Workload and settings	Measured results	Result	Run
Qwen-Image-2512 · BF16 variant · Keep image engine on NVIDIA NGC PyTorch 25.11 + Diffusers 0.40	height: 1024 · width: 1024	workload total: 2 countmachine total · single · sample count not recorded CUDA allocator peak: 54.16 GiBmachine total · max · sample count not recorded seconds per image: 159.02 s/imageper person · mean · sample count not recorded workload passed: 2 countmachine total · single · sample count not recorded images: 2 countmachine total · single · sample count not recorded	Passed	Run 8426a04d →
Z-Image-Turbo · BF16 variant · Keep image engine on NVIDIA NGC PyTorch 25.11 + Diffusers 0.40	height: 1024 · width: 1024	workload total: 2 countmachine total · single · sample count not recorded CUDA allocator peak: 21.67 GiBmachine total · max · sample count not recorded seconds per image: 12.95 s/imageper person · mean · sample count not recorded workload passed: 2 countmachine total · single · sample count not recorded images: 2 countmachine total · single · sample count not recorded	Passed	Run adda7e6d →

Video4 setups · 4 passed

Video, current setups
Setup	Workload and settings	Measured results	Result	Run
Wan2.1 T2V 1.3B · F32 source / BF16 runtime / FP32 VAE variant · NGC PyTorch / Diffusers video	height: 480 · width: 832	CUDA allocator peak: 15.67 GiBmachine total · max · sample count not recorded fps: 15 frames/smachine total · single · sample count not recorded frames: 81 countmachine total · single · sample count not recorded clip seconds: 5.4 smachine total · single · sample count not recorded generation seconds: 562.57 sper person · single · sample count not recorded	Passed	Run f9293514 →
Wan2.2 TI2V 5B · F32 source / BF16 runtime / FP32 VAE variant · NGC PyTorch / Diffusers video	height: 704 · width: 1280	CUDA allocator peak: 28.41 GiBmachine total · max · sample count not recorded fps: 24 frames/smachine total · single · sample count not recorded frames: 121 countmachine total · single · sample count not recorded clip seconds: 5.04 smachine total · single · sample count not recorded generation seconds: 1,225.6 sper person · single · sample count not recorded	Passed	Run 93a2b3aa →
Wan2.1 T2V 1.3B · F32 source / BF16 runtime / FP32 VAE variant · NGC PyTorch / Diffusers video	height: 240 · width: 416	CUDA allocator peak: 15.07 GiBmachine total · max · sample count not recorded fps: 15 frames/smachine total · single · sample count not recorded frames: 33 countmachine total · single · sample count not recorded clip seconds: 2.2 smachine total · single · sample count not recorded generation seconds: 18.62 sper person · single · sample count not recorded	Passed	Run f19c0f4b →
Wan2.2 TI2V 5B · F32 source / BF16 runtime / FP32 VAE variant · NGC PyTorch / Diffusers video	height: 352 · width: 640	CUDA allocator peak: 24.05 GiBmachine total · max · sample count not recorded fps: 24 frames/smachine total · single · sample count not recorded frames: 33 countmachine total · single · sample count not recorded clip seconds: 1.38 smachine total · single · sample count not recorded generation seconds: 27.16 sper person · single · sample count not recorded	Passed	Run 1f246fdb →

Method and terms

How to read and compare these runs

Writing speed (decode, tok/s) measures output after the first word. Concurrent streams may pause during other people’s prefill. First word after includes scheduling and reading the prompt.

Each person describes one request’s experience; median and mean are labelled. Machine total (aggregate) measures the whole cohort over a stated window. Per-person rates are never summed to estimate it.

New (cold) means no reusable prompt KV state. Already read (warm) means the prefix was processed earlier in the same server lifetime. Warm results never supply reading speed; cached-token counts stay unknown when the engine did not record them.

The grid isolates each model, variant, engine version and generic feature. Cold columns use distinct documents; warm cells label shared inputs. Omitted shapes remain in the historical archive. A cell links to one run, never a blend of setups. The 300-second first-word reference is independent of exact-answer and completion checks. A linked note sends runs with missed checks to the bottom ledger.

Configured window is the server allowance. Filled input is the measured prompt, including the chat template where token usage is available. Completion, exact answer checks, decode overlap and the first-word reference are independent gates.

These are finite lab observations. They establish neither general answer quality nor customer delivery qualification. The workload and public method are listed on each run.

Data and changes

Download the published dataset or use Open Local Inference Result (OLIR) for your own runs. One record describes one run; every numeric metric carries scope, statistic, unit and its original field.

Full dataset · JSON One row per number · CSV All runs · OLIR JSON OLIR schema OLIR format Changelog · JSON Dataset SHA-256

Publication history

2026-10-21
Regenerated public records; immutable measurements retained.
Affected run IDs
- Run 1bdb1c2c
2026-10-19
Regenerated public records; immutable measurements retained.
Affected run IDs
2026-10-14
Regenerated public records; immutable measurements retained.
Affected run IDs
2026-10-14
Regenerated public records; immutable measurements retained.
Affected run IDs
2026-10-10
Regenerated public records; immutable measurements retained.
Affected run IDs
- Run 0388c0a8
- Run bf13ac82
2026-10-10
Regenerated public records; immutable measurements retained.
Affected run IDs
2026-10-09
Regenerated public records; immutable measurements retained.
Affected run IDs
2026-10-09
Regenerated public records; immutable measurements retained.
Affected run IDs
- Run f26ae856 (withdrawn)
- Run 13f45764
- Run 1d3b3864
- Run 424b33c3
- Run 4357e9e5
- Run 61def837
- Run 68a30d35
- Run 6a38761f
- Run 6c11f8fa
- Run 729aab16
- Run 78abd8c8
- Run 858a9f76
- Run 94cf38e4
- Run b98a127d
- Run bb974390
- Run cc19d195
2026-10-09
Regenerated public records; immutable measurements retained.
Affected run IDs
2026-10-09
Regenerated public records; immutable measurements retained.
Affected run IDs
2026-10-09
Regenerated public records; immutable measurements retained.
Affected run IDs
2026-10-09
OLIR 0.1: explicit scope and statistic; per-person team median computed from requests alongside the recorded mean. Warm cache-served prefill withdrawn; unknown cached-token counts marked null. Reproduction hashes and vendor digests added where recorded. Public source paths removed; hardware naming normalized.
Affected run IDs
2026-10-07
Corrected publication inventory: existing captured MoE measurements were imported with their original dates and identities. No workload was rerun for the import.
Affected run IDs
2026-10-07
Corrected thermal interpretation: software slowdown was sampled in Nemotron five-person runs. Hardware slowdown was not observed in samples; sampled absence does not prove continuous absence.
Affected run IDs

Cite a single result using the citation on its run page. Original measurement dates are preserved; republishing does not refresh a measurement.

What did not work

Structured tasks: 14 runs missed the full task grade.
Tuning sweep: 3 attempts missed their run checks.
Chat and code: 1 attempts missed their run checks.
Long documents: 32 attempts did not start or missed a completion, timing or answer check.
Team load: 6 runs missed exact-answer or request checks.
Rolling chat: 21 earlier runs missed checks. Full run details remain available.
Answer quality: 6 runs missed one or more API or correctness checks.
KTS-1 standard: 3 gate or block runs did not pass their check.

Full list · 86 failed or partial runs

Failed and partial runs and startup attempts, newest first
Date	Result	Setup	Tried	What happened	Run
2026-10-19	Failed	Qwen3.6-35B-A3B · NVFP4 variant · NVIDIA NGC vLLM 0.29	KTS-1 standard	Standard check failed; the retained internal record contains the failure reason.	Run fac9a72d →
2026-10-19	Partial	Qwen3.6-35B-A3B · NVFP4 variant · NVIDIA NGC vLLM 0.29 · with multi-token prediction	Answer quality	111/114 tasks passed. 109/112 API cases passed.	Run 525c0716 →
2026-10-19	Partial	Qwen3.6-35B-A3B · NVFP4 variant · NVIDIA NGC vLLM · with multi-token prediction	Structured tasks	26/32 tasks passed.	Run 21616041 →
2026-10-19	Failed	Qwen3.6-35B-A3B · NVFP4 variant · NVIDIA NGC vLLM 0.29	KTS-1 standard	Standard check failed; the retained internal record contains the failure reason.	Run bc1a11e1 →
2026-10-10	Failed	Qwen3.6-35B-A3B · NVFP4 variant · NVIDIA NGC vLLM 0.29	KTS-1 standard	Standard check failed; the retained internal record contains the failure reason.	Run bf13ac82 →
2026-10-09	Did not start	Qwen3.8-27B · Mirai S CUDA dev	Long documents	Startup, inference completion, concurrent decode and retrieval correctness are separate gates.	Run 8fd6f651 →
2026-10-07	Failed	Qwen3.8-27B · INT4 variant · NVIDIA NGC vLLM 0.29	Rolling chat	Bounded tuning pilot; incomplete window growth cannot pass the full workload grade.	Run 8a22159d →
2026-10-07	Failed	Qwen3.8-27B · INT4 variant · NVIDIA NGC vLLM 0.29	Rolling chat	Bounded tuning pilot; incomplete window growth cannot pass the full workload grade.	Run 933bd14a →
2026-10-07	Failed	Qwen3.8-27B · INT4 variant · NVIDIA NGC vLLM 0.29	Rolling chat	Bounded tuning pilot; incomplete window growth cannot pass the full workload grade.	Run 6abf3f90 →
2026-10-07	Failed	Qwen3.8-27B · INT4 variant · NVIDIA NGC vLLM 0.29	Rolling chat	Bounded tuning pilot; incomplete window growth cannot pass the full workload grade.	Run 378ddde7 →
2026-10-07	Failed	Qwen3.6-35B-A3B · NVFP4 variant · NVIDIA NGC vLLM 0.29	Rolling chat	49/50 requests passed.	Run 7cde5180 →
2026-10-07	Failed	Qwen3.6-35B-A3B · NVFP4 variant · NVIDIA NGC vLLM 0.29	Rolling chat	49/50 requests passed.	Run 90c568b5 →
2026-10-07	Failed	Qwen3.6-35B-A3B · NVFP4 variant · NVIDIA NGC vLLM 0.29	Rolling chat	42/50 requests passed.	Run b043ed1b →
2026-10-07	Failed	Qwen3.6-35B-A3B · NVFP4 variant · NVIDIA NGC vLLM 0.29	Rolling chat	Bounded tuning pilot; incomplete window growth cannot pass the full workload grade.	Run 7fb62d76 →
2026-10-07	Failed	Qwen3.6-35B-A3B · NVFP4 variant · NVIDIA NGC vLLM 0.29	Rolling chat	19/20 requests passed.	Run 5341cef0 →
2026-10-07	Failed	Qwen3.6-35B-A3B · NVFP4 variant · NVIDIA NGC vLLM 0.29	Rolling chat	18/20 requests passed.	Run 78fe8c2f →
2026-10-07	Failed	Qwen3.6-35B-A3B · NVFP4 variant · NVIDIA NGC vLLM 0.29	Rolling chat	12/20 requests passed.	Run b5aac3fc →
2026-10-07	Failed	Qwen3.6-35B-A3B · NVFP4 variant · NVIDIA NGC vLLM 0.29	Rolling chat	16/20 requests passed.	Run e1c73db9 →
2026-10-07	Failed	Qwen3.6-35B-A3B · NVFP4 variant · NVIDIA NGC vLLM 0.29	Rolling chat	19/20 requests passed.	Run 60ac465f →
2026-10-07	Failed	Qwen3.6-35B-A3B · NVFP4 variant · NVIDIA NGC vLLM 0.29	Rolling chat	12/20 requests passed.	Run a95b7a20 →
2026-10-07	Failed	Qwen3.6-35B-A3B · NVFP4 variant · NVIDIA NGC vLLM 0.29	Rolling chat	46/50 requests passed.	Run 2e4b4a0f →
2026-10-07	Failed	Qwen3.6-35B-A3B · NVFP4 variant · NVIDIA NGC vLLM 0.29	Rolling chat	Bounded tuning pilot; incomplete window growth cannot pass the full workload grade.	Run 9124cbc7 →
2026-10-07	Failed	Qwen3.6-35B-A3B · NVFP4 variant · NVIDIA NGC vLLM 0.29	Rolling chat	1/2 requests passed.	Run 042d9bd0 →
2026-10-07	Failed	Qwen3.6-35B-A3B · NVFP4 variant · NVIDIA NGC vLLM 0.29	Rolling chat	5/50 requests passed.	Run ac1380ca →
2026-10-07	Failed	Qwen3.6-35B-A3B · NVFP4 variant · NVIDIA NGC vLLM 0.29	Rolling chat	0/50 requests passed.	Run b2acdc4b →
2026-10-07	Failed	Qwen3.8-27B · INT4 variant · NVIDIA NGC vLLM 0.29	Rolling chat	60/105 requests passed.	Run c4e68314 →
2026-10-07	Failed	Qwen3.6-35B-A3B · NVFP4 variant · NVIDIA NGC vLLM 0.29	Rolling chat	59/105 requests passed.	Run 5f1b8f8e →
2026-10-07	Failed	Ornith-1.5-35B-A3B · NVFP4 variant · NVIDIA NGC vLLM 0.29 · with multi-token prediction	Long documents	One or more requests did not meet completion, exact retrieval, synthesis-presence, reasoning-off or token-accounting requirements.	Run fcdbf0c4 →
2026-10-07	Partial	Ornith-1.5-35B-A3B · NVFP4 variant · NVIDIA NGC vLLM 0.29 · with multi-token prediction	Answer quality	73/130 tasks passed. 72/124 API cases passed.	Run 18057fc2 →
2026-10-07	Partial	Ornith-1.5-35B-A3B · NVFP4 variant · NVIDIA NGC vLLM · with multi-token prediction	Structured tasks	9/32 tasks passed.	Run 2f65fb17 →
2026-10-07	Failed	Ornith-1.5-35B-A3B · NVFP4 variant · NVIDIA NGC vLLM 0.29 · with multi-token prediction	Team load	39/60 Hidden facts found. 13/20 probes passed.	Run 64ac3c82 →
2026-10-07	Failed	Ornith-1.5-35B-A3B · NVFP4 variant · NVIDIA NGC vLLM 0.29 · with multi-token prediction	Team load	30/48 Hidden facts found. 10/16 probes passed.	Run 68118ad1 →
2026-10-07	Failed	Ornith-1.5-35B-A3B · NVFP4 variant · NVIDIA NGC vLLM 0.29 · with multi-token prediction	Team load	21/24 Hidden facts found. 7/8 probes passed.	Run 30b57437 →
2026-10-07	Partial	Ornith-1.5-35B-A3B · NVFP4 variant · NVIDIA NGC vLLM 0.29	Answer quality	71/130 tasks passed. 71/124 API cases passed.	Run 70d5791f →
2026-10-07	Partial	Ornith-1.5-35B-A3B · NVFP4 variant · NVIDIA NGC vLLM	Structured tasks	10/32 tasks passed.	Run 27103643 →
2026-10-07	Failed	Ornith-1.5-35B-A3B · NVFP4 variant · NVIDIA NGC vLLM 0.29	Team load	39/60 Hidden facts found. 13/20 probes passed.	Run e51d5cbb →
2026-10-07	Failed	Ornith-1.5-35B-A3B · NVFP4 variant · NVIDIA NGC vLLM 0.29	Team load	39/48 Hidden facts found. 13/16 probes passed.	Run 024e2d03 →
2026-10-07	Failed	Ornith-1.5-35B-A3B · NVFP4 variant · NVIDIA NGC vLLM 0.29	Team load	18/24 Hidden facts found. 6/8 probes passed.	Run 5366b32d →
2026-10-07	Failed	Qwen3.6-35B-A3B · NVFP4 variant · NVIDIA NGC vLLM 0.29 · with multi-token prediction	Long documents	One or more requests did not meet completion, exact retrieval, synthesis-presence, reasoning-off or token-accounting requirements.	Run 68146b5a →
2026-10-07	Failed	Qwen3.6-35B-A3B · NVFP4 variant · NVIDIA NGC vLLM 0.29 · with multi-token prediction	Long documents	One or more requests did not meet completion, exact retrieval, synthesis-presence, reasoning-off or token-accounting requirements.	Run 24e87573 →
2026-10-07	Partial	Qwen3.6-35B-A3B · NVFP4 variant · NVIDIA NGC vLLM 0.29 · with multi-token prediction	Answer quality	103/130 tasks passed. 101/124 API cases passed.	Run ffd38b7a →
2026-10-07	Partial	Qwen3.6-35B-A3B · NVFP4 variant · NVIDIA NGC vLLM · with multi-token prediction	Structured tasks	25/32 tasks passed.	Run 1abbd6f0 →
2026-10-07	Partial	Qwen3.6-35B-A3B · NVFP4 variant · NVIDIA NGC vLLM 0.29	Answer quality	102/130 tasks passed. 100/124 API cases passed.	Run d490228d →
2026-10-07	Failed	Qwen3.6-35B-A3B · NVFP4 variant · NVIDIA NGC vLLM 0.29	Answer quality	Bounded fictional tasks only.	Run dae6c36f →
2026-10-07	Partial	Qwen3.6-35B-A3B · NVFP4 variant · NVIDIA NGC vLLM	Structured tasks	26/32 tasks passed.	Run a8379842 →
2026-10-07	Failed	Qwen3.6-35B-A3B · NVFP4 variant · NVIDIA NGC vLLM 0.29	Long documents	One or more requests did not meet completion, exact retrieval, synthesis-presence, reasoning-off or token-accounting requirements.	Run 5eaafaf6 →
2026-10-07	Failed	Qwen3.6-35B-A3B · NVFP4 variant · NVIDIA NGC vLLM 0.29	Long documents	One or more requests did not meet completion, exact retrieval, synthesis-presence, reasoning-off or token-accounting requirements.	Run 1b41e8c8 →
2026-10-07	Failed	NVIDIA-Nemotron-3.5-Lightning-30B-A3B · Q4_K_M GGUF variant · llama.cpp	Long documents	Observed decode overlap was 4 of 5 requested users.	Run 73fcdc42 →
2026-10-07	Failed	NVIDIA-Nemotron-3.5-Lightning-30B-A3B · Q4_K_M GGUF variant · llama.cpp	Long documents	Observed decode overlap was 4 of 5 requested users.	Run 2e4b0560 →
2026-10-07	Failed	NVIDIA-Nemotron-3.5-Lightning-30B-A3B · Q4_K_M GGUF variant · llama.cpp	Long documents	Observed decode overlap was 3 of 5 requested users.	Run b911e8d5 →
2026-10-06	Failed	NVIDIA-Nemotron-3.5-Lightning-30B-A3B · Q4_K_M GGUF variant · llama.cpp	Long documents	Observed decode overlap was 4 of 5 requested users. One or more requests did not meet completion, exact retrieval, synthesis-presence, reasoning-off or token-accounting requirements.	Run e3ccb9cf →
2026-10-06	Failed	NVIDIA-Nemotron-3.5-Lightning-30B-A3B · Q4_K_M GGUF variant · llama.cpp	Long documents	One or more requests did not meet completion, exact retrieval, synthesis-presence, reasoning-off or token-accounting requirements.	Run 1698e65e →
2026-10-06	Failed	Qwen3.8-27B · IQ3_S GGUF variant · llama.cpp · with multi-token prediction	Long documents	Observed decode overlap was 1 of 5 requested users.	Run 6d004e21 →
2026-10-06	Failed	Qwen3.8-27B · IQ3_S GGUF variant · llama.cpp · with multi-token prediction	Long documents	Observed decode overlap was 2 of 5 requested users.	Run bd428b7f →
2026-10-06	Failed	Qwen3.8-27B · IQ3_S GGUF variant · llama.cpp · with multi-token prediction	Long documents	Observed decode overlap was 1 of 5 requested users.	Run 4bfc3fa5 →
2026-10-06	Failed	Qwen3.8-27B · IQ3_S GGUF variant · llama.cpp · with multi-token prediction	Long documents	Observed decode overlap was 1 of 5 requested users.	Run a4cda7bc →
2026-10-06	Failed	Qwen3.8-27B · IQ3_S GGUF variant · llama.cpp · with multi-token prediction	Long documents	Observed decode overlap was 1 of 4 requested users.	Run 23cac8cd →
2026-10-06	Failed	Qwen3.8-27B · IQ3_S GGUF variant · llama.cpp · with multi-token prediction	Long documents	Observed decode overlap was 1 of 3 requested users.	Run f06367aa →
2026-10-06	Failed	Qwen3.8-27B · IQ3_S GGUF variant · llama.cpp · with multi-token prediction	Long documents	Observed decode overlap was 1 of 2 requested users.	Run d00aaae5 →
2026-10-06	Failed	Qwen3.8-27B · NVFP4 variant · SGLang dev	Long documents	Observed decode overlap was 4 of 5 requested users.	Run 8092de5e →
2026-10-06	Failed	Qwen3.8-27B · NVFP4 variant · SGLang dev	Long documents	Observed decode overlap was 4 of 5 requested users.	Run 1281f46a →
2026-10-06	Failed	Qwen3.8-27B · NVFP4 variant · SGLang dev	Long documents	Observed decode overlap was 4 of 5 requested users.	Run 4c073dc4 →
2026-10-06	Failed	Qwen3.8-27B · NVFP4 variant · SGLang dev	Long documents	Observed decode overlap was 4 of 5 requested users.	Run 449efa9b →
2026-10-06	Failed	Qwen3.8-27B · NVFP4 variant · SGLang dev	Long documents	Observed decode overlap was 4 of 5 requested users.	Run b9fd006b →
2026-10-06	Failed	Qwen3.8-27B · NVFP4 variant · SGLang dev · speculative decoding	Long documents	Observed decode overlap was 4 of 5 requested users.	Run 6b3b5039 →
2026-10-06	Failed	Qwen3.8-27B · NVFP4 variant · SGLang dev · speculative decoding	Long documents	Observed decode overlap was 4 of 5 requested users.	Run bf96ae39 →
2026-10-06	Failed	Qwen3.8-27B · NVFP4 variant · SGLang dev · speculative decoding	Long documents	Observed decode overlap was 4 of 5 requested users.	Run 275ce298 →
2026-10-06	Failed	Qwen3.8-27B · NVFP4 variant · SGLang dev · speculative decoding	Long documents	Observed decode overlap was 4 of 5 requested users.	Run 1572c5a8 →
2026-10-06	Failed	Qwen3.8-27B · NVFP4 variant · SGLang dev · speculative decoding	Long documents	Observed decode overlap was 4 of 5 requested users.	Run f08bccbb →
2026-10-06	Failed	Qwen3.8-27B · NVFP4 variant · SGLang dev · speculative decoding	Long documents	Observed decode overlap was 0 of 1 requested users. One or more requests did not meet completion, exact retrieval, synthesis-presence, reasoning-off or token-accounting requirements. Intentionally interrupted to correct collection of retraction and Mamba gauges; this attempt does not establish an inference or memory failure.	Run 3d6ef7db →
2026-10-06	Failed	Qwen3.8-27B · INT4 variant · NVIDIA NGC vLLM 0.29	Long documents	Observed decode overlap was 4 of 5 requested users.	Run 0fef1bb0 →
2026-10-06	Failed	Qwen3.8-27B · INT4 variant · NVIDIA NGC vLLM 0.29	Long documents	Observed decode overlap was 0 of 2 requested users. One or more requests did not meet completion, exact retrieval, synthesis-presence, reasoning-off or token-accounting requirements. The two-user attempt was intentionally cancelled after the relay limitation was diagnosed.	Run 2f6f82fd →
2026-10-06	Failed	Qwen3.8-27B · INT4 variant · NVIDIA NGC vLLM 0.29	Long documents	Observed decode overlap was 0 of 1 requested users. One or more requests did not meet completion, exact retrieval, synthesis-presence, reasoning-off or token-accounting requirements. The stream closed at the 300-second relay idle limit before the first token; the corrected recipe extends that limit.	Run 37c3ae60 →
2026-10-05	Partial	Qwen3.8-27B · NVFP4 variant · SGLang dev · speculative decoding	Structured tasks	15/16 tasks passed.	Run 4d30da78 →
2026-10-05	Partial	Qwen3.8-27B · GSQ-RCO IQ3_S GGUF variant · llama.cpp · with multi-token prediction	Structured tasks	One task returned fenced text instead of one strict JSON value at each concurrency level. Native runtime identity does not satisfy the workload runner's OCI identity gate.	Run d92bd079 →
2026-10-05	Partial	Qwen3.8-27B · NVFP4 variant · SGLang · speculative decoding	Tuning sweep	The selected candidate passed its bounded checks; rejected and unfinished candidates remain in the table.	Run d732a4ff →
2026-10-05	Partial	Qwen3.8-27B · NVFP4 variant · SGLang · speculative decoding	Tuning sweep	Earlier attempt; comparison or runtime-boundary checks stopped the sweep.	Run f8a9c1e5 →
2026-10-05	Partial	Qwen3.8-27B · NVFP4 variant · SGLang · speculative decoding	Tuning sweep	Earlier attempt; comparison or runtime-boundary checks stopped the sweep.	Run 810e1078 →
2026-10-05	Partial	Qwen3.8-27B · INT4 variant · NVIDIA NGC vLLM 0.29	Structured tasks	Optional smoke exhausted its output budget before a final answer.	Run f4c10f9c →
2026-10-05	Failed	Qwen3.8-27B · GSQ-RCO IQ3_S GGUF variant · llama.cpp · with multi-token prediction	Chat and code	The initial native MTP launch rejected an unsupported command option.	Run ee7ff82a →
2026-10-05	Failed	Qwen3.8-27B · GSQ-RCO IQ3_S GGUF variant · llama.cpp · with multi-token prediction	Structured tasks	The initial workload attempt was refused before tasks because authentication was missing.	Run e07e13ba →
2026-10-05	Failed	Qwen3.8-27B · NVFP4 variant · SGLang · speculative decoding	Structured tasks	CUDA allocation failed during context initialization before weight loading. No other engine was running. A later retry after per-file cache advice started successfully; this does not establish the sole cause.	Run b749797c →
2026-10-05	Failed	Qwen3.8-27B · NVFP4 variant · SGLang · speculative decoding	Structured tasks	Pinned ServerArgs is immutable after resolution; credentials now enter the in-process argument parser before resolution	Run 2c5a80f0 →
2026-10-05	Partial	Qwen3.8-27B · NVFP4 variant · SGLang · speculative decoding	Structured tasks	context-specific-exception returned an extra citation at concurrency 1 and fenced JSON at concurrency 2	Run 26ea705e →
2026-10-04	Failed	Qwen3.8-27B · INT4 variant · vLLM 0.26	Structured tasks	Runtime did not become healthy.	Run 290d4246 →
2026-10-04	Failed	Qwen3.8-27B · INT4 variant · vLLM 0.26	Structured tasks	Image identity could not be verified.	Run 26af7cc2 →