- Writing speed, each persondecode_tps · tok/s
- 6.71 tok/sper person · median of 20
- First word afterttft_median_s · s
- 118.86 sper person · median of 20
- Slowest first wordttft_max_s · s
- 169.94 sper person · max of 20
- Machine totalaggregate_decode_tps · tok/s
- 73.35 tok/smachine total · single · sample count not recorded
- Machine total including first-word waitaggregate_e2e_tps · tok/s
- 51.60 tok/smachine total · single · sample count not recorded
- Everyone answered inwall_seconds · s
- 198.44 smachine total · single · sample count not recorded
- Peak memory, whole hostpeak_host_unavailable_gib · GiB
- 93.93 GiBmachine total · max · sample count not recorded
- mtp acceptance ratemtp_acceptance_rate · ratio
- 0.71 ratiomachine total · single · sample count not recorded
Runs on this setup
Earlier runs keep their own engine versions and outcomes. Workload shapes stay separate.
History and measured sweep
| Date | Shape | Each person · median | Machine total | First word · median | Result | Run |
|---|---|---|---|---|---|---|
2026-10-07 | 20 people | 6.71 tok/s | 73.35 tok/s | 118.86 s | Failed | |
2026-10-07 | 16 people | 8.07 tok/s | 80.97 tok/s | 95.07 s | Failed | |
2026-10-07 | 8 people | 23.73 tok/s | 168.69 tok/s | 60.99 s | Failed | |
2026-10-07 | 1 people | 85.24 tok/s | 85.24 tok/s | 10.39 s | Passed |
Each person in this run
First-word wait against writing speed after that word, including pauses. Earlier streams can pause while other people’s prompts are read. The table preserves filled input, cached-token counts and each person’s checks.
Per-person table · all measured request rows
| Person | Filled input | Cached tokens | Output | First word | Writing after first word | Total time | Checks | Run |
|---|---|---|---|---|---|---|---|---|
1 | 40,101 | unknown | 512 | 110.63 s | 5.98 tok/sone person · includes waits | 196.11 s | 3/3 Hidden facts found · completed · checks passed | |
2 | 40,099 | unknown | 512 | 169.94 s | 17.94 tok/sone person · includes waits | 198.43 s | 0/3 Hidden facts found · completed · checks missed | |
3 | 40,105 | unknown | 512 | 95.21 s | 5.14 tok/sone person · includes waits | 194.69 s | 0/3 Hidden facts found · completed · checks missed | |
4 | 40,098 | unknown | 512 | 169.82 s | 18.03 tok/sone person · includes waits | 198.16 s | 0/3 Hidden facts found · completed · checks missed | |
5 | 40,102 | unknown | 512 | 59.11 s | 3.75 tok/sone person · includes waits | 195.21 s | 3/3 Hidden facts found · completed · checks passed | |
6 | 40,104 | unknown | 512 | 62.75 s | 3.88 tok/sone person · includes waits | 194.32 s | 0/3 Hidden facts found · completed · checks missed | |
7 | 40,103 | unknown | 512 | 128.69 s | 7.46 tok/sone person · includes waits | 197.17 s | 3/3 Hidden facts found · completed · checks passed | |
8 | 40,102 | unknown | 512 | 142.71 s | 9.58 tok/sone person · includes waits | 196.04 s | 3/3 Hidden facts found · completed · checks passed | |
9 | 40,105 | unknown | 512 | 62.75 s | 3.95 tok/sone person · includes waits | 192.09 s | 0/3 Hidden facts found · completed · checks missed | |
10 | 40,101 | unknown | 512 | 95.21 s | 5.08 tok/sone person · includes waits | 195.75 s | 3/3 Hidden facts found · completed · checks passed | |
11 | 40,105 | unknown | 512 | 154.83 s | 11.95 tok/sone person · includes waits | 197.61 s | 3/3 Hidden facts found · completed · checks passed | |
12 | 40,100 | unknown | 512 | 93.53 s | 4.98 tok/sone person · includes waits | 196.24 s | 0/3 Hidden facts found · completed · checks missed | |
13 | 40,104 | unknown | 512 | 62.75 s | 3.87 tok/sone person · includes waits | 194.96 s | 3/3 Hidden facts found · completed · checks passed | |
14 | 40,097 | unknown | 512 | 151.24 s | 10.98 tok/sone person · includes waits | 197.76 s | 3/3 Hidden facts found · completed · checks passed | |
15 | 40,102 | unknown | 512 | 128.69 s | 7.47 tok/sone person · includes waits | 197.12 s | 0/3 Hidden facts found · completed · checks missed | |
16 | 40,103 | unknown | 512 | 166.74 s | 16.93 tok/sone person · includes waits | 196.92 s | 3/3 Hidden facts found · completed · checks passed | |
17 | 40,105 | unknown | 512 | 62.75 s | 3.89 tok/sone person · includes waits | 194.01 s | 3/3 Hidden facts found · completed · checks passed | |
18 | 40,106 | unknown | 512 | 159.95 s | 13.47 tok/sone person · includes waits | 197.89 s | 3/3 Hidden facts found · completed · checks passed | |
19 | 40,105 | unknown | 512 | 127.10 s | 7.45 tok/sone person · includes waits | 195.68 s | 3/3 Hidden facts found · completed · checks passed | |
20 | 40,108 | unknown | 512 | 61.07 s | 3.84 tok/sone person · includes waits | 194.01 s | 3/3 Hidden facts found · completed · checks passed |
Per-person table · all measured request rows
| Person | Filled input | Cached tokens | Output | First word | Writing after first word | Total time | Checks | Run |
|---|---|---|---|---|---|---|---|---|
1 | 40,101 | unknown | 512 | 110.63 s | 5.98 tok/sone person · includes waits | 196.11 s | 3/3 Hidden facts found · completed · checks passed | |
2 | 40,099 | unknown | 512 | 169.94 s | 17.94 tok/sone person · includes waits | 198.43 s | 0/3 Hidden facts found · completed · checks missed | |
3 | 40,105 | unknown | 512 | 95.21 s | 5.14 tok/sone person · includes waits | 194.69 s | 0/3 Hidden facts found · completed · checks missed | |
4 | 40,098 | unknown | 512 | 169.82 s | 18.03 tok/sone person · includes waits | 198.16 s | 0/3 Hidden facts found · completed · checks missed | |
5 | 40,102 | unknown | 512 | 59.11 s | 3.75 tok/sone person · includes waits | 195.21 s | 3/3 Hidden facts found · completed · checks passed | |
6 | 40,104 | unknown | 512 | 62.75 s | 3.88 tok/sone person · includes waits | 194.32 s | 0/3 Hidden facts found · completed · checks missed | |
7 | 40,103 | unknown | 512 | 128.69 s | 7.46 tok/sone person · includes waits | 197.17 s | 3/3 Hidden facts found · completed · checks passed | |
8 | 40,102 | unknown | 512 | 142.71 s | 9.58 tok/sone person · includes waits | 196.04 s | 3/3 Hidden facts found · completed · checks passed | |
9 | 40,105 | unknown | 512 | 62.75 s | 3.95 tok/sone person · includes waits | 192.09 s | 0/3 Hidden facts found · completed · checks missed | |
10 | 40,101 | unknown | 512 | 95.21 s | 5.08 tok/sone person · includes waits | 195.75 s | 3/3 Hidden facts found · completed · checks passed | |
11 | 40,105 | unknown | 512 | 154.83 s | 11.95 tok/sone person · includes waits | 197.61 s | 3/3 Hidden facts found · completed · checks passed | |
12 | 40,100 | unknown | 512 | 93.53 s | 4.98 tok/sone person · includes waits | 196.24 s | 0/3 Hidden facts found · completed · checks missed | |
13 | 40,104 | unknown | 512 | 62.75 s | 3.87 tok/sone person · includes waits | 194.96 s | 3/3 Hidden facts found · completed · checks passed | |
14 | 40,097 | unknown | 512 | 151.24 s | 10.98 tok/sone person · includes waits | 197.76 s | 3/3 Hidden facts found · completed · checks passed | |
15 | 40,102 | unknown | 512 | 128.69 s | 7.47 tok/sone person · includes waits | 197.12 s | 0/3 Hidden facts found · completed · checks missed | |
16 | 40,103 | unknown | 512 | 166.74 s | 16.93 tok/sone person · includes waits | 196.92 s | 3/3 Hidden facts found · completed · checks passed | |
17 | 40,105 | unknown | 512 | 62.75 s | 3.89 tok/sone person · includes waits | 194.01 s | 3/3 Hidden facts found · completed · checks passed | |
18 | 40,106 | unknown | 512 | 159.95 s | 13.47 tok/sone person · includes waits | 197.89 s | 3/3 Hidden facts found · completed · checks passed | |
19 | 40,105 | unknown | 512 | 127.10 s | 7.45 tok/sone person · includes waits | 195.68 s | 3/3 Hidden facts found · completed · checks passed | |
20 | 40,108 | unknown | 512 | 61.07 s | 3.84 tok/sone person · includes waits | 194.01 s | 3/3 Hidden facts found · completed · checks passed |
Answer checks
39/60 Hidden facts found13/20 probes20/20 request checks
Method
team-load-v1; distinct generated 40000-token coding contexts plus fresh nonce prefixes; synchronized streams; temperature zero; thinking off; min_tokens=max_tokens=512; separate cold hidden facts probes; no retries.
Limits
Finite lab cohort; configured 262144 context differs from measured inputs. Aggregate uses first-output to final-completion window. Whole-host memory sampled during speed only; missing counters remain unassessed. Broader coding quality and delivery acceptance unassessed.
What did not work
- 39/60 Hidden facts found.
- 13/20 probes passed.
What ran
Exact identity
- Original publisher model
- Ornith-1.5-35B-A3B
- Publisher revision
10fbf86fed7ecee4a061f8b499a618f46001cac1- Variant · format · precision
- NVFP4 variant · safetensors · nvfp4
- Weights SHA-256
648fe83171485f4be03ea58d86089e088c94c2a3dc88922233f9002f30d8bc8b8303cef7d375021d69dcafc60d3fad5f8186995b18b0db5e2a609aadeb8a4f51cb5c5978721a1e87406ae68f1cd5a855326e40ff8bc13832644c392115336987- Engine · version · feature
- NVIDIA NGC vLLM ·
0.29.0+5013de39.dev· with multi-token prediction - Hardware
- NVIDIA GB10, 128 GB unified memory
- Harness / workload
keep-team-load-v1 · 1- Workload suite · version
not recorded- Harness SHA-256
not recorded- Fixture SHA-256
e2097959b1ca0142f2b93edc3ec411e474c0a6622800dd247af7a71bd9c39e83
Workload shape
- People in this run
- 20
- Configured window (tokens)
- 262144
- Configured window (tokens)
- 262144
- KV cache precision
- fp8
- Output token ceiling
- 512
- Feature
- with multi-token prediction
- Thinking
- off
Full reproduction pack available to customers and evaluators on request
All recorded numbers
Metric definitions, scope and statistics
| Metric | Value | Scope | Statistic | Sample count | Basis / source field | Run |
|---|---|---|---|---|---|---|
wall seconds | 198.44 s | machine total | single | not recorded | wall_seconds | |
ttft median s | 118.86 s | per person | median | 20 | ttft_median_s | |
hidden facts total | 60.00 count | machine total | single | not recorded | needle_total | |
ttft max s | 169.94 s | per person | max | 20 | ttft_max_s | |
requests ok | 20.00 count | machine total | single | not recorded | requests_ok | |
hidden facts correct | 39.00 count | machine total | single | not recorded | needle_correct | |
preemptions | 0.00 count | machine total | single | not recorded | preemptions | |
decode overlap | 20.00 count | machine total | single | not recorded | decode_overlap | |
aggregate concurrency | 20.00 count | machine total | single | not recorded | aggregate_concurrency | |
requests total | 20.00 count | machine total | single | not recorded | requests_total | |
mtp acceptance rate | 0.71 ratio | machine total | single | not recorded | mtp_acceptance_rate | |
correctness probes passed | 13.00 count | machine total | single | not recorded | correctness_probes_passed | |
aggregate decode tps | 73.35 tok/s | machine total | single | not recorded | aggregate_tps | |
aggregate e2e tps | 51.60 tok/s | machine total | single | not recorded | aggregate_end_to_end_tps | |
correctness probes total | 20.00 count | machine total | single | not recorded | correctness_probes_total | |
decode tps | 8.28 tok/s | per person | mean | 20 | per_request_decode_tps | |
forced output tokens | 512.00 tokens | machine total | single | not recorded | forced_output_tokens | |
peak host unavailable gib | 93.93 GiB | machine total | max | not recorded | peak_host_unavailable_gib | |
decode tps | 6.71 tok/s | per person | median | 20 | per_person_decode_median_tps |
Data and citation
Skilak Keep evidence, run 64ac3c82131c4ec28814f973, measured 2026-10-07T06:48:36.191129+00:00. https://skilak.ai/keep/resources/evidence/r/64ac3c82131c4ec28814f973
