Answer checks
50/50 turns
Rolling-chat turn checks
| Person · turn | Filled input | First word | Recent · pinned · old | Result | Run |
|---|---|---|---|---|---|
1 · 1 | 20208 | 21.35 s | true · true · recalled | Passed | |
1 · 2 | 40334 | 26.00 s | true · true · recalled | Passed | |
1 · 3 | 60462 | 48.95 s | true · true · recalled | Passed | |
1 · 4 | 80595 | 56.14 s | true · true · recalled | Passed | |
1 · 5 | 100723 | 82.31 s | true · true · recalled | Passed | |
1 · 6 | 120853 | 91.19 s | true · true · recalled | Passed | |
1 · 7 | 80765 | 212.87 s | true · true · recalled | Passed | |
1 · 8 | 100891 | 36.44 s | true · true · recalled | Passed | |
1 · 9 | 121021 | 70.18 s | true · true · recalled | Passed | |
1 · 10 | 80771 | 174.01 s | true · true · recalled | Passed | |
2 · 1 | 20214 | 48.61 s | true · true · recalled | Passed | |
2 · 2 | 40338 | 55.96 s | true · true · recalled | Passed | |
2 · 3 | 60465 | 82.20 s | true · true · recalled | Passed | |
2 · 4 | 80597 | 93.23 s | true · true · recalled | Passed | |
2 · 5 | 100728 | 122.79 s | true · true · recalled | Passed | |
2 · 6 | 120852 | 118.01 s | true · true · recalled | Passed | |
2 · 7 | 80772 | 235.59 s | true · true · recalled | Passed | |
2 · 8 | 100900 | 65.22 s | true · true · recalled | Passed | |
2 · 9 | 121034 | 70.11 s | true · true · recalled | Passed | |
2 · 10 | 80777 | 269.82 s | true · true · recalled | Passed | |
3 · 1 | 20209 | 81.10 s | true · true · recalled | Passed | |
3 · 2 | 40339 | 93.25 s | true · true · recalled | Passed | |
3 · 3 | 60471 | 127.11 s | true · true · recalled | Passed | |
3 · 4 | 80597 | 119.55 s | true · true · recalled | Passed | |
3 · 5 | 100730 | 115.67 s | true · true · recalled | Passed | |
3 · 6 | 120860 | 109.18 s | true · true · recalled | Passed | |
3 · 7 | 80775 | 206.20 s | true · true · recalled | Passed | |
3 · 8 | 100903 | 94.37 s | true · true · recalled | Passed | |
3 · 9 | 121033 | 63.80 s | true · true · recalled | Passed | |
3 · 10 | 80770 | 177.92 s | true · true · recalled | Passed | |
4 · 1 | 20212 | 23.82 s | true · true · recalled | Passed | |
4 · 2 | 40341 | 126.19 s | true · true · recalled | Passed | |
4 · 3 | 60478 | 118.53 s | true · true · recalled | Passed | |
4 · 4 | 80606 | 114.78 s | true · true · recalled | Passed | |
4 · 5 | 100735 | 107.12 s | true · true · recalled | Passed | |
4 · 6 | 120860 | 89.24 s | true · true · recalled | Passed | |
4 · 7 | 80767 | 275.18 s | true · true · recalled | Passed | |
4 · 8 | 100904 | 67.31 s | true · true · recalled | Passed | |
4 · 9 | 121033 | 64.34 s | true · true · recalled | Passed | |
4 · 10 | 80782 | 143.37 s | true · true · recalled | Passed | |
5 · 1 | 20214 | 108.41 s | true · true · recalled | Passed | |
5 · 2 | 40342 | 119.35 s | true · true · recalled | Passed | |
5 · 3 | 60472 | 113.27 s | true · true · recalled | Passed | |
5 · 4 | 80602 | 109.21 s | true · true · recalled | Passed | |
5 · 5 | 100732 | 90.44 s | true · true · recalled | Passed | |
5 · 6 | 120864 | 79.00 s | true · true · recalled | Passed | |
5 · 7 | 80777 | 178.91 s | true · true · recalled | Passed | |
5 · 8 | 100907 | 66.35 s | true · true · recalled | Passed | |
5 · 9 | 121037 | 64.14 s | true · true · recalled | Passed | |
5 · 10 | 80778 | 105.55 s | true · true · recalled | Passed |
Method
Staggered multi-turn fictional archive, exact engine-tokenized rolling gateway; independent recent, pinned and old probes each turn.
Limits
Bounded tuning pilot; incomplete window growth cannot pass the full workload grade. Lab retrieval and rolling-history evidence only. Bounded literal user-source excerpts, with verified source hashes; details outside them remain unavailable. Shared engine cache intervals include overlapping users and are not per-user hits. Gateway path, not authenticated document ingestion or customer acceptance.
What did not work
- Bounded tuning pilot; incomplete window growth cannot pass the full workload grade.
What ran
Exact identity
- Original publisher model
- Qwen3.8-27B
- Publisher revision
1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0- Variant · format · precision
- INT4 variant · safetensors · INT4
- Weights SHA-256
- not recorded
- Engine · version · feature
- NVIDIA NGC vLLM ·
0.29.0+5013de39.dev· plain - Hardware
- NVIDIA GB10, 128 GB unified memory
- Harness / workload
keep-rolling-context-v1 · 1- Workload suite · version
not recorded- Harness SHA-256
not recorded- Fixture SHA-256
not recorded
Workload shape
- People in this run
- 5
- Configured window (tokens)
- 262144
- Configured window (tokens)
- 262144
- Output token ceiling
- 256
Full reproduction pack available to customers and evaluators on request
All recorded numbers
Metric definitions, scope and statistics
| Metric | Value | Scope | Statistic | Sample count | Basis / source field | Run |
|---|---|---|---|---|---|---|
roll ttft max seconds | 275.18 s | per person | max | not recorded | roll_ttft_max_seconds | |
requests ok | 50.00 count | machine total | single | not recorded | requests_ok | |
roll ttft p50 seconds | 192.55 s | per person | median | not recorded | roll_ttft_p50_seconds | |
rolled off abstained | 0.00 count | machine total | single | not recorded | rolled_off_abstained | |
rolled off invalid | 0.00 count | machine total | single | not recorded | rolled_off_invalid | |
rolled off wrong | 0.00 count | machine total | single | not recorded | rolled_off_wrong | |
normal ttft max seconds | 127.11 s | per person | max | not recorded | normal_ttft_max_seconds | |
rolled off correct | 20.00 count | machine total | single | not recorded | rolled_off_correct | |
normal ttft p50 seconds | 85.78 s | per person | median | not recorded | normal_ttft_p50_seconds | |
rolled off total | 20.00 count | machine total | single | not recorded | rolled_off_total | |
requests total | 50.00 count | machine total | single | not recorded | requests_total |
Data and citation
Skilak Keep evidence, run 8a22159d4ed2420f8fe7735f, measured 2026-10-07T21:47:51.419664+00:00. https://skilak.ai/keep/resources/evidence/r/8a22159d4ed2420f8fe7735f
