- First word afterttft_median_s · s
- 0.06 sper person · median · sample count not recorded
Runs on this setup
Earlier runs keep their own engine versions and outcomes. Workload shapes stay separate.
History and measured sweep
| Date | Shape | Each person · median | Machine total | First word · median | Result | Run |
|---|---|---|---|---|---|---|
2026-10-07 | 1 people | not recorded | not recorded | 0.06 s | Partial | |
2026-10-07 | 1 people | not recorded | not recorded | not recorded | Failed |
Answer checks
102/130 tasks100/124 API cases
| Category | Passed / total | Skipped | Run |
|---|---|---|---|
api | 7/8 | 0 | |
code | 16/16 | 0 | |
format | 12/12 | 0 | |
multi_turn | 8/8 | 0 | |
multilingual | 6/6 | 0 | |
numeric | 16/16 | 0 | |
reasoning | 0/4 | 0 | |
refusal | 6/6 | 0 | |
structured_extraction | 13/16 | 0 | |
supplied_context | 8/16 | 0 | |
tools | 8/8 | 0 | |
untrusted_text | 0/8 | 0 | |
vision | 0/0 | 4 |
Method
Deterministic API suite core-v1; SHA256 42c4dc34a066f791a62c9b1dbeb97943988c746309a43a464f0417a02ae9f1da; no model judge; no retries; unchanged endpoint cache; latency excludes local grading; paired calls counted separately. Client concurrency; configured sequence limit 20.
Limits
Bounded fictional tasks only. Unsupported cases are separately counted, never passes. Original publisher name with variant; exact variant repository and revision retained privately. Not customer acceptance.
What did not work
- 102/130 tasks passed.
- 100/124 API cases passed.
What ran
Exact identity
- Original publisher model
- Qwen3.6-35B-A3B
- Publisher revision
not recorded- Variant · format · precision
- NVFP4 variant · safetensors · NVFP4
- Weights SHA-256
- not recorded
- Engine · version · feature
- NVIDIA NGC vLLM ·
0.29.0+5013de39.dev· plain - Hardware
- NVIDIA GB10, 128 GB unified memory
- Harness / workload
keep-api-suite-run-v1 · 1- Workload suite · version
core-v1 · 1- Harness SHA-256
not recorded- Fixture SHA-256
42c4dc34a066f791a62c9b1dbeb97943988c746309a43a464f0417a02ae9f1da
Workload shape
- People in this run
- 1
- Configured window (tokens)
- 262144
- KV cache precision
- fp8
- Feature
- plain
Full reproduction pack available to customers and evaluators on request
All recorded numbers
Metric definitions, scope and statistics
| Metric | Value | Scope | Statistic | Sample count | Basis / source field | Run |
|---|---|---|---|---|---|---|
skipped cases | 4.00 count | machine total | single | not recorded | skipped_cases | |
correctness failures | 19.00 count | machine total | single | not recorded | correctness_failures | |
ttft median s | 0.06 s | per person | median | not recorded | ttft_median_s | |
workload total | 130.00 count | machine total | single | not recorded | workload_total | |
workload passed | 102.00 count | machine total | single | not recorded | workload_passed | |
api cases passed | 100.00 count | machine total | single | not recorded | api_cases_passed | |
ttft p95 s | 0.12 s | per person | p95 | not recorded | ttft_p95_s | |
completion p95 s | 0.81 s | per person | p95 | not recorded | completion_p95_s | |
api cases total | 124.00 count | machine total | single | not recorded | api_cases_total | |
completion median s | 0.19 s | per person | median | not recorded | completion_median_s | |
completeness failures | 9.00 count | machine total | single | not recorded | completeness_failures |
Data and citation
Skilak Keep evidence, run d490228d2adf4b9fa03f0331, measured 2026-10-07T04:23:11.246879+00:00. https://skilak.ai/keep/resources/evidence/r/d490228d2adf4b9fa03f0331
