- First word afterttft_median_s · s
- 0.06 sper person · median · sample count not recorded
Answer checks
103/130 tasks101/124 API cases
| Category | Passed / total | Skipped | Run |
|---|---|---|---|
api | 7/8 | 0 | |
code | 16/16 | 0 | |
format | 12/12 | 0 | |
multi_turn | 8/8 | 0 | |
multilingual | 6/6 | 0 | |
numeric | 16/16 | 0 | |
reasoning | 0/4 | 0 | |
refusal | 6/6 | 0 | |
structured_extraction | 14/16 | 0 | |
supplied_context | 8/16 | 0 | |
tools | 8/8 | 0 | |
untrusted_text | 0/8 | 0 | |
vision | 0/0 | 4 |
Method
Deterministic API suite core-v1; SHA256 42c4dc34a066f791a62c9b1dbeb97943988c746309a43a464f0417a02ae9f1da; no model judge; no retries; unchanged endpoint cache; latency excludes local grading; paired calls counted separately. Client concurrency; configured sequence limit 20.
Limits
Bounded fictional tasks only. Unsupported cases are separately counted, never passes. Original publisher name with variant; exact variant repository and revision retained privately. Not customer acceptance.
What did not work
- 103/130 tasks passed.
- 101/124 API cases passed.
What ran
Exact identity
- Original publisher model
- Qwen3.6-35B-A3B
- Publisher revision
not recorded- Variant · format · precision
- NVFP4 variant · safetensors · NVFP4
- Weights SHA-256
- not recorded
- Engine · version · feature
- NVIDIA NGC vLLM ·
0.29.0+5013de39.dev· with multi-token prediction - Hardware
- NVIDIA GB10, 128 GB unified memory
- Harness / workload
keep-api-suite-run-v1 · 1- Workload suite · version
core-v1 · 1- Harness SHA-256
not recorded- Fixture SHA-256
42c4dc34a066f791a62c9b1dbeb97943988c746309a43a464f0417a02ae9f1da
Workload shape
- People in this run
- 1
- Configured window (tokens)
- 262144
- KV cache precision
- fp8
- Feature
- with multi-token prediction
Full reproduction pack available to customers and evaluators on request
All recorded numbers
Metric definitions, scope and statistics
| Metric | Value | Scope | Statistic | Sample count | Basis / source field | Run |
|---|---|---|---|---|---|---|
skipped cases | 4.00 count | machine total | single | not recorded | skipped_cases | |
correctness failures | 18.00 count | machine total | single | not recorded | correctness_failures | |
ttft median s | 0.06 s | per person | median | not recorded | ttft_median_s | |
workload total | 130.00 count | machine total | single | not recorded | workload_total | |
workload passed | 103.00 count | machine total | single | not recorded | workload_passed | |
api cases passed | 101.00 count | machine total | single | not recorded | api_cases_passed | |
ttft p95 s | 0.12 s | per person | p95 | not recorded | ttft_p95_s | |
completion p95 s | 0.63 s | per person | p95 | not recorded | completion_p95_s | |
api cases total | 124.00 count | machine total | single | not recorded | api_cases_total | |
completion median s | 0.17 s | per person | median | not recorded | completion_median_s | |
completeness failures | 9.00 count | machine total | single | not recorded | completeness_failures |
Data and citation
Skilak Keep evidence, run ffd38b7a2ab2411ea1a39b6e, measured 2026-10-07T04:50:34.708410+00:00. https://skilak.ai/keep/resources/evidence/r/ffd38b7a2ab2411ea1a39b6e
