- First word afterttft_median_s · s
- 0.06 sper person · median · sample count not recorded
Answer checks
73/130 tasks72/124 API cases
| Category | Passed / total | Skipped | Run |
|---|---|---|---|
api | 6/8 | 0 | |
code | 16/16 | 0 | |
format | 6/12 | 0 | |
multi_turn | 2/8 | 0 | |
multilingual | 6/6 | 0 | |
numeric | 16/16 | 0 | |
reasoning | 0/4 | 0 | |
refusal | 6/6 | 0 | |
structured_extraction | 5/16 | 0 | |
supplied_context | 1/16 | 0 | |
tools | 8/8 | 0 | |
untrusted_text | 0/8 | 0 | |
vision | 0/0 | 4 |
Method
Deterministic API suite core-v1; SHA256 42c4dc34a066f791a62c9b1dbeb97943988c746309a43a464f0417a02ae9f1da; no model judge; no retries; unchanged endpoint cache; latency excludes local grading; paired calls counted separately. Client concurrency; configured sequence limit 20.
Limits
Bounded fictional tasks only. Unsupported cases are separately counted, never passes. Original publisher name with variant; exact variant repository and revision retained privately. Not customer acceptance.
What did not work
- 73/130 tasks passed.
- 72/124 API cases passed.
What ran
Exact identity
- Original publisher model
- Ornith-1.5-35B-A3B
- Publisher revision
not recorded- Variant · format · precision
- NVFP4 variant · safetensors · NVFP4
- Weights SHA-256
- not recorded
- Engine · version · feature
- NVIDIA NGC vLLM ·
0.29.0+5013de39.dev· with multi-token prediction - Hardware
- NVIDIA GB10, 128 GB unified memory
- Harness / workload
keep-api-suite-run-v1 · 1- Workload suite · version
core-v1 · 1- Harness SHA-256
not recorded- Fixture SHA-256
42c4dc34a066f791a62c9b1dbeb97943988c746309a43a464f0417a02ae9f1da
Workload shape
- People in this run
- 1
- Configured window (tokens)
- 262144
- KV cache precision
- fp8
- Feature
- with multi-token prediction
Full reproduction pack available to customers and evaluators on request
All recorded numbers
Metric definitions, scope and statistics
| Metric | Value | Scope | Statistic | Sample count | Basis / source field | Run |
|---|---|---|---|---|---|---|
skipped cases | 4.00 count | machine total | single | not recorded | skipped_cases | |
correctness failures | 48.00 count | machine total | single | not recorded | correctness_failures | |
ttft median s | 0.06 s | per person | median | not recorded | ttft_median_s | |
workload total | 130.00 count | machine total | single | not recorded | workload_total | |
workload passed | 73.00 count | machine total | single | not recorded | workload_passed | |
api cases passed | 72.00 count | machine total | single | not recorded | api_cases_passed | |
ttft p95 s | 0.11 s | per person | p95 | not recorded | ttft_p95_s | |
completion p95 s | 0.62 s | per person | p95 | not recorded | completion_p95_s | |
api cases total | 124.00 count | machine total | single | not recorded | api_cases_total | |
completion median s | 0.19 s | per person | median | not recorded | completion_median_s | |
completeness failures | 9.00 count | machine total | single | not recorded | completeness_failures |
Data and citation
Skilak Keep evidence, run 18057fc2b4e64d4f8cca1ee2, measured 2026-10-07T06:50:22.362351+00:00. https://skilak.ai/keep/resources/evidence/r/18057fc2b4e64d4f8cca1ee2
