- First word afterttft_median_s · s
- 0.06 sper person · median · sample count not recorded
Answer checks
71/130 tasks71/124 API cases
| Category | Passed / total | Skipped | Run |
|---|---|---|---|
api | 5/8 | 0 | |
code | 16/16 | 0 | |
format | 7/12 | 0 | |
multi_turn | 1/8 | 0 | |
multilingual | 6/6 | 0 | |
numeric | 15/16 | 0 | |
reasoning | 0/4 | 0 | |
refusal | 6/6 | 0 | |
structured_extraction | 5/16 | 0 | |
supplied_context | 2/16 | 0 | |
tools | 8/8 | 0 | |
untrusted_text | 0/8 | 0 | |
vision | 0/0 | 4 |
Method
Deterministic API suite core-v1; SHA256 42c4dc34a066f791a62c9b1dbeb97943988c746309a43a464f0417a02ae9f1da; no model judge; no retries; unchanged endpoint cache; latency excludes local grading; paired calls counted separately. Client concurrency; configured sequence limit 20.
Limits
Bounded fictional tasks only. Unsupported cases are separately counted, never passes. Original publisher name with variant; exact variant repository and revision retained privately. Not customer acceptance.
What did not work
- 71/130 tasks passed.
- 71/124 API cases passed.
What ran
Exact identity
- Original publisher model
- Ornith-1.5-35B-A3B
- Publisher revision
not recorded- Variant · format · precision
- NVFP4 variant · safetensors · NVFP4
- Weights SHA-256
- not recorded
- Engine · version · feature
- NVIDIA NGC vLLM ·
0.29.0+5013de39.dev· plain - Hardware
- NVIDIA GB10, 128 GB unified memory
- Harness / workload
keep-api-suite-run-v1 · 1- Workload suite · version
core-v1 · 1- Harness SHA-256
not recorded- Fixture SHA-256
42c4dc34a066f791a62c9b1dbeb97943988c746309a43a464f0417a02ae9f1da
Workload shape
- People in this run
- 1
- Configured window (tokens)
- 262144
- KV cache precision
- fp8
- Feature
- plain
Full reproduction pack available to customers and evaluators on request
All recorded numbers
Metric definitions, scope and statistics
| Metric | Value | Scope | Statistic | Sample count | Basis / source field | Run |
|---|---|---|---|---|---|---|
skipped cases | 4.00 count | machine total | single | not recorded | skipped_cases | |
correctness failures | 50.00 count | machine total | single | not recorded | correctness_failures | |
ttft median s | 0.06 s | per person | median | not recorded | ttft_median_s | |
workload total | 130.00 count | machine total | single | not recorded | workload_total | |
workload passed | 71.00 count | machine total | single | not recorded | workload_passed | |
api cases passed | 71.00 count | machine total | single | not recorded | api_cases_passed | |
ttft p95 s | 0.11 s | per person | p95 | not recorded | ttft_p95_s | |
completion p95 s | 0.77 s | per person | p95 | not recorded | completion_p95_s | |
api cases total | 124.00 count | machine total | single | not recorded | api_cases_total | |
completion median s | 0.23 s | per person | median | not recorded | completion_median_s | |
completeness failures | 11.00 count | machine total | single | not recorded | completeness_failures |
Data and citation
Skilak Keep evidence, run 70d5791f1fec47958088dcc3, measured 2026-10-07T05:52:52.779398+00:00. https://skilak.ai/keep/resources/evidence/r/70d5791f1fec47958088dcc3
