SkilakKeep
Request a pilot

Report · resources

Answer quality

Qwen3.8-27B · Mirai S trellis with Q8_0 MTP GGUF variant · Mirai S CUDA dev · with multi-token prediction

Measured results / Answer quality / Run 78abd8c8e6e94c80a00efa84

Measured 2026-10-09 14:00 UTC · Record date 2026-10-09

Partial. 119/122 tasks117/120 API cases

First word afterttft_median_s · s
0.64 sper person · median · sample count not recorded

Answer checks

119/122 tasks117/120 API cases

API suite category outcomes
CategoryPassed / totalSkippedRun
api
6/8
0
code
16/16
0
format
12/12
0
multi_turn
8/8
0
multilingual
6/6
0
numeric
15/16
0
reasoning
0/0
4
refusal
6/6
0
structured_extraction
16/16
0
supplied_context
16/16
0
tools
8/8
0
untrusted_text
8/8
0
vision
0/0
4

Method

Deterministic API suite core-v2; SHA256 0ad0b13a94ad74bcdefe99d12d796b175f56e5777d9642a1dd4c0f170d708398; no model judge; no retries; unchanged endpoint cache; latency excludes local grading; paired calls counted separately. Client concurrency; configured sequence limit 1.

Limits

Bounded fictional tasks only. Unsupported cases are separately counted, never passes. Original publisher name with variant; exact variant repository and revision retained privately. Not customer acceptance.

What did not work

  • 119/122 tasks passed.
  • 117/120 API cases passed.

What ran

Exact identity

Original publisher model
Qwen3.8-27B
Publisher revision
not recorded
Variant · format · precision
Mirai S trellis with Q8_0 MTP GGUF variant · GGUF · Mirai S trellis with Q8_0 MTP
Weights SHA-256
not recorded
Engine · version · feature
Mirai S CUDA · 0.2.0-dev-vector-disabled · with multi-token prediction
Hardware
NVIDIA GeForce RTX 3060, 12 GB VRAM
Harness / workload
keep-api-suite-run-v1 · 1
Workload suite · version
core-v2 · 2
Harness SHA-256
not recorded
Fixture SHA-256
0ad0b13a94ad74bcdefe99d12d796b175f56e5777d9642a1dd4c0f170d708398

Workload shape

People in this run
1
Configured window (tokens)
262144
KV cache precision
q8_0
Feature
with multi-token prediction

Full reproduction pack available to customers and evaluators on request

All recorded numbers

Metric definitions, scope and statistics
All aggregate numeric result fields in this run
MetricValueScopeStatisticSample countBasis / source fieldRun
skipped cases
8.00 count
machine total
single
not recorded
skipped_cases
correctness failures
2.00 count
machine total
single
not recorded
correctness_failures
ttft median s
0.64 s
per person
median
not recorded
ttft_median_s
workload total
122.00 count
machine total
single
not recorded
workload_total
workload passed
119.00 count
machine total
single
not recorded
workload_passed
api cases passed
117.00 count
machine total
single
not recorded
api_cases_passed
ttft p95 s
1.10 s
per person
p95
not recorded
ttft_p95_s
completion p95 s
1.80 s
per person
p95
not recorded
completion_p95_s
api cases total
120.00 count
machine total
single
not recorded
api_cases_total
completion median s
0.89 s
per person
median
not recorded
completion_median_s
completeness failures
1.00 count
machine total
single
not recorded
completeness_failures

Data and citation

Download this run · OLIR JSONAll numbers · CSVOLIR schema
Skilak Keep evidence, run 78abd8c8e6e94c80a00efa84, measured 2026-10-09T14:00:47.877196+00:00.
https://skilak.ai/keep/resources/evidence/r/78abd8c8e6e94c80a00efa84

Ready to scope it?

Start with one workload.

Tell us who uses it, what it can read, and what must stay inside. We will map the build.

Request a pilot