Runs on this setup
Earlier runs keep their own engine versions and outcomes. Workload shapes stay separate.
History and measured sweep
| Date | Shape | Each person · median | Machine total | First word · median | Result | Run |
|---|---|---|---|---|---|---|
2026-10-09 | 2 people | not recorded | not recorded | not recorded | Partial | |
2026-10-09 | 2 people | not recorded | not recorded | not recorded | Partial |
Answer checks
28/32 tasks
Method
Sixteen fictional exact-JSON tasks per user-count level, no retries and no warmup.
Limits
Four failed supplied-context answers are retained. No retrieval or permission test. Manifest is operator evidence; no customer or managed-service qualification.
What did not work
- 28/32 tasks passed.
What ran
Exact identity
- Original publisher model
- Qwen3-8B-Base
- Publisher revision
49e3418fbbbca6ecbdf9608b4d22e5a407081db4- Variant · format · precision
- Q4_K_M GGUF variant · GGUF · Q4_K_M
- Weights SHA-256
- not recorded
- Engine · version · feature
- llama.cpp Metal ·
b11182· plain - Hardware
- Apple M6 Mac mini, 16 GiB unified memory
- Harness / workload
not recorded- Workload suite · version
not recorded- Harness SHA-256
not recorded- Fixture SHA-256
not recorded
Workload shape
- People in this run
- 2
- Configured window (tokens)
- 4096
- KV cache precision
- q8_0
- Output token ceiling
- 512
- Thinking
- off
Full reproduction pack available to customers and evaluators on request
All recorded numbers
Metric definitions, scope and statistics
| Metric | Value | Scope | Statistic | Sample count | Basis / source field | Run |
|---|---|---|---|---|---|---|
workload total | 32.00 count | machine total | single | not recorded | workload_total | |
workload passed | 28.00 count | machine total | single | not recorded | workload_passed |
Data and citation
Skilak Keep evidence, run 93c24d3394ea4bcf8f82d006, measured 2026-10-09. https://skilak.ai/keep/resources/evidence/r/93c24d3394ea4bcf8f82d006
