The bar, check by check
5 agents at once, 20 tasks, 2 attempts each. Every check must hold for the run to meet the bar.
| Observed | Bar | Result | Run |
|---|---|---|---|
Tasks solved: 38/40 (95%) | at least 85% | Met | |
Weakest category: Bug fixes 10/12 | every category at least 60% | Met | |
Malformed tool calls: 0 of 702 | at most 2% | Met | |
Engine, sandbox or harness errors: 0 | 0 | Met | |
Agents stuck repeating a call: 0 | 0 | Met | |
Most requests in flight at once: 5 | all 5 agents | Met | |
Attempts recorded: 40 | 20 tasks × 2 | Met | |
First word, 95% of turns: 2.2 s | at most 30.0 s | Met | |
Writing speed per reply, median: 42.9 tok/s | at least 20 tok/s | Met | |
Time per task, 95% of tasks: 10.9 min | at most 15.0 min | Met |
Every task
20 tasks · 18 solved on every attempt
| Task | Kind of work | Solved | Run |
|---|---|---|---|
ask-dialect-count | Questions about the code | 2/2 | |
ask-locate-null-ordering | Questions about the code | 2/2 | |
ask-parser-defaults | Questions about the code | 2/2 | |
ask-transpile-snowflake | Questions about the code | 2/2 | |
bug-not-in-null | Bug fixes | 2/2 | |
bug-nulls-last | Bug fixes | 1/2 | |
bug-order-offset | Bug fixes | 2/2 | |
bug-postgres-month-format | Bug fixes | 2/2 | |
bug-string-agg-order | Bug fixes | 1/2 | |
bug-substring-negative | Bug fixes | 2/2 | |
data-customers-dedupe | Data jobs | 2/2 | |
data-monthly-revenue | Data jobs | 2/2 | |
data-oversold | Data jobs | 2/2 | |
data-suppliers-clean | Data jobs | 2/2 | |
data-top-skus-sql | Data jobs | 2/2 | |
data-transpile-folder | Data jobs | 2/2 | |
feat-cli-file-output | New features | 2/2 | |
feat-median | New features | 2/2 | |
feat-nullif-greatest | New features | 2/2 | |
multi-revenue-report | Multi-step job | 2/2 |
Method
Agent team bench agent-team-v1: 5 agents at once, each in its own copy of a pinned 147K-line open-source Python repository plus generated fictional shop data. 20 tasks (bug fixes, features, code questions, data reports, one multi-step job) x 2 repetitions. OpenAI tool calls (list, read, search, write, edit, sandboxed shell, finish), checked against each tool schema. Hidden graders read the final files and answers, never the agent's claim; code changes must also keep 424 repository tests passing. Commands run in a network-denied container. Limits per task: 50 model turns, 20 minutes, 8192 output tokens per turn. Temperature 0.6, top-p 0.95, seeded requests. Before the model started, every task failed in its planted state and passed with its reference solution in the same sandbox.
Limits
Lab workload, not a KTS-1 block and not customer acceptance. One repository and one fictional shop; Keep's own minimal agent loop, not a product coding harness. The model may have seen this open-source repository in training; the planted bugs and shop data are new. Engines do not guarantee identical output under concurrent batching. Original publisher name with variant; exact variant repository and revision retained privately.
What ran
Exact identity
- Original publisher model
- Qwen3.6-35B-A3B
- Publisher revision
995ad96eacd98c81ed38be0c5b274b04031597b0- Variant · format · precision
- NVFP4 variant · safetensors · NVFP4
- Weights SHA-256
- not recorded
- Engine · version · feature
- NVIDIA NGC vLLM ·
0.29.0+5013de39.dev· with multi-token prediction - Hardware
- NVIDIA GB10, 128 GB unified memory
- Harness / workload
keep-agent-team-result-v1 · 1- Workload suite · version
agent-team-v1 · 1- Harness SHA-256
484ed3e2b82f09ee19c87be819253811cb273628abfb3c9471a59746605e537d- Fixture SHA-256
6451aabd1c9994d0f73355c962a5eaf9258381b3f2554e53db916cd9a605552a
Workload shape
- People in this run
- 5
- Configured window (tokens)
- 262144
- KV cache precision
- fp8
- Output token ceiling
- 8192
- Feature
- with multi-token prediction
Full reproduction pack available to customers and evaluators on request
All recorded numbers
Metric definitions, scope and statistics
| Metric | Value | Scope | Statistic | Sample count | Basis / source field | Run |
|---|---|---|---|---|---|---|
tasks passed | 38.00 count | machine total | single | not recorded | agent_team.tasks_passed | |
tasks total | 40.00 count | machine total | single | not recorded | agent_team.tasks_total | |
tool calls | 702.00 count | machine total | single | not recorded | agent_team.tool_calls | |
protocol errors | 0.00 count | machine total | single | not recorded | agent_team.protocol_errors | |
engine errors | 0.00 count | machine total | single | not recorded | agent_team.engine_errors | |
loops | 0.00 count | machine total | single | not recorded | agent_team.loops | |
model requests | 599.00 count | machine total | single | not recorded | agent_team.model_requests | |
peak concurrent requests | 5.00 count | machine total | max | not recorded | agent_team.peak_concurrent_requests | |
mean concurrent requests | 2.96 count | machine total | mean | not recorded | agent_team.mean_concurrent_requests | |
ttft p50 s | 1.02 s | per person | median | not recorded | agent_team.ttft_p50_s | |
ttft p95 s | 2.21 s | per person | p95 | not recorded | agent_team.ttft_p95_s | |
per request decode p50 tps | 42.90 tok/s | per person | median | not recorded | agent_team.per_request_decode_p50_tps | |
task p50 seconds | 66.93 s | per person | median | not recorded | agent_team.task_p50_seconds | |
task p95 seconds | 656.02 s | per person | p95 | not recorded | agent_team.task_p95_seconds | |
max prompt tokens | 51,639.00 tokens | machine total | max | not recorded | agent_team.max_prompt_tokens | |
wall seconds | 2,083.33 s | machine total | single | not recorded | agent_team.wall_seconds |
Data and citation
Skilak Keep evidence, run 1bdb1c2c257c141430090354, measured 2026-10-21T08:07:59.563191+00:00. https://skilak.ai/keep/resources/evidence/r/1bdb1c2c257c141430090354
