SkilakKeep
Request a pilot

Report · resources

Many people. Full context.

Our specialty: long inputs, many people at once. Measured on real hardware, with a run behind every result.

Updated · 93 setups · 284 runsJSONCSVSchemaChanges

Keep resources / Measured results

Apple M6 Mac mini, 16 GiB unified memory. Memory figures are whole-host unless marked.

People at once × filled context

Room for the whole team.

KTS-1 runs in progress. Published grids will appear here as standard test suites complete.

Method and terms · Export grid →

Headline results

One job, one measured run. Text jobs use the current default where one is published.

KTS-1 runs in progress. Published headline results will appear here as standard test suites complete. Historical runs remain in the archive below.

Historical archive

Historical pre-standard observations and earlier exploratory sweeps on this hardware, with original measurements and setup history. Each run page contains its workload shape, request rows and provenance links.

Charts and answer checks

KTS-1 runs in progress. Comparison charts and answer checks will appear here as standard test suites complete.

Chat and code7 setups · 0 passed
Chat and code, current setups
SetupWorkload and settingsMeasured resultsResultRun
Qwen3-8B-Base · Q4_K_M GGUF variant · llama.cpp b11182
concurrency: 2 · configured window: 5120 · kv dtype: q8_0 · thinking: false
First word after: 11.27 sper person · median · sample count not recorded
chat decode tps: 21.38 tok/sper person · mean · sample count not recorded
Whole-host unavailable peak: 14.34 GiBmachine total · max · sample count not recorded
Not graded
Qwen3-8B-Base · Q4_K_M GGUF variant · llama.cpp b11182
concurrency: null · configured window: 9216 · kv dtype: q8_0 · thinking: false
Whole-host unavailable peak: 14.44 GiBmachine total · max · sample count not recorded
Failed
Qwen3-8B-Base · Q4_K_M GGUF variant · llama.cpp b11182
concurrency: 2 · configured window: 5120 · kv dtype: q8_0 · thinking: false
First word after: 11.26 sper person · median · sample count not recorded
code ttft mean s: 11.28 sper person · mean · sample count not recorded
code decode tps: 21.52 tok/sper person · mean · sample count not recorded
Whole-host unavailable peak: 14.34 GiBmachine total · max · sample count not recorded
Not graded
Qwen3-8B-Base · Q4_K_M GGUF variant · llama.cpp b11182
concurrency: null · configured window: 18432 · kv dtype: q8_0 · thinking: false
Whole-host unavailable peak: 14.71 GiBmachine total · max · sample count not recorded
Failed
Qwen3-8B-Base · Q4_K_M GGUF variant · llama.cpp b11182
concurrency: 2 · configured window: 4096 · kv dtype: q8_0 · thinking: false
First word after: 0.14 sper person · median · sample count not recorded
code ttft mean s: 0.14 sper person · mean · sample count not recorded
code decode tps: 28.35 tok/sper person · mean · sample count not recorded
Whole-host unavailable peak: 13.37 GiBmachine total · max · sample count not recorded
Not graded
Qwen3-8B-Base · Q4_K_M GGUF variant · llama.cpp b11182
concurrency: 2 · configured window: 4096 · kv dtype: q8_0 · thinking: false
workload total: 32 countmachine total · single · sample count not recorded
workload passed: 28 countmachine total · single · sample count not recorded
Partial
Qwen3-8B-Base · Q4_K_M GGUF variant · llama.cpp b11182
concurrency: 2 · configured window: 4096 · kv dtype: q8_0 · thinking: false
First word after: 0.14 sper person · median · sample count not recorded
chat decode tps: 28.36 tok/sper person · mean · sample count not recorded
Whole-host unavailable peak: 13.37 GiBmachine total · max · sample count not recorded
Not graded
Embeddings6 setups · 0 passed
Embeddings, current setups
SetupWorkload and settingsMeasured resultsResultRun
EmbeddingGemma 2 · FP32 variant · Transformers CPU 5.19
configured window: 262144 · dimensions: 512
passages per second: 12.26 passages/smachine total · single · sample count not recorded
recall at 5: 1 ratiomachine total · single · sample count not recorded
mrr: 0.97 ratiomachine total · single · sample count not recorded
recall at 10: 1 ratiomachine total · single · sample count not recorded
truncated passages: 0 countmachine total · single · sample count not recorded
Not graded
EmbeddingGemma 2 · FP32 variant · Transformers CPU 5.19
configured window: 262144 · dimensions: 256
passages per second: 12.26 passages/smachine total · single · sample count not recorded
recall at 5: 0.99 ratiomachine total · single · sample count not recorded
mrr: 0.93 ratiomachine total · single · sample count not recorded
recall at 10: 1 ratiomachine total · single · sample count not recorded
truncated passages: 0 countmachine total · single · sample count not recorded
Not graded
EmbeddingGemma 2 · FP32 variant · Transformers CPU 5.19
configured window: 262144 · dimensions: 128
passages per second: 12.26 passages/smachine total · single · sample count not recorded
recall at 5: 1 ratiomachine total · single · sample count not recorded
mrr: 0.91 ratiomachine total · single · sample count not recorded
recall at 10: 1 ratiomachine total · single · sample count not recorded
truncated passages: 0 countmachine total · single · sample count not recorded
Not graded
EmbeddingGemma 2 · FP32 variant · Transformers CPU 5.19
configured window: 262144 · dimensions: 768
passages per second: 12.26 passages/smachine total · single · sample count not recorded
recall at 5: 1 ratiomachine total · single · sample count not recorded
mrr: 0.97 ratiomachine total · single · sample count not recorded
recall at 10: 1 ratiomachine total · single · sample count not recorded
truncated passages: 0 countmachine total · single · sample count not recorded
Not graded
Qwen3-0.6B-Base · FP32 variant · SentenceTransformers CPU 6.1
configured window: 32768 · dimensions: 1024
passages per second: 1.64 passages/smachine total · single · sample count not recorded
recall at 5: 0.9 ratiomachine total · single · sample count not recorded
mrr: 0.79 ratiomachine total · single · sample count not recorded
recall at 10: 0.94 ratiomachine total · single · sample count not recorded
truncated passages: 0 countmachine total · single · sample count not recorded
Not graded
all-MiniLM-L6-v2 · FP32 variant · SentenceTransformers CPU 6.1
configured window: 256 · dimensions: 384
passages per second: 152.1 passages/smachine total · single · sample count not recorded
recall at 5: 0.66 ratiomachine total · single · sample count not recorded
mrr: 0.48 ratiomachine total · single · sample count not recorded
recall at 10: 0.74 ratiomachine total · single · sample count not recorded
truncated passages: 40 countmachine total · single · sample count not recorded
Not graded

Method and terms

How to read and compare these runs

Writing speed (decode, tok/s) measures output after the first word. Concurrent streams may pause during other people’s prefill. First word after includes scheduling and reading the prompt.

Each person describes one request’s experience; median and mean are labelled. Machine total (aggregate) measures the whole cohort over a stated window. Per-person rates are never summed to estimate it.

New (cold) means no reusable prompt KV state. Already read (warm) means the prefix was processed earlier in the same server lifetime. Warm results never supply reading speed; cached-token counts stay unknown when the engine did not record them.

The grid isolates each model, variant, engine version and generic feature. Cold columns use distinct documents; warm cells label shared inputs. Omitted shapes remain in the historical archive. A cell links to one run, never a blend of setups. The 300-second first-word reference is independent of exact-answer and completion checks. A linked note sends runs with missed checks to the bottom ledger.

Configured window is the server allowance. Filled input is the measured prompt, including the chat template where token usage is available. Completion, exact answer checks, decode overlap and the first-word reference are independent gates.

These are finite lab observations. They establish neither general answer quality nor customer delivery qualification. The workload and public method are listed on each run.

Data and changes

Download the published dataset or use Open Local Inference Result (OLIR) for your own runs. One record describes one run; every numeric metric carries scope, statistic, unit and its original field.

Full dataset · JSONOne row per number · CSVAll runs · OLIR JSONOLIR schemaOLIR formatChangelog · JSONDataset SHA-256
Publication history

Cite a single result using the citation on its run page. Original measurement dates are preserved; republishing does not refresh a measurement.

What did not work

  • Structured tasks: 2 runs missed the full task grade.
  • Chat and code: 2 attempts missed their run checks.
Full list · 4 failed or partial runs
Failed and partial runs and startup attempts, newest first
DateResultSetupTriedWhat happenedRun
2026-10-09
Partial
Qwen3-8B-Base · Q4_K_M GGUF variant · llama.cpp b11182
Structured tasks
28/32 tasks passed.
2026-10-09
Failed
Qwen3-8B-Base · Q4_K_M GGUF variant · llama.cpp b11182
Chat and code
Incomplete cohort; two-user filled-context speed not assessed in this attempt.
2026-10-09
Partial
Qwen3-8B-Base · Q4_K_M GGUF variant · llama.cpp b11182
Structured tasks
28/32 tasks passed.
2026-10-09
Failed
Qwen3-8B-Base · Q4_K_M GGUF variant · llama.cpp b11182
Chat and code
Incomplete cohort; two-user filled-context speed not assessed in this attempt.

Ready to scope it?

Start with one workload.

Tell us who uses it, what it can read, and what must stay inside. We will map the build.

Request a pilot