Answer checks
?/100 correct and cited
Method
Network-denied CPU inference, four threads, batch four, one warmup. Exact cosine search over 400 fictional passages and 100 labelled questions; one relevant passage per question. Full-rank MRR. Truncated vectors are re-normalized; dimension rows share one embedding pass.
Limits
Measured retrieval only, with no customer qualification threshold. Exact dense float32 array size excludes payloads and vector database overhead. One timing run; no reranking, ingestion, answer generation or GPU measurement.
What ran
Exact identity
- Original publisher model
- Qwen3-0.6B-Base
- Publisher revision
not recorded- Variant · format · precision
- FP32 variant · safetensors · FP32
- Weights SHA-256
- not recorded
- Engine · version · feature
- SentenceTransformers CPU ·
6.1.0· plain - Hardware
- Apple Silicon laptop CPU, 16 GiB unified memory
- Harness / workload
keep-embedding-eval-v1 · 1- Workload suite · version
not recorded- Harness SHA-256
not recorded- Fixture SHA-256
not recorded
Workload shape
- Configured window (tokens)
- 32768
- corpus sha256
- d7fb630b25a7d5128c44491d1a8f21bde479711f79520f1ce95ea8bd8a0ae1f6
- dimensions
- 1024
- dtype
- float32
Full reproduction pack available to customers and evaluators on request
All recorded numbers
Metric definitions, scope and statistics
| Metric | Value | Scope | Statistic | Sample count | Basis / source field | Run |
|---|---|---|---|---|---|---|
passages per second | 1.64 passages/s | machine total | single | not recorded | passages_per_second | |
query seconds | 6.86 s | per person | single | not recorded | query_seconds | |
recall at 5 | 0.90 ratio | machine total | single | not recorded | recall_at_5 | |
mrr | 0.79 ratio | machine total | single | not recorded | mrr | |
questions | 100.00 count | machine total | single | not recorded | questions | |
passage seconds | 244.24 s | machine total | single | not recorded | passage_seconds | |
vector bytes | 1,638,400.00 bytes | machine total | single | not recorded | vector_bytes | |
passages | 400.00 count | machine total | single | not recorded | passages | |
index file bytes | 1,638,528.00 bytes | machine total | single | not recorded | index_file_bytes | |
recall at 10 | 0.94 ratio | machine total | single | not recorded | recall_at_10 | |
truncated passages | 0.00 count | machine total | single | not recorded | truncated_passages |
Data and citation
Skilak Keep evidence, run 96f87869701b4783ba4bf5d6, measured 2026-10-07T14:37:01.394807+00:00. https://skilak.ai/keep/resources/evidence/r/96f87869701b4783ba4bf5d6
