Answer checks
?/100 correct and cited
Method
Network-denied CPU inference, four threads, batch four, one warmup. Exact cosine search over 400 fictional passages and 100 labelled questions; one relevant passage per question. Full-rank MRR. Truncated vectors are re-normalized; dimension rows share one embedding pass.
Limits
Measured retrieval only, with no customer qualification threshold. Exact dense float32 array size excludes payloads and vector database overhead. One timing run; no reranking, ingestion, answer generation or GPU measurement.
What ran
Exact identity
- Original publisher model
- EmbeddingGemma 2
- Publisher revision
914f7f89142e33e77833254d9c9b90c3cef7303b- Variant · format · precision
- FP32 variant · safetensors · FP32
- Weights SHA-256
- not recorded
- Engine · version · feature
- Transformers CPU ·
5.19.0· plain - Hardware
- Apple Silicon laptop CPU, 16 GiB unified memory
- Harness / workload
keep-embedding-eval-v1 · 1- Workload suite · version
not recorded- Harness SHA-256
not recorded- Fixture SHA-256
not recorded
Workload shape
- Configured window (tokens)
- 262144
- corpus sha256
- d7fb630b25a7d5128c44491d1a8f21bde479711f79520f1ce95ea8bd8a0ae1f6
- dimensions
- 256
- dtype
- float32
Full reproduction pack available to customers and evaluators on request
All recorded numbers
Metric definitions, scope and statistics
| Metric | Value | Scope | Statistic | Sample count | Basis / source field | Run |
|---|---|---|---|---|---|---|
passages per second | 12.26 passages/s | machine total | single | not recorded | passages_per_second | |
query seconds | 2.57 s | per person | single | not recorded | query_seconds | |
recall at 5 | 0.99 ratio | machine total | single | not recorded | recall_at_5 | |
mrr | 0.93 ratio | machine total | single | not recorded | mrr | |
questions | 100.00 count | machine total | single | not recorded | questions | |
passage seconds | 32.63 s | machine total | single | not recorded | passage_seconds | |
vector bytes | 409,600.00 bytes | machine total | single | not recorded | vector_bytes | |
passages | 400.00 count | machine total | single | not recorded | passages | |
index file bytes | 409,728.00 bytes | machine total | single | not recorded | index_file_bytes | |
recall at 10 | 1.00 ratio | machine total | single | not recorded | recall_at_10 | |
truncated passages | 0.00 count | machine total | single | not recorded | truncated_passages |
Data and citation
Skilak Keep evidence, run 0791e5c1cbba43cabcfa102e, measured 2026-10-07T14:39:45.948052+00:00. https://skilak.ai/keep/resources/evidence/r/0791e5c1cbba43cabcfa102e
