- query p50 secondsquery_p50_seconds · s
- 25.61 sper person · median · sample count not recorded
- query p95 secondsquery_p95_seconds · s
- 78.08 sper person · p95 · sample count not recorded
Answer checks
25/40 correct and cited
Method
Versioned fictional documents; authenticated Open WebUI upload, retrieval and chat with deterministic answer/citation grading. Recall and MRR at ten use labelled source passages; failed questions remain in the total denominator. A fresh system nonce prevents shared-prefix latency reuse.
Limits
Lab cohort, not customer qualification. Reranking and application BM25 are separate recipe arms. Qdrant storage excludes model/document storage. Exact LLM and service pins are retained in the private source record.
What ran
Exact identity
- Original publisher model
- EmbeddingGemma 2
- Publisher revision
914f7f89142e33e77833254d9c9b90c3cef7303b- Variant · format · precision
- float32 variant · safetensors · float32
- Weights SHA-256
- not recorded
- Engine · version · feature
- Transformers text CPU ·
5.19.0· plain - Hardware
- NVIDIA GB10, 128 GB unified memory
- Harness / workload
keep-rag-trial-v1 · 1- Workload suite · version
not recorded- Harness SHA-256
not recorded- Fixture SHA-256
not recorded
Workload shape
- answer model
- Qwen/Qwen3.6-35B-A3B
- answer variant
- nvfp4; fp8 KV; with multi-token prediction
- corpus sha256
- 51e025e244e9860a3a9a91d69f910bc75ddcbc33c717d4ca8e957501a5a15ae2
- dimensions
- 512
- dtype
- float32
- reranker model
- not recorded
Full reproduction pack available to customers and evaluators on request
All recorded numbers
Metric definitions, scope and statistics
| Metric | Value | Scope | Statistic | Sample count | Basis / source field | Run |
|---|---|---|---|---|---|---|
ingest seconds | 326.19 s | machine total | single | not recorded | ingest_seconds | |
citation correct | 25.00 count | machine total | single | not recorded | citation_correct | |
query p50 seconds | 25.61 s | per person | median | not recorded | query_p50_seconds | |
recall at 5 | 0.66 ratio | machine total | single | not recorded | recall_at_5 | |
retrieval p50 seconds | 3.22 s | per person | median | not recorded | retrieval_p50_seconds | |
retrieval p95 seconds | 6.23 s | per person | p95 | not recorded | retrieval_p95_seconds | |
qdrant bytes | 798,002,995.00 bytes | machine total | single | not recorded | qdrant_bytes | |
query p95 seconds | 78.08 s | per person | p95 | not recorded | query_p95_seconds | |
mrr | 0.60 ratio | machine total | single | not recorded | mrr | |
answer and citation correct | 25.00 count | machine total | single | not recorded | answer_and_citation_correct | |
completed questions | 40.00 count | machine total | single | not recorded | completed_questions | |
answer correct | 25.00 count | machine total | single | not recorded | answer_correct | |
questions | 40.00 count | machine total | single | not recorded | questions | |
indexed chunks | 548.00 count | machine total | single | not recorded | indexed_chunks | |
pages | 800.00 count | machine total | single | not recorded | pages | |
recall at 10 | 0.78 ratio | machine total | single | not recorded | recall_at_10 | |
retrieval questions | 32.00 count | per person | single | not recorded | retrieval_questions |
Data and citation
Skilak Keep evidence, run 641fc07e3f82447a87a1fd40, measured 2026-10-08T09:35:49.495129+00:00. https://skilak.ai/keep/resources/evidence/r/641fc07e3f82447a87a1fd40
