- query p50 secondsquery_p50_seconds · s
- 31.31 sper person · median · sample count not recorded
- query p95 secondsquery_p95_seconds · s
- 66.64 sper person · p95 · sample count not recorded
Answer checks
27/40 correct and cited
Method
Versioned fictional documents; authenticated Open WebUI upload, retrieval and chat with deterministic answer/citation grading. Recall and MRR at ten use labelled source passages; failed questions remain in the total denominator. A fresh system nonce prevents shared-prefix latency reuse.
Limits
Lab cohort, not customer qualification. Reranking and application BM25 are separate recipe arms. Qdrant storage excludes model/document storage. Exact LLM and service pins are retained in the private source record.
What ran
Exact identity
- Original publisher model
- EmbeddingGemma 2
- Publisher revision
914f7f89142e33e77833254d9c9b90c3cef7303b- Variant · format · precision
- float32 variant · safetensors · float32
- Weights SHA-256
- not recorded
- Engine · version · feature
- Transformers text CPU ·
5.19.0· plain - Hardware
- NVIDIA GB10, 128 GB unified memory
- Harness / workload
keep-rag-trial-v1 · 1- Workload suite · version
not recorded- Harness SHA-256
not recorded- Fixture SHA-256
not recorded
Workload shape
- answer model
- Qwen/Qwen3.6-35B-A3B
- answer variant
- nvfp4; fp8 KV; with multi-token prediction
- corpus sha256
- 51e025e244e9860a3a9a91d69f910bc75ddcbc33c717d4ca8e957501a5a15ae2
- dimensions
- 512
- dtype
- float32
- query generation
- on
- reranker model
- BAAI/bge-reranker-v2-m3
- Thinking
- off
Full reproduction pack available to customers and evaluators on request
All recorded numbers
Metric definitions, scope and statistics
| Metric | Value | Scope | Statistic | Sample count | Basis / source field | Run |
|---|---|---|---|---|---|---|
ingest seconds | 322.24 s | machine total | single | not recorded | ingest_seconds | |
citation correct | 27.00 count | machine total | single | not recorded | citation_correct | |
query p50 seconds | 31.31 s | per person | median | not recorded | query_p50_seconds | |
recall at 5 | 0.69 ratio | machine total | single | not recorded | recall_at_5 | |
retrieval p50 seconds | 5.23 s | per person | median | not recorded | retrieval_p50_seconds | |
retrieval p95 seconds | 7.16 s | per person | p95 | not recorded | retrieval_p95_seconds | |
qdrant bytes | 798,011,230.00 bytes | machine total | single | not recorded | qdrant_bytes | |
query p95 seconds | 66.64 s | per person | p95 | not recorded | query_p95_seconds | |
mrr | 0.69 ratio | machine total | single | not recorded | mrr | |
answer and citation correct | 27.00 count | machine total | single | not recorded | answer_and_citation_correct | |
completed questions | 39.00 count | machine total | single | not recorded | completed_questions | |
answer correct | 27.00 count | machine total | single | not recorded | answer_correct | |
questions | 40.00 count | machine total | single | not recorded | questions | |
indexed chunks | 833.00 count | machine total | single | not recorded | indexed_chunks | |
pages | 800.00 count | machine total | single | not recorded | pages | |
recall at 10 | 0.75 ratio | machine total | single | not recorded | recall_at_10 | |
retrieval questions | 32.00 count | per person | single | not recorded | retrieval_questions |
Data and citation
Skilak Keep evidence, run 740e0bebe94f4d1f87f8506d, measured 2026-10-08T17:22:27.944305+00:00. https://skilak.ai/keep/resources/evidence/r/740e0bebe94f4d1f87f8506d
