Fix benchmark ground truth for disk_index_siftsmall 10-query workload#1261
Fix benchmark ground truth for disk_index_siftsmall 10-query workload#1261partychen wants to merge 2 commits into
Conversation
There was a problem hiding this comment.
Pull request overview
Updates the ground-truth fixture used by the disk_index_siftsmall_learn_256pts_data.fbin + disk_index_sample_query_10pts.fbin 10-query benchmark workload so that exhaustive/flat-search benchmarks compare against brute-force KNN results (fixing the recall < 1.0 mismatch described in #1172).
Changes:
- Replaced the Git LFS object for
disk_index_10pts_idx_uint32_truth_search_res.binwith newly generated brute-force ground truth. - Increased the ground-truth payload size (408 → 808 bytes), consistent with
compute_groundtruth’s on-disk format ofMetadata + ids + distancesfor 10 queries × K=10.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
|
Thanks! Are there any other workflows using the older file? Asking to check if it might break any workflow. |
Thanks for calling this out. I checked all references to the existing file. The disk-provider regression tests still use the older The benchmark examples were the other consumers. They now reference the new So the existing file remains unchanged for workflows that depend on its current |
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #1261 +/- ##
==========================================
- Coverage 90.16% 90.16% -0.01%
==========================================
Files 509 509
Lines 97234 97238 +4
==========================================
+ Hits 87672 87675 +3
- Misses 9562 9563 +1
Flags with carried forward coverage won't be shown. Click here to find out more. 🚀 New features to boost your workflow:
|
Reference Issues/PRs
Fixes #1172.
What does this implement/fix? Briefly explain your changes.
This PR adds an exact brute-force ground-truth file for benchmark examples using
the
disk_index_siftsmall_learn_256pts_data.fbinanddisk_index_sample_query_10pts.fbinworkload.test_data/disk_index_search/disk_index_10pts_idx_uint32_exact_ground_truth.bin,generated using
compute_groundtruth.ground-truth file.
This fixes the recall mismatch where
flat-searchcould report recall below1.0 because the benchmark configurations previously referenced results that
were not exact brute-force ground truth.
Any other comments?
Validated with:
cargo test --locked --workspace