Skip to content

Fix benchmark ground truth for disk_index_siftsmall 10-query workload#1261

Open
partychen wants to merge 2 commits into
microsoft:mainfrom
partychen:juchen/fix-test-data
Open

Fix benchmark ground truth for disk_index_siftsmall 10-query workload#1261
partychen wants to merge 2 commits into
microsoft:mainfrom
partychen:juchen/fix-test-data

Conversation

@partychen

@partychen partychen commented Jul 17, 2026

Copy link
Copy Markdown
  • Does this PR have a descriptive title that could go in our release notes?
  • Does this PR add any new dependencies?
  • Does this PR modify any existing APIs?
  • Is the change to the API backwards compatible?
  • Should this result in any changes to our documentation, either updating existing docs or adding new ones?

Reference Issues/PRs

Fixes #1172.

What does this implement/fix? Briefly explain your changes.

This PR adds an exact brute-force ground-truth file for benchmark examples using
the disk_index_siftsmall_learn_256pts_data.fbin and
disk_index_sample_query_10pts.fbin workload.

  • Added
    test_data/disk_index_search/disk_index_10pts_idx_uint32_exact_ground_truth.bin,
    generated using compute_groundtruth.
  • Updated the applicable benchmark example configurations to use the new exact
    ground-truth file.
  • Added no dependencies and made no API changes.

This fixes the recall mismatch where flat-search could report recall below
1.0 because the benchmark configurations previously referenced results that
were not exact brute-force ground truth.

Any other comments?

Validated with:

  • cargo test --locked --workspace
  • All benchmark example JSON files parse successfully.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Updates the ground-truth fixture used by the disk_index_siftsmall_learn_256pts_data.fbin + disk_index_sample_query_10pts.fbin 10-query benchmark workload so that exhaustive/flat-search benchmarks compare against brute-force KNN results (fixing the recall < 1.0 mismatch described in #1172).

Changes:

  • Replaced the Git LFS object for disk_index_10pts_idx_uint32_truth_search_res.bin with newly generated brute-force ground truth.
  • Increased the ground-truth payload size (408 → 808 bytes), consistent with compute_groundtruth’s on-disk format of Metadata + ids + distances for 10 queries × K=10.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

@harsha-simhadri

Copy link
Copy Markdown
Contributor

Thanks! Are there any other workflows using the older file? Asking to check if it might break any workflow.

@partychen

Copy link
Copy Markdown
Author

Thanks! Are there any other workflows using the older file? Asking to check if it might break any workflow.

Thanks for calling this out. I checked all references to the existing file.

The disk-provider regression tests still use the older
disk_index_10pts_idx_uint32_truth_search_res.bin file and expect its contents
to remain unchanged.

The benchmark examples were the other consumers. They now reference the new
disk_index_10pts_idx_uint32_exact_ground_truth.bin file, which contains the
brute-force KNN ground truth.

So the existing file remains unchanged for workflows that depend on its current
results, while benchmark recall calculations use the new exact ground-truth
file. I also ran cargo test --locked --workspace, and all tests passed.

@codecov-commenter

codecov-commenter commented Jul 20, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 90.16%. Comparing base (62e201a) to head (0cfc89f).
⚠️ Report is 1 commits behind head on main.

Additional details and impacted files

Impacted file tree graph

@@            Coverage Diff             @@
##             main    #1261      +/-   ##
==========================================
- Coverage   90.16%   90.16%   -0.01%     
==========================================
  Files         509      509              
  Lines       97234    97238       +4     
==========================================
+ Hits        87672    87675       +3     
- Misses       9562     9563       +1     
Flag Coverage Δ
miri 90.16% <ø> (-0.01%) ⬇️
unittests 89.83% <ø> (-0.01%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.
see 5 files with indirect coverage changes

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

diskann-benchmark\example\flat-index.json uses graph ground truth which is not equal

4 participants