Add algorithm and benchmark for filtered range search by magdalendobson · Pull Request #1228 · microsoft/DiskANN

magdalendobson · 2026-07-06T15:37:56Z

This PR adds an algorithm for filtered range search to the DiskANN repository. Details of the algorithm, along with the experiments supporting it, are at this Wiki page.

This PR:

Adds the new algorithm to diskann, along with integration tests.
Adds the new algorithm as an option in diskann-benchmark, along with an integration test.
Adds filtered range groundtruth files in test_data.

Some enhancements/fixes to range search along the way:

Added range groundtruth in test_data for yfcc.
Fixed a bug where range search was not always honoring the maximum number of returned results.
Added integration tests for range search making sure a set maximum number of results is actually respected.

Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>

Copilot

Pull request overview

This PR introduces filtered range search into the diskann search stack and wires it into diskann-benchmark, along with new groundtruth artifacts and integration tests. It also tightens unfiltered range-search behavior to better respect max_returned and adds regression tests to prevent future violations.

Changes:

Added FilteredRange search type to diskann (new implementation + exports) and created dedicated integration tests/baselines.
Extended diskann-benchmark / diskann-benchmark-core to accept a new "filtered-range" search phase and run end-to-end integration tests.
Added new (LFS-backed) range-search groundtruth files and test inputs.

Reviewed changes

Copilot reviewed 22 out of 32 changed files in this pull request and generated 3 comments.

Show a summary per file

File	Description
test_data/yfcc/groundtruth_rad_75000.rangeres	Adds YFCC range-search groundtruth pointer (LFS).
test_data/yfcc/groundtruth_rad_110000_filtered.rangeres	Adds YFCC filtered-range groundtruth pointer (LFS).
test_data/disk_index_search/groundtruth_rad_125000_filtered.rangeres	Adds filtered-range groundtruth pointer (LFS) for benchmark integration test data.
diskann/test/generated/graph/test/cases/range_search/max_results_respected_means_no_second_round.json	New baseline for range-search `max_returned` behavior.
diskann/test/generated/graph/test/cases/range_search/max_results_respected_and_second_round_triggered.json	New baseline for second-round triggering when `max_returned` allows it.
diskann/test/generated/graph/test/cases/filtered_range_search/two_round_search.json	New filtered-range baseline.
diskann/test/generated/graph/test/cases/filtered_range_search/max_results_respected_means_no_second_round.json	New filtered-range baseline.
diskann/test/generated/graph/test/cases/filtered_range_search/max_results_respected_and_second_round_triggered.json	New filtered-range baseline.
diskann/test/generated/graph/test/cases/filtered_range_search/inner_radius_filtering.json	New filtered-range baseline for inner radius behavior.
diskann/test/generated/graph/test/cases/filtered_range_search/divisible_by_four_filter_second_round_triggered.json	New filtered-range baseline for selective filter + second round.
diskann/test/generated/graph/test/cases/filtered_range_search/divisible_by_four_filter_no_second_round_from_max_results.json	New filtered-range baseline for selective filter + `max_returned` interaction.
diskann/test/generated/graph/test/cases/filtered_range_search/divisible_by_four_filter_no_second_round_from_l_search.json	New filtered-range baseline for selective filter + larger `starting_l`.
diskann/test/generated/graph/test/cases/filtered_range_search/basic_range_search.json	New filtered-range baseline for basic search.
diskann/src/graph/test/cases/range_search.rs	Exposes shared helpers/baseline struct and adds new `max_returned` regression tests.
diskann/src/graph/test/cases/mod.rs	Registers new filtered-range test module.
diskann/src/graph/test/cases/filtered_range_search.rs	Adds integration tests and baselines for filtered range search.
diskann/src/graph/search/range_search.rs	Ensures second round + frontier expansion honor `max_returned`.
diskann/src/graph/search/mod.rs	Adds filtered-range module and re-exports `FilteredRange`.
diskann/src/graph/search/inline_filter_search.rs	Makes internal return type/function `pub(crate)` for reuse by filtered-range search.
diskann/src/graph/search/filtered_range_search.rs	Implements the new `FilteredRange` search type and its internal second-round expansion.
diskann/src/graph/glue.rs	Adds `PartialEq/Eq` derives for `Decision<T>`.
diskann-benchmark/src/main.rs	Adds benchmark integration tests for range and filtered-range JSON examples.
diskann-benchmark/src/inputs/graph_index.rs	Adds `"filtered-range"` search phase schema + validation.
diskann-benchmark/src/index/search/range.rs	Generalizes range runner to support both range and filtered-range backends via parameter mapping.
diskann-benchmark/src/index/search/plugins.rs	Adds filtered-range plugin entry.
diskann-benchmark/src/index/result.rs	Adds `RangeSearchResults::new_filtered` constructor for filtered-range summaries.
diskann-benchmark/src/index/inmem/spherical.rs	Wires filtered-range plugin into spherical in-mem benchmark path.
diskann-benchmark/src/index/benchmarks.rs	Wires filtered-range plugin into benchmark registry + job runner.
diskann-benchmark/example/graph-index-filter-range.json	Adds example job config for `"filtered-range"`.
diskann-benchmark-core/src/search/mod.rs	Documents filtered-range support.
diskann-benchmark-core/src/search/graph/mod.rs	Exposes new `filtered_range` module.
diskann-benchmark-core/src/search/graph/filtered_range.rs	Adds benchmark-core searcher + aggregation plumbing for filtered-range.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

+                    InternalSearchStats {
+                        cmps,
+                        hops: hops + range_stats.hops,
+                        range_search_second_round: true,
+                    },


+//! Tests for filtered range search using an always-true filter.
+//!
+//! These cover the filtered-range cases directly and validate the filtered-range
+//! behavior against its own baselines.


Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>

codecov-commenter · 2026-07-10T18:38:43Z

Codecov Report

❌ Patch coverage is 95.54205% with 44 lines in your changes missing coverage. Please review.
✅ Project coverage is 90.36%. Comparing base (9123eb5) to head (1e51766).
⚠️ Report is 13 commits behind head on main.

Files with missing lines	Patch %	Lines
...kann/src/graph/test/cases/filtered_range_search.rs	96.29%	13 Missing ⚠️
...-benchmark-core/src/search/graph/filtered_range.rs	86.51%	12 Missing ⚠️
diskann-benchmark/src/index/benchmarks.rs	88.67%	6 Missing ⚠️
diskann/src/graph/test/cases/range_search.rs	94.05%	6 Missing ⚠️
diskann-benchmark/src/index/search/plugins.rs	50.00%	3 Missing ⚠️
diskann/src/graph/search/filtered_range_search.rs	99.33%	2 Missing ⚠️
diskann-benchmark/src/index/search/range.rs	96.29%	1 Missing ⚠️
diskann-benchmark/src/inputs/graph_index.rs	95.65%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #1228      +/-   ##
==========================================
+ Coverage   89.64%   90.36%   +0.71%     
==========================================
  Files         503      510       +7     
  Lines       95761    97751    +1990     
==========================================
+ Hits        85845    88330    +2485     
+ Misses       9916     9421     -495

Flag	Coverage Δ
miri	`90.36% <95.54%> (+0.71%)`	⬆️
unittests	`90.03% <95.54%> (+0.72%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines	Coverage Δ
diskann-benchmark-core/src/search/graph/mod.rs	`100.00% <ø> (ø)`
diskann-benchmark/src/index/inmem/spherical.rs	`100.00% <ø> (ø)`
diskann-benchmark/src/index/result.rs	`88.36% <100.00%> (+39.32%)`	⬆️
diskann-benchmark/src/main.rs	`91.81% <100.00%> (+0.27%)`	⬆️
diskann/src/graph/glue.rs	`86.02% <ø> (ø)`
diskann/src/graph/search/inline_filter_search.rs	`97.95% <100.00%> (ø)`
diskann/src/graph/search/range_search.rs	`97.76% <100.00%> (+<0.01%)`	⬆️
diskann-benchmark/src/index/search/range.rs	`97.01% <96.29%> (+97.01%)`	⬆️
diskann-benchmark/src/inputs/graph_index.rs	`58.87% <95.65%> (+6.32%)`	⬆️
diskann/src/graph/search/filtered_range_search.rs	`99.33% <99.33%> (ø)`
... and 5 more

... and 67 files with indirect coverage changes

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

hildebrandmw

Thanks Magdalen. I've left a number of comments throughout the PR, but they all fall into a few general themes.

This PR duplicates a lot of logic when that duplication should be abstracted instead. The worst offender is probably DistanceFiltered.
We need to move away from messing around with the internal of scratch space (in_range, range_frontier, visited etc) - that can quickly become very brittle.
The logic in the filtered search implementation can be tightened.
We should start tagging baselines with more verbose descriptions of what the test is covering and what the corner cases being exercised are.
Please keep to high level trait objects in diskann-benchmark instead of generics. The former is significantly better for compile times.

In addition, is the intention of this algorithm for research, or is there a production ask for it? I ask because maintaining such algorithms/code is not free. If this is meant for research, please put it behind an experimental feature flag to avoid committing us to it long term.

hildebrandmw · 2026-07-10T15:32:53Z


+#[derive(Debug, Clone, serde::Serialize, serde::Deserialize)]
+pub(super) struct RangeSearchBaseline {
+    pub(super) grid_size: usize,


Request: can you add a "description" field to describe what's being tests and what the expected results are? Something like this.

We want to help future developers a little.

hildebrandmw · 2026-07-10T15:38:33Z


            let stats = if scratch.in_range.len()
                >= ((self.starting_l() as f32) * self.initial_slack()) as usize
+                && scratch.in_range.len() < self.max_returned().unwrap_or(usize::MAX)


Can this bug fix be split out into its own little PR?

Done, I split this and the new tests out to #1253

hildebrandmw · 2026-07-10T15:42:37Z

+                    merged
+                        .entry(neighbor.id)
+                        .and_modify(|best: &mut Neighbor<_>| {
+                            if neighbor.distance < best.distance {


What's happening here? Why are we expecting repeats for IDs and why is picking the best the right choice?

Overwriting the distance was an error and I've gotten rid of it now. We are expecting repeats because we are merging the best elements found so far with all the filter-satisfying elements found so far, which might have overlap.

hildebrandmw · 2026-07-10T15:44:14Z

+            in_range.sort_unstable_by(|left, right| {
+                left.distance
+                    .total_cmp(&right.distance)
+                    .then_with(|| left.id.cmp(&right.id))


Don't Neighbors already sort themselves?

No, because scratch.best is merged with the predicate-satisfying points, which are not in sorted order.

hildebrandmw · 2026-07-10T15:49:38Z

+            average_precision,
+        })
+    }
+}


Please add tests like all the other entries in search/graph/

hildebrandmw · 2026-07-10T19:16:27Z

+    }
+
+    /// Create range search with full options.
+    #[allow(clippy::too_many_arguments)]


We should probably listen to clippy here. This is a lot of arguments, and while it mirrors Range - I don't think that's sufficient justification. There are several problems:

There's no real documentation about how these are expected to affect the algorithm.

Callers of FilteredRange::with_options just have a sea of values, which can be difficult to remember.

Any changes to the parameters requires a breaking change for all callers.

It would be way better to use a builder interface for these. As a side-benefit, the same builder could be used for both Range and FilteredRange.

Thanks, I set up a builder

hildebrandmw · 2026-07-10T19:46:50Z

+        PP: glue::SearchPostProcess<S::SearchAccessor, T, O> + Send + Sync,
+        OB: SearchOutputBuffer<O> + Send + ?Sized,
+    {
+        async move {


This and filtered_range_search_internal need at least one clean up pass, if not more. The intermediate storage states are messy and often redundant.

Several things I notices:

The computation self.range() * self.range_slack() happens often, especially within loops. Should this not be a pre-computed value?

We may be able to skip merged entirely with a vec + sort + dedup

I don't know what's happening with scratch.in_range. It's overwritten in scratch simply to pass it as an argument to filtered_range_search_internal (this could be completely avoided and likely save an allocation). Further, it is mutated in filtered_range_search_internal but that mutation is never observed. So why is it being pushed to?

The initialization of scratch.range_frontier is sketchy - it's really relying on the pooled variable being cleared properly to prevent garbage.

Many repetitions of the pattern self.max_returned().unwrap_or(usize::MAX), which presumably is related to the bug fix in range_search. Perhaps this should be a dedicated helper or done just once?

Basically, there are a lot of little things and implicit coupling that could do with a solid scrutinization. Much of this is inherited from range search, but many of my complaints also hold for range search; newer code should be held to a higher standard.

Thanks, I believe I have fixed most of these now. I have:

Removed computation self.range() * self.range_slack() in several places. In some cases it was being used mistakenly, because we don't test for range_slack > 1.0 sufficiently (at all). I will follow up with another PR with tests for this value.

We now use a vec + sort + dedup to gather all points within the radius

I got rid of use of scratch.in_range and now directly modify only scratch.visited and scratch.range_frontier, which I now properly clear and then initialize

pattern self.max_returned().unwrap_or(usize::MAX) is now called just twice, once in each search function

hildebrandmw · 2026-07-10T19:48:27Z

 ///
 /// Both variants carry the item `T` since rejected items are useful for graph navigation.
-#[derive(Debug, Clone, Copy)]
+#[derive(Debug, Clone, Copy, PartialEq, Eq)]


Why are PartialEq and Eq needed? The code compiles fine without them.

I think this was leftover from an intermediate state. Removed now.

hildebrandmw · 2026-07-10T19:50:10Z

+    let (filtered_stats, filtered_results) =
+        run_filtered_range_search(&index, query.as_slice(), filtered_range, &filter);
+
+    let baseline = RangeSearchBaseline {


There's probably a better way to construct these baselines. A constructor taking the stats object and the Range could automatically fill in a lot of the fields (and we could capture all the fields of `Range~ rather then just like 3).

Thanks for the suggestion, I added a constructor

hildebrandmw · 2026-07-10T19:53:40Z

-pub(crate) fn run<I>(
-    runner: &dyn Range<I>,
+pub(crate) fn run<I, R, F>(
+    runner: &R,


Why did this change from a &dyn Range to a true generic? The former is quite important to keep compile times down.

Presumably it's because there's a different in the parameters being passed. Please find an alternate solution that isn't as regressive on compile times.

I found an alternative way to do this, so the generic isn't used anymore

…en/range_filter

While working on #1228, I noticed that there were some issues with our current implementation of range search and its testing. The main issue with testing is that there were no tests ensuring the `max_results` parameter was respected. I have added two tests that ensure this now. The main code had several issues with how `max_results` was handled: 1. The `max_results` parameter was allowed to be less than the initial L_search. This is a conceptual issue because the user expects `max_results` to stop the search from continuing for too long, and the compute used in the initial search will always be controlled by `initial_search_l`. 2. A `max_results` check was not enforced before deciding to continue to the second round search. This meant that if the max results was reached via the initial search, it might not be respected. 3. The second round search was not terminated when `max_results` was reached, meaning it would continue to perform unnecessary work. This PR fixes these issues by adding additional checks of `max_results` at the correct points in the code. --------- Co-authored-by: Magdalen Manohar <mmanohar@microsoft.com>

Magdalen Manohar and others added 30 commits May 14, 2026 17:56

finish up recall computation patch

97b36ef

Merge branch 'main' of github.com:microsoft/DiskANN

07b3671

Merge branch 'main' of github.com:microsoft/DiskANN

eac3ffb

fix conflict

17780f8

fix conflict

43eb517

Merge branch 'main' of github.com:microsoft/DiskANN

1d3a52b

Merge branch 'main' of github.com:microsoft/DiskANN

17eac62

Merge branch 'main' of github.com:microsoft/DiskANN

0ab4baa

Merge branch 'main' of github.com:microsoft/DiskANN

4ddca60

fix conflict

54ee01b

Merge branch 'main' of github.com:microsoft/DiskANN

9e1743f

Merge branch 'main' of github.com:microsoft/DiskANN

93504e6

Merge branch 'main' of github.com:microsoft/DiskANN

b7c27ce

Merge branch 'main' of github.com:microsoft/DiskANN

1dafc55

Merge branch 'main' of github.com:microsoft/DiskANN

33285e2

Merge branch 'main' of github.com:microsoft/DiskANN

824bdb3

Merge branch 'main' of github.com:microsoft/DiskANN

826600f

Merge branch 'main' of github.com:microsoft/DiskANN

1bea9c5

add range groundtruth calculator, rename vector_filters_file

c864722

fmt, clippy, add missing file

bd05e0f

Potential fix for pull request finding

1477ac4

Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>

Potential fix for pull request finding

4f6f80a

Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>

fix: return error for filter bitmap mismatch

2044ba1

standardize naming to kebab case

df844be

standardize variable names, one more kebab-case instance

6827f9d

merge with main

eb70a6f

add range filter algorithm

5edc8e9

cargo fmt

db3f379

add range searhc

37796db

add benchmark search file

39a319f

magdalendobson requested review from a team and Copilot July 6, 2026 19:13

Copilot started reviewing on behalf of magdalendobson July 6, 2026 19:14 View session

Copilot AI reviewed Jul 6, 2026

View reviewed changes

Potential fix for pull request finding

0bb0279

Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>

magdalendobson linked an issue Jul 7, 2026 that may be closed by this pull request

Support filtered range search #1179

Open

Magdalen Manohar added 2 commits July 10, 2026 18:05

fix copilot induced errors

8374b99

regenerate baselines after bug fix

1e51766

hildebrandmw requested changes Jul 10, 2026

View reviewed changes

Merge branch 'main' of github.com:microsoft/DiskANN into users/magdal…

c2ad6f9

…en/range_filter

magdalendobson mentioned this pull request Jul 13, 2026

Clean up range search #1253

Merged

Magdalen Manohar added 4 commits July 13, 2026 16:42

remove range search parts to split out to separate PR

1895710

added a description field for each test

cb78955

overwrite benchmarks to include descriptions

c79529f

fmt + clippy

e5bf7ba

Magdalen Manohar added 10 commits July 14, 2026 18:31

address comments about filtered range search

eefcd3e

use vector sort + dedup instead of hashset

918a6c0

get rid of unnecessary PartialEq and Eq

bbf3a35

add tests

4585f84

add a builder for range search params

b0a6677

get rid of duplicate builder

9571fd0

change old description

251a0ba

remove duplicated code

1440049

tighten up benchmark

85bf3af

remove edits done by clippy

e30697f

magdalendobson force-pushed the users/magdalen/range_filter branch from fff67b6 to e30697f Compare July 15, 2026 14:21

Magdalen Manohar added 2 commits July 16, 2026 16:06

fix merge conflict

e4bbced

fmt

34fb7a0

Uh oh!

Conversation

magdalendobson commented Jul 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

codecov-commenter commented Jul 10, 2026

Codecov Report

Uh oh!

hildebrandmw left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

magdalendobson commented Jul 6, 2026 •

edited

Loading