Intra-segment search for histogram and range aggs - #22531
Open
prudhvigodithi wants to merge 11 commits into
Open
Intra-segment search for histogram and range aggs#22531prudhvigodithi wants to merge 11 commits into
prudhvigodithi wants to merge 11 commits into
Conversation
date_histogram, auto_date_histogram, and range share the filter-rewrite optimization (FilterRewriteOptimizationContext#tryOptimize -> point-tree traversal). Under intra-segment search a segment is split into doc-id-range partitions collected on different threads, so getLeafCollector (hence tryOptimize) runs once per partition. The point-tree traversal counts over the whole segment, so running it per partition multiplied bucket counts (see opensearch-project#18016). Fix: - No sub-agg: keep the O(1) whole-cell bulk count (pointTree.size()) but do not partition it (it cannot be restricted to a doc-id range). These aggs opt into intra only when a sub-aggregation is present (factories.countAggregators() > 0), so plain histograms/ranges stay on the sequential fast path and never duplicate. - With sub-agg: make the traversal partition-aware. The partition's [minDocId, maxDocId), known in ContextIndexSearcher#searchLeaf, is carried to the collection layer via a thread-scoped value on SearchContext (withPartitionDocIdRange returning a Releasable, used with try-with-resources). SubAggRangeCollector clamps collectDocId/ collectDocIdSet to that range, so each partition collects and sub-aggregates only its own docs and the partitions sum correctly. A side-channel is required because Lucene's Collector#getLeafCollector has no doc-bounds parameter (bounds go to the scorer; see apache/lucene#13542). Numeric histogram uses pure doc-by-doc collection (no filter-rewrite path), so it opts into intra unconditionally and benefits from parallel collection. composite is intentionally not opted in: it collects through a stateful, ordered whole-segment queue that is not partition-safe; making it intra-safe is separate work. Verified byte-identical bucket and sub-agg results across partition_strategy segment/balanced/force and slice counts, with a ~2.7-3.6x speedup for date_histogram/auto_date_histogram/range + sub-agg on large/skewed segments. Signed-off-by: Prudhvi Godithi <pgodithi@amazon.com>
Signed-off-by: Prudhvi Godithi <pgodithi@amazon.com>
Contributor
PR Reviewer Guide 🔍(Review updated until commit e4a2f86)Here are some key observations to aid the review process:
|
Contributor
PR Code Suggestions ✨Latest suggestions up to e4a2f86 Explore these optional code suggestions:
Previous suggestionsSuggestions up to commit 34d9214
Suggestions up to commit 52acf52
Suggestions up to commit 6dc98cd
Suggestions up to commit a5f51eb
Suggestions up to commit 2363127
|
Contributor
|
❌ Gradle check result for 32737dd: FAILURE Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change? |
Signed-off-by: Prudhvi Godithi <pgodithi@amazon.com>
Contributor
|
Persistent review updated to latest commit 9c60093 |
Contributor
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## main #22531 +/- ##
============================================
- Coverage 73.50% 73.47% -0.04%
+ Complexity 76507 76496 -11
============================================
Files 6104 6104
Lines 346618 346628 +10
Branches 49888 49886 -2
============================================
- Hits 254793 254675 -118
- Misses 71541 71648 +107
- Partials 20284 20305 +21 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
Member
Author
|
@jainankitk can you take a look at this PR? |
prudhvigodithi
changed the title
prudhvigodithi
marked this pull request as ready for review
July 22, 2026 20:54
prudhvigodithi
requested review from
Bukhtawar,
alchemist51,
andrross,
ashking94,
cwperks,
dbwiddis,
gbbafna,
jed326,
kotwanikunal,
mch2,
msfroh,
owaiskazi19,
reta and
sachinpkale
as code owners
July 22, 2026 20:54
Member
Author
Coming from the above comment #22531 (comment) the mixed results are explained by which collection path runs, not segment size. Both datasets are UTC so the filter-rewrite fast path is eligible, but at runtime a per-segment check against search.max_aggregation_rewrite_filters (default 3000) decides whether BKD actually runs: it builds one range-filter per bucket and declines if that exceeds 3000, falling back to the skip-list / doc-by-doc path.
From profile and debug logs:For the scope of this PR for filter-rewrite aggregations (date_histogram, auto_date_histogram, range/date_range), this PR enables intra-segment partitioning only when the fast path declines. In that case collection falls back to the doc-ID-ordered skip-list / doc-by-doc path, which partitions safely and parallelizes cleanly — the measured wins (−7% to −54%). When the fast path applies, we keep the query sequential: its BKD point-tree walk is value-ordered / not partition-aware, so partitioning would duplicate the walk per partition and regress. Numeric histogram has no filter-rewrite path and always partitions which a clean win. Deferred to follow-up PRs
In short: this PR ships the unambiguous, no-regression wins (fast-path-declines + numeric histogram); the fast-path-applies wins need a per-segment runtime mechanism and are tracked as follow-ups. |
Contributor
|
❌ Gradle check result for 6dc98cd: FAILURE Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change? |
Signed-off-by: Prudhvi Godithi <pgodithi@amazon.com>
Contributor
|
Persistent review updated to latest commit 52acf52 |
Signed-off-by: Prudhvi Godithi <pgodithi@amazon.com>
Contributor
|
Persistent review updated to latest commit 34d9214 |
Contributor
Member
Author
Intra-segment improvement queries (with full query shapes)
Top-level filter-rewrite aggs,
|
| # | Query | Why intra-eligible | big5 (116M, 13 seg) | http_logs logs-241998 (181M) |
|---|---|---|---|---|
| 1 | numeric histogram + avg | numeric histogram (no BKD fast path) | 1758→1480 (−15.8%) | 6047→2833 (−53.2%) |
| 2 | numeric histogram (no sub-agg) | numeric histogram | 886→743 (−16.1%) | 3371→1561 (−53.7%) |
| 3 | date_histogram + avg, non-UTC, 1m | fast path declines (non-UTC) | 2366→1887 (−20.2%) | 5519→2831 (−48.7%) |
| 4 | date_histogram + avg, non-UTC, 1h | fast path declines (non-UTC) | 1983→1674 (−15.6%) | 4806→2866 (−40.4%) |
| 5 | date_histogram + stats, non-UTC, 1m | fast path declines (non-UTC) | 2272→1860 (−18.1%) | 5341→2805 (−47.5%) |
| 6 | date_histogram (no sub-agg), non-UTC, 1m | fast path declines (non-UTC) | 1265→1012 (−20.0%) | 3004→1548 (−48.5%) |
| 7 | auto_date_histogram + avg, non-UTC | fast path declines (non-UTC) | 3210→2894 (−9.8%) | 6784→4517 (−33.4%) |
| 8 | range + stats, overlapping ranges | fast path declines (overlapping) | 1644→1460 (−11.2%) | 5223→3177 (−39.2%) |
| 9 | date_histogram + avg, scripted | fast path declines (script) | 3152→2705 (−14.2%) | 6934→4321 (−37.7%) |
http_logs logs-* (all 8 shards, ~247M docs, 93 segments),
| # | Query | http_logs logs-* (8 shards, 93 seg) |
|---|---|---|
| 1 | numeric histogram + avg | 6219→3687 (−40.7%) |
| 2 | numeric histogram (no sub-agg) | 3667→1943 (−47.0%) |
| 3 | date_histogram + avg, non-UTC, 1m | 6016→4000 (−33.5%) |
| 4 | date_histogram + avg, non-UTC, 1h | 5405→3801 (−29.7%) |
| 5 | date_histogram + stats, non-UTC, 1m | 6457→3934 (−39.1%) |
| 6 | date_histogram (no sub-agg), non-UTC, 1m | 3575→2196 (−38.6%) |
| 7 | auto_date_histogram + avg, non-UTC | 8257→5946 (−28.0%) |
| 8 | range + stats, overlapping ranges | 6304→4332 (−31.3%) |
| 9 | date_histogram + avg, scripted | 7998→6246 (−21.9%) |
date_histogram / auto_date as a sub-agg (nested under an intra-eligible parent)
| Query | big5 (116M, 13 seg) | http_logs logs-241998 (181M) |
|---|---|---|
| numeric histogram + date_histogram sub-agg (calendar day) | 1954→1611 (−17.6%) | 4805→2940 (−38.8%) |
| numeric histogram + auto_date_histogram sub-agg (12 buckets) | 2548→2080 (−18.4%) | 5568→3223 (−42.1%) |
Slice-count sensitivity (mode=auto, big5, main matrix)
big5 has 13 segments. The intra win depends on max_slice_count relative to segment count: with too few slices, work distributes at segment granularity (segments rarely split). With enough slices, individual large segments get partitioned and the win appears.
| Query | big5 slice=4 (balanced) |
big5 slice=4 (force) |
big5 slice=8 (balanced) |
|---|---|---|---|
| numeric histogram + avg | 1620→1623 (+0.2%) | 1871→1886 (+0.8%) | 1758→1480 (−15.8%) |
| numeric histogram (no sub-agg) | 871→858 (−1.5%) | 964→936 (−2.9%) | 886→743 (−16.1%) |
| date_histogram + avg, non-UTC, 1m | 2534→2523 (−0.4%) | 2605→2630 (+1.0%) | 2366→1887 (−20.2%) |
| date_histogram + avg, non-UTC, 1h | 2003→1998 (−0.2%) | 2030→2010 (−1.0%) | 1983→1674 (−15.6%) |
| date_histogram + stats, non-UTC, 1m | 2366→2402 (+1.5%) | 2442→2428 (−0.6%) | 2272→1860 (−18.1%) |
| date_histogram (no sub-agg), non-UTC, 1m | 1379→1334 (−3.3%) | 1383→1362 (−1.5%) | 1265→1012 (−20.0%) |
| auto_date_histogram + avg, non-UTC | 3001→3023 (+0.7%) | 3037→2970 (−2.2%) | 3210→2894 (−9.8%) |
| range + stats, overlapping ranges | 1634→1616 (−1.1%) | 1651→1613 (−2.3%) | 1644→1460 (−11.2%) |
| date_histogram + avg, scripted | 3187→3163 (−0.8%) | 3128→3131 (+0.1%) | 3152→2705 (−14.2%) |
Low bucket count
| # | Query | top-level buckets (big5/httplogs) | big5 (116M) | http_logs logs-241998 (181M) |
|---|---|---|---|---|
| 1 | date_histogram YEAR, non-UTC + avg | 1 | 1354→1255 (−7.3%) | 3034→1887 (−37.8%) |
| 2 | date_histogram MONTH, non-UTC + stats | 1 | 1344→1290 (−4.0%) | 2895→1837 (−36.5%) |
| 3 | date_histogram DAY, non-UTC + avg | 13 / 8 | 1472→1370 (−6.9%) | 5051→3281 (−35.0%) |
| 4 | numeric histogram WIDE interval + avg | 5 / 3 | 1763→1562 (−11.4%) | 4052→2372 (−41.5%) |
| 5 | range 3 wide non-overlapping + avg | 3 | 481→492 (+2.3%) | 1250→1246 (−0.3%) |
| 6 | range 3 wide overlapping + avg | 3 | 1994→1860 (−6.7%) | 4137→2638 (−36.2%) |
| 7 | auto_date_histogram buckets=3, non-UTC + avg | 2 | 2925→2320 (−20.7%) | 6739→4462 (−33.8%) |
| 8 | date_histogram DAY, non-UTC + cardinality sub | 13 / 8 | 1187→1127 (−5.1%) | 4578→3515 (−23.2%) |
| 9 | date_histogram WEEK, non-UTC + avg | 2–3 | 1431→1317 (−8.0%) | 3010→2106 (−30.0%) |
| 10 | numeric histogram WIDE + date_histogram DAY sub | 5 / 29 (outer) | 2731→2484 (−9.0%) | 6737→4453 (−33.9%) |
Query shapes
##### 1. numeric histogram + avg
{ "size": 0, "aggs": {
"h": { "histogram": { "field": "metrics.size", "interval": 100 },
"aggs": { "a": { "avg": { "field": "metrics.size" } } } } } }
##### 2. numeric histogram (no sub-agg)
{ "size": 0, "aggs": {
"h": { "histogram": { "field": "metrics.size", "interval": 100 } } } }
##### 3. date_histogram + avg, non-UTC, 1m
{ "size": 0, "aggs": {
"h": { "date_histogram": { "field": "@timestamp", "fixed_interval": "1m", "time_zone": "+05:30" },
"aggs": { "a": { "avg": { "field": "metrics.size" } } } } } }
##### 4. date_histogram + avg, non-UTC, 1h
{ "size": 0, "aggs": {
"h": { "date_histogram": { "field": "@timestamp", "fixed_interval": "1h", "time_zone": "+05:30" },
"aggs": { "a": { "avg": { "field": "metrics.size" } } } } } }
##### 5. date_histogram + stats, non-UTC, 1m
{ "size": 0, "aggs": {
"h": { "date_histogram": { "field": "@timestamp", "fixed_interval": "1m", "time_zone": "+05:30" },
"aggs": { "s": { "stats": { "field": "metrics.size" } } } } } }
##### 6. date_histogram (no sub-agg), non-UTC, 1m
{ "size": 0, "aggs": {
"h": { "date_histogram": { "field": "@timestamp", "fixed_interval": "1m", "time_zone": "+05:30" } } } }
##### 7. auto_date_histogram + avg, non-UTC
{ "size": 0, "aggs": {
"h": { "auto_date_histogram": { "field": "@timestamp", "buckets": 100, "time_zone": "+05:30" },
"aggs": { "a": { "avg": { "field": "metrics.size" } } } } } }
##### 8. range + stats, overlapping ranges
{ "size": 0, "aggs": {
"r": { "range": { "field": "metrics.size", "ranges": [
{ "to": 1000 }, { "from": 500, "to": 10000 },
{ "from": 5000, "to": 100000 }, { "from": 50000 } ] },
"aggs": { "s": { "stats": { "field": "metrics.size" } } } } } }
##### 9. date_histogram + avg, scripted
{ "size": 0, "aggs": {
"h": { "date_histogram": { "field": "@timestamp", "fixed_interval": "1m", "script": { "source": "_value" } },
"aggs": { "a": { "avg": { "field": "metrics.size" } } } } } }
##### 10. numeric histogram + date_histogram **sub-agg** (Table 2)
{ "size": 0, "aggs": {
"p": { "histogram": { "field": "metrics.size", "interval": 1000 },
"aggs": { "c": { "date_histogram": { "field": "@timestamp", "calendar_interval": "day" } } } } } }
##### 11. numeric histogram + auto_date_histogram **sub-agg** (Table 2)
{ "size": 0, "aggs": {
"p": { "histogram": { "field": "metrics.size", "interval": 1000 },
"aggs": { "c": { "auto_date_histogram": { "field": "@timestamp", "buckets": 12 } } } } } }
##### Lower bucket count queries
// C1 date_histogram YEAR non-UTC + avg (1 bucket)
{ "size": 0, "aggs": { "h": { "date_histogram": { "field": "@timestamp", "calendar_interval": "year", "time_zone": "+05:30" },
"aggs": { "a": { "avg": { "field": "size" } } } } } }
// C2 date_histogram MONTH non-UTC + stats (1 bucket) — C9 uses "week"
{ "size": 0, "aggs": { "h": { "date_histogram": { "field": "@timestamp", "calendar_interval": "month", "time_zone": "+05:30" },
"aggs": { "s": { "stats": { "field": "size" } } } } } }
// C4 numeric histogram WIDE interval + avg (few buckets; big5 interval 2000)
{ "size": 0, "aggs": { "h": { "histogram": { "field": "size", "interval": 1000000 },
"aggs": { "a": { "avg": { "field": "size" } } } } } }
// C5 range 3 wide non-overlapping + avg (fast path applies -> no intra)
{ "size": 0, "aggs": { "r": { "range": { "field": "size",
"ranges": [ { "to": 10000 }, { "from": 10000, "to": 100000 }, { "from": 100000 } ] },
"aggs": { "a": { "avg": { "field": "size" } } } } } }
// C6 range 3 wide OVERLAPPING + avg (fast path declines -> intra)
{ "size": 0, "aggs": { "r": { "range": { "field": "size",
"ranges": [ { "to": 50000 }, { "from": 10000, "to": 200000 }, { "from": 100000 } ] },
"aggs": { "a": { "avg": { "field": "size" } } } } } }
// C7 auto_date_histogram buckets=3 non-UTC + avg (2 buckets)
{ "size": 0, "aggs": { "h": { "auto_date_histogram": { "field": "@timestamp", "buckets": 3, "time_zone": "+05:30" },
"aggs": { "a": { "avg": { "field": "size" } } } } } }
// C10 numeric histogram WIDE + date_histogram DAY sub (few buckets both levels)
{ "size": 0, "aggs": { "p": { "histogram": { "field": "size", "interval": 1000000 },
"aggs": { "d": { "date_histogram": { "field": "@timestamp", "calendar_interval": "day" },
"aggs": { "a": { "avg": { "field": "size" } } } } } } }
Contributor
|
It will be useful to summarize the performance runs here. Few things I will be curious to see are: a) With this change how are standard workloads like Big5, HttpLogs, NOAA looking. We should ensure there is no regression in those workloads. b) With lower bucket counts, do we see improvement and no regression c) Pointed tests where improvement is seen which you have covered via multiple tests results in this PR already. To summarize that would be helpful |
Member
Author
|
{"run-benchmark-test": "id_3"} |
1 similar comment
Member
Author
|
{"run-benchmark-test": "id_3"} |
sohami
reviewed
Aug 6, 2026
Member
Author
|
{"run-benchmark-test": "id_3"} |
Signed-off-by: Prudhvi Godithi <pgodithi@amazon.com>
Member
Author
Validation on the NOAA benchmark datasetValidated intra-segment aggregation on a opensearch-benchmark NOAA workload Setup: Real workload queries: fast path applies → gate declines intra (no-regression)
Intra-eligible queries: clean wins
Query shapes// range-date-histo (top-level range → date_histogram, UTC; fast path applies → no intra)
{ "size": 0, "aggs": { "tmax": { "range": { "field": "TMAX",
"ranges": [ {"to":-10},{"from":-10,"to":0},{"from":0,"to":10},{"from":10,"to":20},{"from":20,"to":30},{"from":30} ] },
"aggs": { "date": { "date_histogram": { "field": "date", "calendar_interval": "1w" } } } } } }
// range-date-histo-with-metrics
{ "size": 0, "aggs": { "tmax": { "range": { "field": "TMAX",
"ranges": [ {"to":-10},{"from":-10,"to":0},{"from":0,"to":10},{"from":10,"to":20},{"from":20,"to":30},{"from":30} ] },
"aggs": { "date": { "date_histogram": { "field": "date", "calendar_interval": "1w" },
"aggs": { "tmin": {"min":{"field":"TMIN"}}, "tavg": {"avg":{"field":"TAVG"}}, "tmax": {"max":{"field":"TMAX"}} } } } } } }
// range-auto-date-histo-with-metrics
{ "size": 0, "aggs": { "tmax": { "range": { "field": "TMAX",
"ranges": [ {"to":-10},{"from":-10,"to":0},{"from":0,"to":10},{"from":10,"to":20},{"from":20,"to":30},{"from":30} ] },
"aggs": { "date": { "auto_date_histogram": { "field": "date", "buckets": 20 },
"aggs": { "tmin": {"min":{"field":"TMIN"}}, "tavg": {"avg":{"field":"TAVG"}}, "tmax": {"max":{"field":"TMAX"}} } } } } } }
// date-histo-histo (top-level date_histogram UTC → numeric histogram; fast path applies → no intra)
{ "size": 0, "aggs": { "date": { "date_histogram": { "field": "date", "calendar_interval": "1w" },
"aggs": { "tavg": { "histogram": { "field": "TAVG", "interval": 10 } } } } } }
// --- Intra-eligible variants (fast path declines / absent → intra engages) ---
// date_histogram 1d NON-UTC + avg
{ "size": 0, "aggs": { "date": { "date_histogram": { "field": "date", "fixed_interval": "1d", "time_zone": "+05:30" },
"aggs": { "a": { "avg": { "field": "TMAX" } } } } } }
// date_histogram week NON-UTC + stats
{ "size": 0, "aggs": { "date": { "date_histogram": { "field": "date", "calendar_interval": "week", "time_zone": "+05:30" },
"aggs": { "s": { "stats": { "field": "TMAX" } } } } } }
// auto_date_histogram NON-UTC + avg
{ "size": 0, "aggs": { "date": { "auto_date_histogram": { "field": "date", "buckets": 100, "time_zone": "+05:30" },
"aggs": { "a": { "avg": { "field": "TMAX" } } } } } }
// range OVERLAPPING + avg
{ "size": 0, "aggs": { "r": { "range": { "field": "TMAX", "ranges": [ {"to":0},{"from":-10,"to":20},{"from":10} ] },
"aggs": { "a": { "avg": { "field": "TAVG" } } } } } }
// numeric histogram (TAVG) + avg
{ "size": 0, "aggs": { "h": { "histogram": { "field": "TAVG", "interval": 5 },
"aggs": { "a": { "avg": { "field": "TMAX" } } } } } }
// date_histogram YEAR NON-UTC + avg (low bucket = 3)
{ "size": 0, "aggs": { "date": { "date_histogram": { "field": "date", "calendar_interval": "year", "time_zone": "+05:30" },
"aggs": { "a": { "avg": { "field": "TMAX" } } } } } }
// cardinality(station.id)
{ "size": 0, "aggs": { "c": { "cardinality": { "field": "station.id" } } } }
// numeric histogram WIDE + stats (few buckets = 63)
{ "size": 0, "aggs": { "h": { "histogram": { "field": "TMAX", "interval": 50 },
"aggs": { "s": { "stats": { "field": "TAVG" } } } } } } |
Member
Author
a) I have tested few standard workload queries which either gated by fast path or the normal ones (does not support intra), did not see any regression. Will run the PR benchmark runs similar to #22531 (comment) (at the moment having issues with GitHub CI's https://www.githubstatus.com/). Edit (Aug 7th 2026):
b) I Have updated the above comments with lower bucket count as well, still see the improvement will add it to the summary c) Summary
|
Signed-off-by: Prudhvi Godithi <pgodithi@amazon.com>
Member
Author
|
{"run-benchmark-test": "id_3"} |
Contributor
|
Persistent review updated to latest commit e4a2f86 |
Contributor
|
The Jenkins job url is https://build.ci.opensearch.org/job/benchmark-pull-request/8392/ . Final results will be published once the job is completed. |
Member
Author
|
{"run-benchmark-test": "id_11"} |
Contributor
|
The Jenkins job url is https://build.ci.opensearch.org/job/benchmark-pull-request/8393/ . Final results will be published once the job is completed. |
Contributor
|
❌ Gradle check result for e4a2f86: FAILURE Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change? |
Contributor
Benchmark ResultsBenchmark Results for Job: https://build.ci.opensearch.org/job/benchmark-pull-request/8393/
|
Contributor
Benchmark Baseline Comparison ResultsBenchmark Results for Job: https://build.ci.opensearch.org/job/benchmark-compare/293/
|
Contributor
Benchmark ResultsBenchmark Results for Job: https://build.ci.opensearch.org/job/benchmark-pull-request/8392/
|
Contributor
Benchmark Baseline Comparison ResultsBenchmark Results for Job: https://build.ci.opensearch.org/job/benchmark-compare/294/
|
Contributor
Member
Author
|
{"run-benchmark-test": "id_3"} |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
PENDING: Adding of testsIntra-segment search for filter-rewrite aggregations. The four aggregation types opted in:
What this PR does. It makes the filter-rewrite family intra-segment-capable across three cases, each handled by one of two mechanisms:
tryOptimizewill decline — non-UTC rounding, script/missing value, non-indexed field, overlapping ranges, or nested under a parent bucket agg — the aggregation runs an O(docs) doc-by-doc scan, which is naturally partition-safe (scorer-bounded) and parallelizes. So we let it partition.pointTree.size()bulk count is sub-linear (single-digit ms even on 116 M docs local big5 tests).Partitioning cannot beat it so we deliberately do not partition — the aggregation runs the fast path exactly as it does today.Numeric
histogramhas no filter-rewrite path (pure doc-by-doc) and opts in unconditionally.Results — all tests
Filter-rewrite agg +
avgsub-agg, big5 1 segment (116 M)No sub-agg — the hybrid gate (big5 1 segment)
The gate predicts upfront whether the O(1) fast path applies. When it declines → partition the O(docs) scan (win); when it applies → stay sequential (already ~ms, correctly a no-op).
Same four aggs across topologies (win on skew, no-op when balanced)
Real big5 OSB workload queries, big5 1 segment
range-with-metricsrange-auto-date-histo-with-metricsrange-auto-date-histodate_histogram_hourly_aggrange-agg-1Related Issues
Check List
By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.