feat(filters): RelationshipFilter, CollectionFilter M2M fix, and declarative FilterSet - #639
Draft
cofin wants to merge 29 commits into
Draft
feat(filters): RelationshipFilter, CollectionFilter M2M fix, and declarative FilterSet#639cofin wants to merge 29 commits into
cofin wants to merge 29 commits into
Conversation
cofin
force-pushed
the
feat/relationship-filters
branch
from
December 15, 2025 15:57
fd03568 to
ae4a4a1
Compare
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #639 +/- ##
==========================================
+ Coverage 81.21% 81.28% +0.07%
==========================================
Files 99 99
Lines 8300 8385 +85
Branches 1124 1140 +16
==========================================
+ Hits 6741 6816 +75
- Misses 1232 1237 +5
- Partials 327 332 +5 ☔ View full report in Codecov by Sentry. 🚀 New features to boost your workflow:
|
cofin
marked this pull request as draft
December 20, 2025 17:40
cofin
force-pushed
the
feat/relationship-filters
branch
4 times, most recently
from
January 19, 2026 22:11
cc4261a to
274a49c
Compare
cofin
force-pushed
the
feat/relationship-filters
branch
from
February 6, 2026 18:21
274a49c to
47f6d8b
Compare
cofin
force-pushed
the
feat/relationship-filters
branch
from
April 26, 2026 18:53
47f6d8b to
e758d1b
Compare
cofin
changed the title
cofin
added a commit
to cofin/advanced-alchemy
that referenced
this pull request
Apr 30, 2026
…er tests CI Python 3.11 surfaced two SQLite async failures (no such table: uuid_item_gw1) on aiosqlite_engine, while sync passed and local runs were green. Likely cause: SchemaManager's process-level "_created_schemas" cache marks the async schema as done after the session-loop fixture runs, but the on-disk DDL hasn't actually flushed when the function-loop test connection opens. Fix: have the per-test skip fixtures call metadata.create_all(checkfirst=True) on the test's own engine before the session fixture chain hands over a transaction-bound connection. checkfirst=True makes this a cheap no-op when tables already exist, so other backends pay nothing. Refs: PR feedback on PR litestar-org#639 CI run
github-actions
Bot
added
area/dependencies
area/docs
area/example-apps
area/exceptions
area/private-api
size: large
and removed
size: small
labels
Adds RelationshipFilter class enabling efficient filtering across SQLAlchemy relationships using EXISTS subqueries. This addresses two issues: - Issue litestar-org#364: Add StatementFilters for relationships & association patterns - Issue litestar-org#505: Fix MultiFilter with Many-to-Many generating incorrect SQL Features: - Support for one-to-many, many-to-one, many-to-many relationships - Nested relationship filtering (3+ levels) - negate parameter for NOT EXISTS queries - CollectionFilter auto-detection of relationship attributes - MultiFilter JSON API integration The implementation uses correlated EXISTS subqueries for optimal query performance with single database round-trip regardless of nesting depth.
Adds integration tests covering: - Many-to-many relationship filtering - Many-to-one relationship filtering - Negative filtering (NOT EXISTS) - CollectionFilter delegation to RelationshipFilter - MultiFilter JSON API integration Tests use existing uuid_test_session fixtures for consistency.
Mock engines don't support actual database operations, so the RelationshipFilter tests cannot run meaningfully with them. This follows the same pattern used in test_filters.py and other integration tests.
Spanner doesn't support direct UNIQUE constraints, which are used in the UUID test models (IntegrationUUIDTag.name has unique=True).
Move Spanner and Oracle skips to dedicated fixtures that run before the uuid_test_session fixtures. This ensures the skip happens before schema creation, which fails on Spanner (UNIQUE constraints) and Oracle (schema isolation with xdist groups).
Add pytest_collection_modifyitems hook to skip relationship filter tests for Spanner and Oracle engines before fixtures are set up. This prevents errors from schema creation with UNIQUE constraints that Spanner doesn't support.
MySQL (asyncmy) has the same schema isolation issues with xdist groups as Oracle, causing "table doesn't exist" errors.
Per PRD v2.0 (.agents/specs/relationship-filters/prd.md §6.1), v1 of RelationshipFilter ships with EXISTS as the only path. Removes: - use_exists field on RelationshipFilter - _build_join_query method - JOIN dispatch + negate guard in append_to_statement - use_exists pass-through in MultiFilter._create_relationship_filter - test_relationship_filter_join_negate_raises (covers removed code path) EXISTS is the well-understood, SQL-standard path that works on every supported backend. JOIN mode can be added later behind a flag without public API churn if a real use case appears. Refs: .agents/specs/relationship-filters/prd.md (PRD v2.0)
…er tests CI Python 3.11 surfaced two SQLite async failures (no such table: uuid_item_gw1) on aiosqlite_engine, while sync passed and local runs were green. Likely cause: SchemaManager's process-level "_created_schemas" cache marks the async schema as done after the session-loop fixture runs, but the on-disk DDL hasn't actually flushed when the function-loop test connection opens. Fix: have the per-test skip fixtures call metadata.create_all(checkfirst=True) on the test's own engine before the session fixture chain hands over a transaction-bound connection. checkfirst=True makes this a cheap no-op when tables already exist, so other backends pay nothing. Refs: PR feedback on PR litestar-org#639 CI run
Convert advanced_alchemy/filters.py (1508 lines) into a six-submodule
package with re-export parity. No behavioral change; every existing
public name still imports from advanced_alchemy.filters.
Layout:
filters/_base.py ABCs, type vars, FilterMap/LogicalOperatorMap
filters/_pagination.py LimitOffset, OrderBy
filters/_search.py SearchFilter, NotInSearchFilter
filters/_relationship.py RelationshipFilter
filters/_columns.py BeforeAfter, OnBeforeAfter, Collection*, Null*,
ComparisonFilter, operators_map, VALID_OPERATORS
filters/_logical.py ExistsFilter, NotExistsFilter, FilterGroup,
MultiFilter
filters/__init__.py Re-exports + FilterTypes TypeAlias
CollectionFilter relies on a lazy import of RelationshipFilter to avoid a
circular dep — the relationship-handling path is only reached when the
field is a relationship attribute.
Tests:
- New tests/unit/test_filters/test_public_surface.py: 74 parametrized
cases asserting every name in __all__ imports from the public package
and resolves to the same object as imported via its expected submodule.
- tests/integration/test_filters.py adjusted: import and_/or_ from
sqlalchemy directly rather than relying on the old single-file
filters.py to leak them through.
Verification: mypy clean, pyright clean, ruff clean, 1388 unit tests +
41 sqlite filter integration tests pass.
Refs: relationship-filters PRD v2.0 Phase 2 (advanced-alchemy-38c.10)
Add the foundation for the declarative ``FilterSet`` facade. No parsing
or compilation engine yet — those land in Phases 4-5.
* ``advanced_alchemy.exceptions.FilterValidationError`` — request-time
aggregation of per-field coercion errors.
* ``advanced_alchemy.filters._filterset`` (new):
- ``UNSET`` sentinel — singleton, falsy, distinct from ``None``.
- ``FieldSpec`` — frozen dataclass holding (path, column, filter).
- ``BaseFieldFilter`` ABC — owns supported_lookups / default_lookup,
validates the ``lookups=[...]`` constructor override, exposes the
``coerce`` and ``compile`` abstract methods.
* ``advanced_alchemy.filters`` package re-exports ``UNSET``,
``BaseFieldFilter``, ``FieldSpec``.
* Re-export parity test (``test_public_surface``) updated for the new
names; future-annotations leftover from Phase 2 dropped.
* New unit suite ``test_filterset_bootstrap`` covers the sentinel,
exception, dataclass, and ABC contract surface.
Implement the first three concrete Tier 2 field filters declared on ``advanced_alchemy.filters._fields``: * ``StringFilter`` — exact, iexact, contains, icontains, startswith, istartswith, endswith, iendswith, in, not_in, isnull. ``contains`` / ``icontains`` compile to ``SearchFilter``; the rest map onto ``ComparisonFilter`` / ``CollectionFilter`` / ``Null(Not)Filter``. * ``NumberFilter`` — int / float / Decimal coercion via ``type_=``; ``between`` parses comma-separated pairs; supports gt/gte/lt/lte/in/ not_in/isnull plus exact. * ``BooleanFilter`` — exact + isnull, accepting the canonical truthy/falsy tokens (``true``/``false``, ``1``/``0``, ``yes``/``no``, ``on``/``off``). Also fix a latent bug in ``_columns.operators_map``: ``iendswith`` was sharing ``istartswith``'s ``ilike(v + "%")`` lambda, so it silently did prefix matching. ``StringFilter`` exposes the lookup directly, so the fix lands here. Regression test guards both lookups. The package re-exports the three new filters; the parity test is updated accordingly.
Round out the built-in field-filter set on ``advanced_alchemy.filters._fields``: * ``DateFilter`` — exact/gt/gte/lt/lte/between/in/not_in/isnull plus date-part lookups year/month/day. Values parsed via ``date.fromisoformat``. * ``DateTimeFilter`` — superset adding hour/minute/second extraction; parses with ``datetime.fromisoformat``. * ``UUIDFilter`` — exact/in/not_in/isnull; rejects malformed strings via ``uuid.UUID``. * ``EnumFilter(enum=...)`` — accepts enum value or member name; rejects unknown tokens with a friendly error listing valid members. Introduce a small Tier 1 primitive ``DatePartFilter`` that backs the date/time-part lookups (``EXTRACT(part FROM column) <op> value``). Lives next to the field filters since that is where it is consumed. All four filters and ``DatePartFilter`` are re-exported from the ``advanced_alchemy.filters`` package; the parity test is updated.
``OrderingFilter(allowed=[...])`` is a special-case Tier 2 declaration that emits ``ORDER BY`` clauses instead of ``WHERE`` predicates: * Coerces a comma-separated value (with optional ``-`` prefix) into a list of ``(field, direction)`` pairs. * Rejects any field not in the allowed list with a friendly error so HTTP callers cannot order by arbitrary columns. * Compiles to a small Tier 1 ``OrderingApply`` filter that chains the ordered ``.order_by()`` calls onto a ``Select`` statement (and is a no-op for non-SELECT statements). Both names are re-exported from ``advanced_alchemy.filters``; parity test updated.
…3.5) Implements the keystone of the FilterSet facade. ``FilterSet`` subclasses validate every declared field path against ``Meta.model`` at import time: * Walks the MRO for ``BaseFieldFilter`` instances; subclass declarations win over inherited ones. * Resolves each ``foo__bar__baz`` path through SQLAlchemy's mapper, enforcing ``Meta.allowed_relationships`` and ``Meta.max_relationship_depth`` (default 2). * Rejects unknown columns, unknown/disallowed relationships, depth violations, terminal segments that resolve to a relationship rather than a column, dunder/keyword field names, and unsupported lookups — all with line-numbered, human-readable messages. * Builds frozen ``_field_specs`` (``MappingProxyType``) and a ``_lookup_index`` keyed by ``(name, lookup)`` for fast lookup. Auto-generation via ``Meta.auto_fields`` infers a sensible filter for each column from its SQLAlchemy ``python_type``; ``Meta.auto_lookups`` overrides the default lookup catalog per field. Explicit class declarations always win. Special-case filters that do not bind to a column (``OrderingFilter``) opt out via ``BaseFieldFilter.binds_to_column = False`` and validate their own model bindings. Parsing/compilation/OpenAPI methods are deliberately left to Phases 4-6; only the class-creation surface lands here.
…+4.2)
Implement the request-time facade for ``FilterSet``: takes a raw query
mapping and returns a populated instance whose ``invocations``
property lists ``(field, lookup, coerced_value)`` triples ready for
Phase 5 compilation.
* ``FilterSet.from_query_params(Mapping[str, str | Sequence[str]])`` —
HTTP path; runs each value through the field filter's ``coerce()``.
* ``FilterSet.from_dict(Mapping[str, Any])`` — programmatic path;
values that aren't strings or sequences-of-strings are accepted
verbatim so callers passing native types (a ``date``, a list of
``int``) skip redundant string-shaped coercion.
* Key resolution honors the dunder split: ``title`` matches the field
with its default lookup; ``title__icontains`` resolves the trailing
segment against the field's enabled lookup catalog. Relationship
paths (``author__name``, ``author__name__iexact``) work the same way.
* All ``ValueError``s are aggregated into a single
``FilterValidationError`` mapping field name → message; callers get
every problem in one pass.
* ``Meta.strict`` (default ``False``) toggles unknown-key behavior:
silent ignore vs. fail-fast aggregation under the original key.
* ``FilterValidationError.to_dict()`` renders an HTTP-friendly payload
(``{"type": "filter_validation", "errors": {...}}``) for framework
exception handlers.
Round out the parsing surface with regression coverage for the documented edge cases per PRD §11.3: * Malformed UUIDs / dates / year integers — every error funnels into ``FilterValidationError.errors`` keyed by the resolved field name. * Enum coercion by value, by name, and the unknown-member branch. * Case-sensitive lookups: ``title__ICONTAINS`` doesn't bind to the ``icontains`` lookup; non-strict mode silently ignores it; strict mode reports the original key. * A field declared with a lookup subset rejects unsupported lookups the same way (silent vs. strict). * Empty / blank query values for set lookups raise instead of silently passing.
… 5.1+5.2) Adds the keystone compilation pass that turns parsed FilterSet invocations into the Tier 1 statement filters consumed by the existing filter machinery: * ``_compile_path(path, leaf)`` wraps a leaf filter in nested ``RelationshipFilter`` instances right-to-left for relationship traversal; depth-0 paths return the leaf unchanged. * ``FilterSet.to_filters()`` walks ``_invocations`` in declaration order, calls each field filter's ``compile``, applies the path wrapping, and appends the ``OrderingFilter`` output (``OrderingApply``) last so the WHERE clause stays stable across calls.
Adds tests/integration/test_filterset_e2e.py covering the full declarative pipeline (from_query_params → to_filters → append_to_statement) against real database engines. Three layers of coverage: * Sync + async parametrized over the engine matrix using the shared uuid_models_dba fixture (Book/Author one-hop relationship). * A query-count assertion confirming a relationship-traversing FilterSet compiles to a single SELECT round trip. * Hermetic SQLite-only inline-model tests for depth-2 traversal (Post → Author → Org → Country) and an empty-params no-op path.
Adds FilterSet.to_openapi_parameters() and the per-filter _openapi_schema hook on BaseFieldFilter so each declared (field, lookup) becomes an OpenAPI 3 parameter object. OrderingFilter overrides the emit pass to produce a single parameter with an enum of allowed and '-'-prefixed values.
…r (Phase 6.2) TestGoldenOutput asserts the full output for a comprehensive fixture FilterSet covering UUID/String/Number/Enum + a relationship traversal + OrderingFilter, so any drift in shape, ordering, or content surfaces in a diff. TestPerFilterSchemas parametrizes the eight built-in field filter classes (StringFilter, NumberFilter[int], BooleanFilter, DateFilter, DateTimeFilter, UUIDFilter, EnumFilter[str], EnumFilter[int]) and asserts the schema for every supported lookup, plus a focused suite for the OrderingFilter override. Also bumps the uv.lock mysql-connector-python exclude-newer-package timestamp resolved by uv during this branch's test runs.
… 7.2) Companion to the existing depth-1 query-count assertion. Confirms a nested ``RelationshipFilter`` chain does not introduce extra round trips, satisfying the PRD's headline performance guarantee.
Adds the Phase 8 documentation surface for the relationship-filtering work: a user guide that covers ``RelationshipFilter`` (Tier 1) and ``FilterSet`` (Tier 2) with adoption guidance from prior approaches; a cross-reference from the API reference; three runnable examples matching the existing ``examples/standalone_json.py`` idiom; and a 1.10.0 changelog entry. All additions are non-breaking.
* widen the title overline to match the longer page title; * convert three grid tables to ``list-table`` so wide cells (e.g. SQL fragments, lookup catalogs) no longer overflow column boundaries; * split the ``FieldSpec`` ``Attributes:`` block into per-attribute docstrings so dataclass introspection and the prose section don't emit duplicate Sphinx object descriptions.
Three small follow-ups uncovered by the Phase 9 quality gate: * annotate ``items`` in ``_split_csv`` so its dual-branch initialisation resolves to ``list[str]`` under pyright; * type ``OrderingApply.orderings`` factory and field annotation as a string forward reference so pyright stops widening the element to ``Unknown`` when the dataclass is instantiated; * cast the iterable in ``_looks_like_query_value`` to ``Sequence[Any]`` so item iteration over a list/tuple narrowed from ``Any`` produces a known item type; * add the new ``relationship-filtering.rst`` user guide to ``NON_EXECUTABLE_DOCS`` so the Sybil discovery test passes — the new doc's snippets are illustrative, with runnable counterparts in ``examples/filterset_*.py``.
Running ``mypy`` over the full configured package set surfaced four pre-existing issues that the narrower ``mypy advanced_alchemy`` invocation hid. Address them now so ``make type-check`` is clean: * ``test_filterset_openapi._param_named``: replace ``pytest.fail`` (which mypy treats as returning) with ``raise AssertionError`` so the function's declared return type holds on every path. * ``test_filterset_bootstrap``: cast ``Table.c.id`` to ``Column[Any]`` before passing to ``FieldSpec`` — SQLAlchemy returns ``KeyedColumnElement[Any]`` from the column accessor. * ``test_public_surface``: assert the resolved submodule string is non-``None`` after the ``pytest.skip`` guard so mypy narrows the type before the ``import_module`` call. * drop the ``asyncmy`` entry from the missing-imports override — ``asyncmy`` ships type stubs now and the override has no effect.
Convert the four SQLite-only FilterSet e2e tests onto the same engine matrix as their siblings. The helpers.DuckDBCleaner DELETEs in reverse-of-information-schema order, so FK-constrained topologies hit silently-swallowed FK violations. Drop clean_tables in favor of the connection-binding pattern from uuid_test_session_sync — session bound to a connection-level transaction that rolls back at teardown — which works across SQLite, DuckDB, and FK-enforcing backends without dialect-specific cleanup.
cofin
force-pushed
the
feat/relationship-filters
branch
from
May 2, 2026 22:08
48c1962 to
e4a0cbf
Compare
cofin
changed the title
Member
|
Looks great, thanks @cofin 🚀. It looks like an old PR, how much work do we need to do a merge? |
Member
Author
|
I'm not sure I'm sold on the API yet, and I didn't want to commit us to a binding change. I'll have to review this once more with a fresh perspective now that some time has passed. |
|
Hi everyone 👋 I've been following the discussions around dynamic filtering and this PR. Since building a robust, Django-like filterset architecture is a common need, I recently built and open-sourced a standalone library to tackle exactly this: alchemy-filterset. It's built on top of advanced-alchemy, sqlalchemy 2.0, and pydantic v2. Some of the core features include: Django-style lookups (e.g., field__icontains, field__gt) Nested relationship filtering and AssociationProxy support Field negation via a not__ prefix Built-in pagination, global search, and dynamic ordering Safe type casting for strict databases like PostgreSQL I thought I'd share it here in case anyone landing on this thread is looking for a ready-to-use, decoupled solution right now. I would be absolutely thrilled to hear any feedback from the core team or anyone in the community if you get a chance to check it out! Thanks for all the great work on advanced-alchemy! 🚀 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds relationship-aware filtering and a declarative
FilterSetfacade to Advanced Alchemy.What's new
RelationshipFilter— relationship traversal in a single queryCorrelated
EXISTSsubquery that walks ORM relationships:secondarytable or association objectnegate=TrueproducesNOT EXISTSOne query per filter, no N+1, no
DISTINCTworkaround.CollectionFilterover relationship attributesCollectionFilter("tags", [...])now auto-detects the relationship attribute and emits anEXISTSsubquery instead of the previous broken column comparison. Composite primary keys raise a clear error pointing users at the explicitRelationshipFilterform. Closes #505.MultiFilteraccepts{"type": "relationship", ...}JSON-driven callers can now compose relationship filters alongside the existing comparison, search, and exists clauses.
Declarative
FilterSetA class-based filter container that maps query-string parameters onto the underlying filters through the familiar
__-suffix lookup syntax.Highlights:
BooleanFilter,DateFilter,DatePartFilter,DateTimeFilter,EnumFilter,NumberFilter,StringFilter,UUIDFilter, plusOrderingFilter.FilterValidationError.Metacontrols:model,allowed_relationships,max_relationship_depth,strict.PostFilter.openapi_parameters()returns the JSON-schema fragment for direct embedding in framework-generated docs.SELECTagainst the parent model — no extra round trips, no implicitJOINs introduced by the filter (asserted at depth-1 and depth-2 in the integration suite).Package reorganization
advanced_alchemy.filtersis now a package split across focused modules (_base,_columns,_fields,_filterset,_logical,_pagination,_relationship,_search). The public surface is unchanged: every name in the previous__all__is still importable fromadvanced_alchemy.filtersand resolves to the same object — guarded by a re-export parity test.Documentation and examples
docs/usage/repositories/relationship-filtering.rstexamples/filterset_basic.pyexamples/filterset_relationships.pyexamples/multifilter_json.py1.10.0Backwards compatibility
Additive across the board. No public APIs renamed or removed; no signatures changed. Existing filter usage continues to work unchanged.
Closes
CollectionFilterover many-to-many.FilterSetproposal.Test plan
tests/unit/test_filters/)RelationshipFilter+CollectionFilterM2M (tests/integration/test_relationship_filters.py)FilterSetend-to-end across the engine matrix (tests/integration/test_filterset_e2e.py)SELECTassertion at depth-1 and depth-2 relationship pathsmake lintpasses (ruff, codespell, sphinx-lint, mypy strict, pyright strict, slotscheck)Known backend-test caveats
A small number of integration tests skip on:
Tagfixture uses a directUNIQUEconstraint the Spanner emulator does not accept.xdistworker groups produces "table doesn't exist" errors on these specific tests.These are test-infrastructure limitations, not filter correctness issues, and are tracked for follow-up.