RFC: Derived Columns

[brendancooley]

Problem

Schemas often include columns that are deterministic functions of other columns. Today, users must compute these outside of dataframely before validation:

df = df.with_columns(
    age=(pl.date.today() - pl.col("birth_date")).dt.total_days() // 365
)
validated = PersonSchema.validate(df)

This scatters transformation logic across the codebase and breaks the "schema as source of truth" model.

Proposed API

1. @dy.derived() decorator

Define derived columns alongside rules, using the same pattern:

class PersonSchema(dy.Schema):
    birth_date = dy.Date(nullable=False)
    first_name = dy.String(nullable=False)
    last_name = dy.String(nullable=False)

    # Derived columns
    age = dy.Int64(nullable=False)
    full_name = dy.String(nullable=False)

    @dy.derived("age")
    def derive_age(cls) -> pl.Expr:
        return (pl.date.today() - cls.birth_date.col).dt.total_days() // 365.25

    @dy.derived("full_name")
    def derive_full_name(cls) -> pl.Expr:
        return cls.first_name.col + pl.lit(" ") + cls.last_name.col

2. Schema.with_derived() method

Explicitly apply derivations to a dataframe:

# Input only needs source columns
df = pl.DataFrame({
    "birth_date": [date(1990, 5, 15), date(2000, 1, 1)],
    "first_name": ["Alice", "Bob"],
    "last_name": ["Smith", "Jones"],
})

# Add derived columns
df_with_derived = PersonSchema.with_derived(df)
# Now has: birth_date, first_name, last_name, age, full_name

# Then validate as usual
validated = PersonSchema.validate(df_with_derived)

Expected Behavior

• Derived columns are optional in input. with_derived() adds them if missing, overwrites if present.
• Lazy frames are preserved. with_derived() returns a LazyFrame if given a LazyFrame.
• Invalid targets error at class definition time. @dy.derived("x") raises if x is not a column in the schema.

Open Questions

1. Circular dependencies? Should we detect/error on a derived from b derived from a? If so, this requires topological sorting of derivations—is this doable?

2. Chained derivations? Should derived columns be allowed to depend on other derived columns? e.g., age derived from birth_date, then is_adult derived from age. This would require ordering derivations correctly (topological sort).

3. Serialization? Should derived column expressions be included in Schema.serialize()? Unclear how this interacts with the existing serialization machinery.

[kszlim]

Would be a nice feature, curious how it would handle multiple outputs/combinatorially generated features/etc.

[brendancooley]

Would be a nice feature, curious how it would handle multiple outputs/combinatorially generated features/etc.

Do you have a concrete example in mind Kevin Lim (@kszlim) ? I think the topological sort for sets of dependent features is pretty straightforward to implement, but I'm not exactly sure what you mean by "combinatorially generated features".

[kszlim]

Let's say you want to compute some feature across N different lags.

Or even generating features using column selectors.

[brendancooley]

Let's say you want to compute some feature across N different lags.

Or even generating features using column selectors.

so long as you can represent these as an expression we should be able to grab the dependencies via root_names and recursively derive them prior to executing the dependent columns?

[kszlim]

Yeah, but it'd be nice if you could handle it with your api:

ie.

    @dy.derived("age")
    def derive_age(cls) -> pl.Expr:
        return (pl.date.today() - cls.birth_date.col).dt.total_days() // 365.25

Will always output just a single column right?

I'm guessing @dy.derived("age") just does something like .alias("age") to the output expression?

[brendancooley]

Yeah, but it'd be nice if you could handle it with your api:

ie.

    @dy.derived("age")
    def derive_age(cls) -> pl.Expr:
        return (pl.date.today() - cls.birth_date.col).dt.total_days() // 365.25

Will always output just a single column right?

I'm guessing @dy.derived("age") just does something like .alias("age") to the output expression?

Ah I see, yeah this API isn't quite expressive enough to derive multiple columns in one go. How would you want to express that? Having a list of strings in @dy.derived() doesn't feel quite right.

[kszlim]

Haha i don't know, that's why I brought it up as a use-case as opposed to proposing a solution. I think the design for this would be quite difficult.

I also think that ideally the "derived" api should make it so that you can can statically refer to your column.

Ie. for regular columns, you can do something like:

validated = PersonSchema.validate(df)
validated.select(validated. birth_date.col)

You should make sure that something similar can be done with the derived fields too.

Perhaps it would need to be changed so that you have something more like this:

    @dy.derived("age")
    def age(cls) -> pl.Expr:
        return (pl.date.today() - cls.birth_date.col).dt.total_days() // 365.25

Used like

validated = PersonSchema.validate(df)
validated.select(validated.age.col)

I think it'd be important to ensure the design works with python static type checking/IDE completion.

[brendancooley]

I think it'd be important to ensure the design works with python static type checking/IDE completion.

I think we get at this in the api by defining the target columns on the schema

class PersonSchema(dy.Schema):
    birth_date = dy.Date(nullable=False)
    first_name = dy.String(nullable=False)
    last_name = dy.String(nullable=False)

    # Derived columns
    age = dy.Int64(nullable=False)
    full_name = dy.String(nullable=False)

    # derivation class methods below
    ...

But this does mean that you have to pair every declared column in the schema with an associated classmethod that does the derivation which is maybe a little restrictive. The nice thing about this though is that if users pass a data frame that already has "age" and "full_name" derived to PersonSchema.validate() it works just fine. The @dy.derived methods just give instructions on how to fill them in when absent