Dataflow Templates 2026-08-11-00_RC00

feat(Spanner): integrate SourceConfigParser to centralize shard confi…

Dataflow Templates 2026-08-04-00_RC00

What's Changed

Additional New Features

• feat(sourcedb-to-spanner): Add uniform splitting support for extended… by @jsuhani-2026 in #4028
• [Data Validation] Add Custom Transformation ITs by @aasthabharill in #4062
• Adds Azure dependencies to the YAML template by @chamikaramj in #4088
• [Data Validation] Sharded E2E test by @aasthabharill in #4076
• [Data Validation] Add Primary Key IT by @aasthabharill in #4077
• Add Datastream to Spanner IT for PostgreSQL partitioned tables by @jsuhani-2026 in #4081
• [Data Validation] Max Columns Wide Row IT by @aasthabharill in #4083
• [Data Validation] Add wide row tests for max column and table name lengths by @aasthabharill in #4093
• Add Datastream to Spanner IT for PostgreSQL inheritance tables by @jsuhani-2026 in #4082
• test(spanner): add tests for creating tables w/o primary keys by @hengfengli in #4073
• [Data Validation] Add GCSSpannerDVSchemaTransformationsIT by @aasthabharill in #4080
• [Data Validation] Add Wide Row IT for Max PK Columns by @aasthabharill in #4090
• [Data Validation] Add wide row tests for max string length by @aasthabharill in #4094
• [Data Validation] Add wide row tests for 10 MiB cell and 8KiB PK by @aasthabharill in #4092

Bug fixes

• build: update to latest import-spec version by @fbiville in #4084

Improvements

• refactor: simplify template logic by @fbiville in #3582
• update gsutils ref by @derrickaw in #4098

Full Changelog: 2026-07-28-00_RC00...2026-08-04-00_RC00

Dataflow Templates 2026-07-28-00_RC00

What's Changed

Additional New Features

• [Data Validation] Add Integration Testing Framework and Core Matching IT by @aasthabharill in #4031
• [Data Validation] Add GCSSpannerDvRenamedColumnsIT and GCSSpannerDvRenamedTableIT by @aasthabharill in #4055
• Adding support for custom data generation using JAR by @shreyakhajanchi in #4024
• [Data Validation] Add GCSSpannerDvSchemaIT by @aasthabharill in #4054
• [Data Validation] First E2E Integration Test + Base class by @aasthabharill in #4061
• [Data Validation] Add Sharded Setup IT by @aasthabharill in #4067

Bug fixes

• try fixing missing delta lake read class by @derrickaw in #4038
• Run Validation ITs using DirectRunner by @aasthabharill in #4057

Improvements

• update package version by @derrickaw in #4044

New Contributors

• @yonatandt made their first contribution in #4039

Full Changelog: 2026-07-21-00_RC00...2026-07-28-00_RC00

Dataflow Templates 2026-07-21-00_RC00

What's Changed

Additional New Features

• test: add integration test for PostgreSQL partitioned tables by @jsuhani-2026 in #4018
• feat(yaml): Add Pub/Sub to Iceberg YAML template by @ihebimad in #4020
• add readFromDeltaLake transform by @derrickaw in #4027
• feat: add option to execute Spanner index DDL statements in parallel during import. by @pratickchokhani in #4036

Full Changelog: 2026-07-14-00_RC00...2026-07-21-00_RC00

Dataflow Templates 2026-07-14-00_RC00

What's Changed

Additional New Features

• Bulk migration source refactoring - moved mapper into the source connector by @bharadwaj-aditya in #4006
• Add doc for using directrunner in spanner ITs for local testing by @dhwanilpatel in #3996
• feat(sourcedb-to-spanner): add fail-fast validation for PostgreSQL no… by @jsuhani-2026 in #3984

Improvements

• refactor: rename and simplify overlay token parsing by @fbiville in #3967
• Update FirestoreToFirestore documentation to explicitly state Datastore API requirement by @michaeltle-goog in #4025

Package Upgrades

• Upgrade to Beam 2.75.0 by @shunping in #4015

Full Changelog: 2026-07-08-00_RC00...2026-07-14-00_RC00

Dataflow Templates 2026-07-08-00_RC00

What's Changed

Additional New Features

• Reverse replication source seperation - 3 by @bharadwaj-aditya in #3952
• feat: add support for UUID and PG_UUID types across Spanner migration utilities. by @pratickchokhani in #3905
• feat(Spanner): implement Standardize shard config in live migration. by @pratickchokhani in #3934
• refactor(sourcedb-to-spanner): Create db source specific folders by @bharadwaj-aditya in #3979
• refactor: refine firewall logic and optimize enabled APIs by @srozsnyai in #3562
• Separate bulk migration pr2 connectors by @bharadwaj-aditya in #3981
• feat(sourcedb-to-spanner): support PostgreSQL inheritance tables via … by @jsuhani-2026 in #3950
• test(spanner-to-sourcedb): Add Spanner-to-Spanner integration tests by @srozsnyai in #3977
• Directrunner fix for live/reverse template by @dhwanilpatel in #3957

Bug fixes

• Increase timeout in SpannerToMySqlCustomTransformationLT by @aasthabharill in #3963
• Datastream LT fixes by @aasthabharill in #3970
• Fix import/export failures by @aasthabharill in #3980
• switch to existing artifact bucket by @derrickaw in #3913
• fix label dependabot PRs workflow by @derrickaw in #3987
• pin commons-configuration2 to less vulnerable version 2.15 by @derrickaw in #3966

Improvements

• refactor: remove unused parameter by @fbiville in #3975
• refactor: clarify source and source provider class names by @fbiville in #3974
• test: make schema queries idempotent by @fbiville in #3971
• update workflows to be node 24 compatible by @derrickaw in #3991
• Anomaly detection template improvements by @claudevdm in #3990

Package Upgrades

• build: bump import-spec by @fbiville in #3964

New Contributors

• @sandesh-jagdale-tp made their first contribution in #3844
• @dhwanilpatel made their first contribution in #3957

Full Changelog: 2026-06-30-00_RC00...2026-07-08-00_RC00

Dataflow Templates 2026-06-30-00_RC02

What's Changed

Additional New Features

• first checkin for reverse replication source seperation by @bharadwaj-aditya in #3930
• Seperated live migration by @bharadwaj-aditya in #3942
• Seperated live migration 2 by @bharadwaj-aditya in #3947
• Seperated live migration 3 by @bharadwaj-aditya in #3951
• [Data Validations] Add shard_id column to MismatchedRecords table by @aasthabharill in #3954
• Add Iceberg to AlloyDB YAML template by @ihebimad in #3932

Bug fixes

• Update generate_yaml_java_templates_test.py to newer copyright by @derrickaw in #3940
• Fix workflows static analysis findings by @derrickaw in #3946
• fix(Spanner): Fixed shard file config structure in reverse replication FT tests. by @pratickchokhani in #3933

Improvements

• refactor(spanner-to-sourcedb): complete reverse replication source se… by @bharadwaj-aditya in #3943

Full Changelog: 2026-06-23-00_RC00...2026-06-30-00_RC00

Dataflow Templates 2026-06-23-00_RC00

What's Changed

Additional New Features

• feat(spanner-to-sourcedb): add Spanner-to-Spanner reverse replication support by @srozsnyai in #3773

Bug fixes

• [yaml] - fix auto_sharding naming by @derrickaw in #3772
• fix code path in yaml/python readmes by @derrickaw in #3936

Improvements

• upgrade activemq to 5.19.2 to match beam by @derrickaw in #3862

New Contributors

• @srozsnyai made their first contribution in #3773

Full Changelog: 2026-06-16-00_RC00...2026-06-23-00_RC00

Dataflow Templates 2026-06-16-00_RC00

What's Changed

Additional New Features

• [Data Validation] Add support for Custom Transformation by @aasthabharill in #3875
• Add support for Remote UDF by @adrw-google in #3860

Bug fixes

• fix: do not close the session after constraint drop by @fbiville in #3908
• feat: integrate CommonTemplateJvmInitializer for SSL certificate staging in SpannerToSourceDb pipeline. by @pratickchokhani in #3910

Improvements

• Dont throw exception in async dofn by @claudevdm in #3928

Full Changelog: 2026-06-09-00_RC00...2026-06-16-00_RC00

Dataflow Templates 2026-06-09-00_RC00

Note: This release is being rolled out. It may not be in your region yet.

What's Changed

Additional New Features

• feat: Add migrations-functional-testing agent skill for Dataflow pipeline testing by @aasthabharill in #3883
• Add spannerChangeStreamTvfNameList parameters for 3 change stream templates by @tianz101 in #3838
• feat(Spanner): integrate SourceConfigParser to centralize shard configuration loading for SpannerToSourceDb pipelines by @pratickchokhani in #3840
• Add project-context.md for sourcedb-to-spanner by @aasthabharill in #3899
• Terraform template for infra setup for starting a migration by @shreyakhajanchi in #3867

Improvements

• Upgrade to Beam 2.74.0 by @Abacn in #3850
• Updates the job builder util package workflow to also publish a YAML provider listing by @chamikaramj in #3891

Full Changelog: 2026-06-02-00_RC01...2026-06-09-00_RC00