Skip to content

[python] Introduce 'product' aggregator function#8718

Open
PyRSA wants to merge 14 commits into
apache:masterfrom
PyRSA:feature/pypaimon-product-agg-impl
Open

[python] Introduce 'product' aggregator function#8718
PyRSA wants to merge 14 commits into
apache:masterfrom
PyRSA:feature/pypaimon-product-agg-impl

Conversation

@PyRSA

@PyRSA PyRSA commented Jul 18, 2026

Copy link
Copy Markdown
Contributor

Purpose

This PR introduces the Python Decimal implementation and aligns the Python FieldProductAgg implementation with the Java implementation.

The changes include:

  • Introduce the Python Decimal data structure.
  • Support multiplication for all numeric types.
  • Add overflow checks for integral types (TINYINT, SMALLINT, INT, BIGINT).
  • Support DECIMAL multiplication with precision and scale handling.
  • Add comprehensive unit tests.

Tests

  • python -m pytest pypaimon/tests/test_decimal.py -q
  • python -m pytest pypaimon/tests/test_field_aggregators.py::FieldProductAggTest -q
  • python -m pytest pypaimon/tests/test_aggregation_e2e.py -q
  • python -m pytest pypaimon/tests/test_aggregation_merge_function.py -q
  • git diff --check

@JingsongLi

Copy link
Copy Markdown
Contributor

Python’s product function does not align with Java semantics. DECIMAL division is silently rounded, whereas Java throws an error; the default decimal precision=28 causes the valid DECIMAL(38,*) operation to fail; FLOAT/DOUBLE division by zero throws an exception, whereas Java returns Infinity; the shared Decimal encoder/decoder also has issues with scale > 18 and negative canonical bytes.

@PyRSA

PyRSA commented Jul 20, 2026

Copy link
Copy Markdown
Contributor Author

Python’s product function does not align with Java semantics. DECIMAL division is silently rounded, whereas Java throws an error; the default decimal precision=28 causes the valid DECIMAL(38,*) operation to fail; FLOAT/DOUBLE division by zero throws an exception, whereas Java returns Infinity; the shared Decimal encoder/decoder also has issues with scale > 18 and negative canonical bytes.

Good catch, thanks! Fixed.
The Python implementation now matches the Java behavior for DECIMAL division and FLOAT/DOUBLE division by zero. It also includes the related Decimal compatibility fixes.

@JingsongLi

Copy link
Copy Markdown
Contributor

Please resolve conflicts.

@PyRSA

PyRSA commented Jul 21, 2026

Copy link
Copy Markdown
Contributor Author

Please resolve conflicts.

Resolved, thanks!

@PyRSA

PyRSA commented Jul 21, 2026

Copy link
Copy Markdown
Contributor Author

The failure appears to be unrelated to this PR. It is caused by a transient dependency download failure in the paimon-lumina module. I'll wait for CI to be rerun.

Error:  Failed to execute goal on project paimon-lumina: Could not resolve dependencies for project org.apache.paimon:paimon-lumina:jar:2.0-SNAPSHOT: Failed to collect dependencies at org.aliyun.lumina:lumina-jni:jar:0.4.0: Failed to read artifact descriptor for org.aliyun.lumina:lumina-jni:jar:0.4.0: Could not transfer artifact org.aliyun.lumina:lumina-jni:pom:0.4.0 from/to lumina (https://dlf-mvn-repo.oss-cn-shanghai.aliyuncs.com/mvn-repo/release): transfer failed for https://dlf-mvn-repo.oss-cn-shanghai.aliyuncs.com/mvn-repo/release/org/aliyun/lumina/lumina-jni/0.4.0/lumina-jni-0.4.0.pom: Remote host terminated the handshake: SSL peer shut down incorrectly -> [Help 1]
Error:  
Error:  To see the full stack trace of the errors, re-run Maven with the -e switch.
Error:  Re-run Maven using the -X switch to enable full debug logging.
Error:  
Error:  For more information about the errors and possible solutions, please read the following articles:
Error:  [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/DependencyResolutionException
Error:  
Error:  After correcting the problems, you can resume the build with the command
Error:    mvn <args> -rf :paimon-lumina
Error: Process completed with exit code 1.

@JingsongLi

Copy link
Copy Markdown
Contributor

multiplication is performed outside the high-precision context. The issue still occurs in the latest head: 1234567890123456789 * 1000000000000000001—the trailing digits are truncated.

@PyRSA

PyRSA commented Jul 21, 2026

Copy link
Copy Markdown
Contributor Author

multiplication is performed outside the high-precision context. The issue still occurs in the latest head: 1234567890123456789 * 1000000000000000001—the trailing digits are truncated.

Thanks. Fixed!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants