GH-3667: Close input readers when ParquetRewriter setup fails#3662
Open
anxkhn wants to merge 1 commit into
Open
GH-3667: Close input readers when ParquetRewriter setup fails#3662anxkhn wants to merge 1 commit into
anxkhn wants to merge 1 commit into
Conversation
…fails ParquetRewriter.getFileReaders opens a TransParquetFileReader for each input file into a local list. If opening a later file throws IOException, it was rethrown as IllegalArgumentException without closing the readers already opened for the earlier files, leaking their SeekableInputStream handles (ParquetRewriter.close() only ends the writer and does not close input readers). One corrupt, missing, or permission-denied file among several valid ones could therefore leak up to N open input streams per failed rewrite/merge/join setup. Close the readers already opened before rethrowing, using AutoCloseables.close so every reader is released and any close failures are aggregated as suppressed exceptions on the original IllegalArgumentException. Add a ParquetRewriterTest case that supplies a valid input file followed by one that fails to open and asserts the first reader's stream is closed. Signed-off-by: Anas Khan <83116240+anxkhn@users.noreply.github.com>
Member
|
I would recommend creating an issue for this. Or at least do not mark it as |
anxkhn
changed the title
Contributor
Author
|
Created #3667 and updated the PR title and body to reference it. Thanks for the guidance. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Rationale for this change
ParquetRewriter.getFileReadersopens aTransParquetFileReaderfor each input file into a local list. If opening a later file throwsIOException, it is rethrown asIllegalArgumentExceptionwithout closing the readers already opened for the earlier files, so theirSeekableInputStreamhandles leak.ParquetRewriter.close()only ends the writer and does not close the input readers, so nothing else releases them either. A single corrupt, missing, or permission-denied file among several valid inputs therefore leaks up to N open input streams per failed rewrite/merge/join setup.What changes are included in this PR?
In
getFileReaders, close the readers already opened before rethrowing, usingorg.apache.parquet.util.AutoCloseables.close(...)so every reader is released and any close failures are aggregated as suppressed exceptions on the originalIllegalArgumentException. This matches the existing cleanup idiom in the module. The change is limited to thecatch (IOException)block; the success path is unchanged.Are these changes tested?
Yes. A new
ParquetRewriterTestcase (testInputFileReadersClosedWhenLaterInputFileFailsToOpen) supplies a valid input file followed by anInputFilewhosenewStream()throws, asserts that constructing theParquetRewriterthrowsIllegalArgumentException, and asserts the first reader stream was closed. The fullParquetRewriterTestclass passes (96 tests, 0 failures, 0 errors).Are there any user-facing changes?
No public API changes. A failed
ParquetRewritersetup now closes input streams it had already opened.Closes #3667