fix: properly rehydrate Eventarc CloudEvents from Raw Pub/Sub fallback - #786
Open
ajperel wants to merge 1 commit into
Open
fix: properly rehydrate Eventarc CloudEvents from Raw Pub/Sub fallback#786ajperel wants to merge 1 commit into
ajperel wants to merge 1 commit into
Conversation
## Summary When an HTTP trigger receives a CloudEvent, it strict-validates the incoming HTTP headers. However, Google Front End (GFE) aggressively rejects certain HTTP headers (e.g. `ce-subject` strings ending with a trailing space on the wire). When Eventarc encounters these strict rejections during a push delivery, it falls back to Raw Pub/Sub message delivery, stuffing all of the critical `ce-` metadata into the Pub/Sub message attributes instead of the HTTP headers. Previously, `legacyPubSubEventMiddleware` failed to recognize this fallback behavior. It assumed the body was a legacy Gen-1 Pub/Sub event wrapper if `ce-type` was missing from the HTTP headers, leading to payload shift and parsing failures in downstream SDKs like `firebase-functions`. This addresses one of the primary root causes of firebase/firebase-functions#1922 (which contains a live reproduction case). ### Reproduction 1. Deploy a Firestore CloudEvent trigger. 2. Create a Firestore document with a trailing space in its ID (e.g., `Test Document2 `). 3. Eventarc falls back to Raw Pub/Sub because the HTTP specification forbids OWS (optional whitespace) at the end of header values (`ce-subject: documents/... ` is illegal on the wire). 4. The Cloud Framework receives the raw Pub/Sub payload but fails to extract the CloudEvent wrapper, crashing the function execution. ## Fix Extracts a `rehydrateCloudEvent` helper inside `pubsub_middleware.ts` to cleanly catch and intercept Raw Pub/Sub fallbacks: 1. **Header Promotion**: Promotes all `ce-` prefixed (and unprefixed) Pub/Sub attributes into `req.headers`. 2. **Payload Extraction**: Base64 decodes `message.data` and attempts a JSON parse (unless skipped due to non-JSON `ce-datacontenttype`). 3. **Safe Fallback**: If JSON parsing fails (e.g. `application/protobuf`), it mirrors `bodyParser.raw()` by assigning the `Buffer` directly to `req.body` and passing it to the handler safely. Because Express `req.headers` is just a JavaScript dictionary, assigning headers with trailing spaces internally perfectly bypasses the wire-level HTTP parser constraints that broke the initial Eventarc push. ## Testing - **Manual Verification**: We bundled this patched framework into a local `firebase-functions` dependency and successfully deployed it to a production playground executing `firebase deploy --only functions`. Successfully verified via Cloud Logging that documents with trailing spaces trigger the fallback and process correctly. - Added unit test: **Eventarc fallback CloudEvent (JSON payload)** to verify successful promotion of attributes to headers and JSON deserialization. - Added unit test: **Eventarc fallback CloudEvent (Unprefixed attributes & Binary payload)** to ensure CloudEvents with binary contents strictly bypass JSON parsing and arrive as Raw Buffers. - Full suite green: `109 passing`.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
When an HTTP trigger receives a CloudEvent, it strict-validates the incoming HTTP headers. However, Google Front End (GFE) aggressively rejects certain HTTP headers (e.g.
ce-subjectstrings ending with a trailing space on the wire).When Eventarc encounters these strict rejections during a push delivery, it falls back to Raw Pub/Sub message delivery, stuffing all of the critical
ce-metadata into the Pub/Sub message attributes instead of the HTTP headers.Previously,
legacyPubSubEventMiddlewarefailed to recognize this fallback behavior. It assumed the body was a legacy Gen-1 Pub/Sub event wrapper ifce-typewas missing from the HTTP headers, leading to payload shift and parsing failures in downstream SDKs likefirebase-functions.This addresses one of the primary root causes of firebase/firebase-functions#1922 (which contains a live reproduction case).
Reproduction
Test Document2).ce-subject: documents/...is illegal on the wire).Fix
Extracts a
rehydrateCloudEventhelper insidepubsub_middleware.tsto cleanly catch and intercept Raw Pub/Sub fallbacks:ce-prefixed (and unprefixed) Pub/Sub attributes intoreq.headers.message.dataand attempts a JSON parse (unless skipped due to non-JSONce-datacontenttype).application/protobuf), it mirrorsbodyParser.raw()by assigning theBufferdirectly toreq.bodyand passing it to the handler safely.Because Express
req.headersis just a JavaScript dictionary, assigning headers with trailing spaces internally perfectly bypasses the wire-level HTTP parser constraints that broke the initial Eventarc push.Testing
firebase-functionsdependency and successfully deployed it to a production playground executingfirebase deploy --only functions. Successfully verified via Cloud Logging that documents with trailing spaces trigger the fallback and process correctly.109 passing.