Skip to content

fix: properly rehydrate Eventarc CloudEvents from Raw Pub/Sub fallback - #786

Open
ajperel wants to merge 1 commit into
GoogleCloudPlatform:mainfrom
ajperel:fix-eventarc-fallback
Open

fix: properly rehydrate Eventarc CloudEvents from Raw Pub/Sub fallback#786
ajperel wants to merge 1 commit into
GoogleCloudPlatform:mainfrom
ajperel:fix-eventarc-fallback

Conversation

@ajperel

@ajperel ajperel commented Jul 29, 2026

Copy link
Copy Markdown

Summary

When an HTTP trigger receives a CloudEvent, it strict-validates the incoming HTTP headers. However, Google Front End (GFE) aggressively rejects certain HTTP headers (e.g. ce-subject strings ending with a trailing space on the wire).

When Eventarc encounters these strict rejections during a push delivery, it falls back to Raw Pub/Sub message delivery, stuffing all of the critical ce- metadata into the Pub/Sub message attributes instead of the HTTP headers.

Previously, legacyPubSubEventMiddleware failed to recognize this fallback behavior. It assumed the body was a legacy Gen-1 Pub/Sub event wrapper if ce-type was missing from the HTTP headers, leading to payload shift and parsing failures in downstream SDKs like firebase-functions.

This addresses one of the primary root causes of firebase/firebase-functions#1922 (which contains a live reproduction case).

Reproduction

  1. Deploy a Firestore CloudEvent trigger.
  2. Create a Firestore document with a trailing space in its ID (e.g., Test Document2 ).
  3. Eventarc falls back to Raw Pub/Sub because the HTTP specification forbids OWS (optional whitespace) at the end of header values (ce-subject: documents/... is illegal on the wire).
  4. The Cloud Framework receives the raw Pub/Sub payload but fails to extract the CloudEvent wrapper, crashing the function execution.

Fix

Extracts a rehydrateCloudEvent helper inside pubsub_middleware.ts to cleanly catch and intercept Raw Pub/Sub fallbacks:

  1. Header Promotion: Promotes all ce- prefixed (and unprefixed) Pub/Sub attributes into req.headers.
  2. Payload Extraction: Base64 decodes message.data and attempts a JSON parse (unless skipped due to non-JSON ce-datacontenttype).
  3. Safe Fallback: If JSON parsing fails (e.g. application/protobuf), it mirrors bodyParser.raw() by assigning the Buffer directly to req.body and passing it to the handler safely.

Because Express req.headers is just a JavaScript dictionary, assigning headers with trailing spaces internally perfectly bypasses the wire-level HTTP parser constraints that broke the initial Eventarc push.

Testing

  • Manual Verification: We bundled this patched framework into a local firebase-functions dependency and successfully deployed it to a production playground executing firebase deploy --only functions. Successfully verified via Cloud Logging that documents with trailing spaces trigger the fallback and process correctly.
  • Added unit test: Eventarc fallback CloudEvent (JSON payload) to verify successful promotion of attributes to headers and JSON deserialization.
  • Added unit test: Eventarc fallback CloudEvent (Unprefixed attributes & Binary payload) to ensure CloudEvents with binary contents strictly bypass JSON parsing and arrive as Raw Buffers.
  • Full suite green: 109 passing.

## Summary

When an HTTP trigger receives a CloudEvent, it strict-validates the incoming HTTP headers. However, Google Front End (GFE) aggressively rejects certain HTTP headers (e.g. `ce-subject` strings ending with a trailing space on the wire).

When Eventarc encounters these strict rejections during a push delivery, it falls back to Raw Pub/Sub message delivery, stuffing all of the critical `ce-` metadata into the Pub/Sub message attributes instead of the HTTP headers.

Previously, `legacyPubSubEventMiddleware` failed to recognize this fallback behavior. It assumed the body was a legacy Gen-1 Pub/Sub event wrapper if `ce-type` was missing from the HTTP headers, leading to payload shift and parsing failures in downstream SDKs like `firebase-functions`.

This addresses one of the primary root causes of firebase/firebase-functions#1922 (which contains a live reproduction case).

### Reproduction

1. Deploy a Firestore CloudEvent trigger.
2. Create a Firestore document with a trailing space in its ID (e.g., `Test Document2 `).
3. Eventarc falls back to Raw Pub/Sub because the HTTP specification forbids OWS (optional whitespace) at the end of header values (`ce-subject: documents/... ` is illegal on the wire).
4. The Cloud Framework receives the raw Pub/Sub payload but fails to extract the CloudEvent wrapper, crashing the function execution.

## Fix

Extracts a `rehydrateCloudEvent` helper inside `pubsub_middleware.ts` to cleanly catch and intercept Raw Pub/Sub fallbacks:
1. **Header Promotion**: Promotes all `ce-` prefixed (and unprefixed) Pub/Sub attributes into `req.headers`.
2. **Payload Extraction**: Base64 decodes `message.data` and attempts a JSON parse (unless skipped due to non-JSON `ce-datacontenttype`).
3. **Safe Fallback**: If JSON parsing fails (e.g. `application/protobuf`), it mirrors `bodyParser.raw()` by assigning the `Buffer` directly to `req.body` and passing it to the handler safely.

Because Express `req.headers` is just a JavaScript dictionary, assigning headers with trailing spaces internally perfectly bypasses the wire-level HTTP parser constraints that broke the initial Eventarc push.

## Testing

- **Manual Verification**: We bundled this patched framework into a local `firebase-functions` dependency and successfully deployed it to a production playground executing `firebase deploy --only functions`. Successfully verified via Cloud Logging that documents with trailing spaces trigger the fallback and process correctly.
- Added unit test: **Eventarc fallback CloudEvent (JSON payload)** to verify successful promotion of attributes to headers and JSON deserialization.
- Added unit test: **Eventarc fallback CloudEvent (Unprefixed attributes & Binary payload)** to ensure CloudEvents with binary contents strictly bypass JSON parsing and arrive as Raw Buffers.
- Full suite green: `109 passing`.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant