[Performance] High monitor lock contention in DefaultHeartBeatController and blockingRegister during cold startups on mid-range devices

[juanfelippopeya]

[READ] Step 1: Are you in the right place?

Yes, this issue describes performance overhead, massive thread lock contention, and CPU core starvation originating directly from the firebase-common heartbeats storage and initialization path in this repository.

[REQUIRED] Step 2: Describe your environment

• Android Studio version: Android Studio Ladybug (2024.2.1) (Profiling conducted on physical Moto G53 5G, Snapdragon 695, Android 14, API 34).
• Firebase Component: firebase-common (DefaultHeartBeatController & JavaDataStorage interactions affecting firebase-installations, firebase-messaging, firebase-perf, and firebase-sessions).
• Component version: com.google.firebase:firebase-common:22.1.0 (as resolved under Firebase BoM 34.15.0).

---

[REQUIRED] Step 3: Describe the problem

While the deterministic deadlock reported in previous issues (such as nested runBlocking calls within DataStore edits) was successfully resolved in PR #8182 (firebase-common:22.1.0), severe monitor lock contention still plagues the heartbeat storage path under high-concurrency cold startups (First Install scenario).

During early application startup, multiple modular Firebase components (FCM, Remote Config, Performance, Sessions) attempt to initialize and write/read metadata in parallel. On mid-range devices like the Moto G53 5G, this concurrent access to DefaultHeartBeatController triggers a massive thread synchronization storm.

Background thread lock queues of up to 938 ms are created. This heavy context-switching overhead saturates the CPU, forcing the OS scheduler to evict high-priority UI threads (RenderThread) to slower efficiency cores (Little Cores). This results in severe UI rendering delays, inflating EGLContext creation from a typical <5 ms to 220 ms, and triggering severe frame jank (588 ms Choreographer#doFrame).

Steps to reproduce:

1. Integrate Firebase BoM 34.15.0 (with firebase-common:22.1.0) into a medium-to-large multi-module application using multiple Firebase SDKs (FCM, Remote Config, Performance, Sessions).
2. Perform a First Install run (cold start with no cached preferences or metadata) on a mid-range octa-core device (e.g., Moto G53 5G, Snapdragon 695 with 2x Gold cores + 6x Silver cores on Android 14).
3. Capture a system trace using Perfetto spanning the application initialization and first frame rendering.
4. Analyze the trace in the Perfetto UI or via trace_processor_shell. Observe massive concurrent Thread::Init slices, thread lock contentions, and DefaultHeartBeatController monitor bottlenecks on background threads.

Perfetto Trace Evidence (Extracted via trace_processor):

Across a batch of 16 Baseline Profile test runs, 100% of the traces exhibited severe firebase-common monitor and thread lock contention, creating severe bottleneck queues:

-- Querying monitor contentions and thread lock bottlenecks on background threads:
SELECT name, ts, dur/1000000.0 as dur_ms 
FROM slice 
WHERE name LIKE '%monitor contention%' AND (name LIKE '%firebase%' OR name LIKE '%Firebase%')

-- Output Examples (Worst-Case Peak timings from RC29.3 benchmark batch):
1. Thread: "Firebase Background Thread #3" (TID 9955)
   Slice: "monitor contention with owner Firebase Background Thread #3 at void com.google.firebase.heartbeatinfo.DefaultHeartBeatController..."
   Duration: 938.67 ms (CRITICAL)

2. Thread: "Firebase-Messag" (TID 9933)
   Slice: "Lock contention on a monitor lock (blocking from com.google.firebase.messaging.FirebaseMessaging.blockingRegister)"
   Duration: 569.04 ms

3. Thread: "Firebase Background Thread #1" (TID 9915)
   Slice: "monitor contention on DefaultHeartBeatController"
   Duration: 754.75 ms

4. Thread: "Firebase Background Thread #0" (TID 9911)
   Slice: "monitor contention on DefaultHeartBeatController"
   Duration: 325.50 ms

Secondary Impact: CPU Core Starvation and Render Thread Degradation

Because of this parallel thread storm (including Thread::Init running 130+ times for newly spawned SharedPreferences and Firebase worker threads), the CPU Gold cores are fully choked.

CPU scheduling analysis (sched table) shows:

• The RenderThread is starved of big cores and is scheduled 68.9% of its active CPU time on Little Cores (CPUs 0-5) (487.32 ms out of 706.79 ms).
• This starvation degrades native GPU driver initialization, causing Creating EGLContext on the RenderThread to take an astronomical 220.65 ms (blocking the Main Thread's first layout draw for 259.99 ms).

---

Relevant Code:

Even though PR #8182 eliminated the cyclic deadlock, the underlying design of HeartBeatInfoStorage and DefaultHeartBeatController relies heavily on global Java monitor locks (synchronized methods) wrapping asynchronous/blocking data access under high concurrency.

When 5-8 sub-modules of Firebase hit these methods in parallel on different background workers at the exact same millisecond on a cold start, they queue up linearly on DefaultHeartBeatController:

// Sychronized methods in HeartBeatInfoStorage.java still trigger massive queues under high parallel load
synchronized void storeHeartBeat(long millis, String userAgentString) {
    // Highly concurrent threads queue up here while background DataStore transactions are in-flight,
    // causing up to ~938ms of monitor contention even if they eventually unlock without deadlocking.
    ...
}