Deploying Apache Celebron for Large-Scale Spark SQL Workloads

[sid-habu]

We are exploring the deployment of Apache Celebron as an external shuffle service for Spark on Kubernetes. Our Spark environment runs multiple concurrent and isolated Spark SQL jobs, each potentially shuffling anywhere from a few gigabytes to tens of terabytes of data. Given these workload characteristics, we are evaluating the best approach for deploying Celebron.

According to the Celebron deployment guide, worker pods should be placed on nodes with local SSD volumes for optimal performance. A key question we have is whether to:

Deploy a single large Celebron cluster with multiple workers that collectively provide sufficient disk space (e.g., a cluster with ~50-100 TB of local SSD storage), serving all concurrent jobs.

Deploy multiple smaller, job-specific Celebron clusters, where each job or a group of jobs gets a dedicated shuffle service instance.

A centralized Celebron cluster could simplify resource management and improve disk utilization across workloads. However, it also introduces potential bottlenecks, failure domains, and increased cross-job interference, especially if a few large shuffle-heavy jobs dominate the cluster.

On the other hand, running multiple independent Celebron clusters (e.g., per namespace or per workload category) could provide better isolation but may lead to underutilization of local SSD resources.

Has anyone deployed Celebron at this scale? What trade-offs should we consider when deciding between a single large Celebron cluster vs. multiple smaller clusters? Are there best practices for tuning Celebron’s worker distribution and disk allocation?

Looking forward to insights from the community!

[sid-habu]

@SteNicholas Do you have any insights on the above deployment related question?

[swapneshkumar-d11]

Facing the same issue. Waiting for insights from the community

[ssharma]

Appreciate any inputs from the community. The project looks very promising 🙏🏽

[jakubfijolek]

I know this question was asked long time ago but as it pops up on google results I will reply either way.

Our setup: 3 separate spark clusters in different regions. Each one: 50-600 Yarn nodemanagers (Spark on Yarn) on EC2 ASG, scaled dynamically. Shuffle sizes hitting up >50TB on regular basis. 3 static celeborn nodes + single master (we had issues with multi master) + 0-20 dynamic celeborn nodes triggered by autoscaler when large jobs are submitted.
We are using celeborn since Sep 2025

We had single cluster that is set up across multiple geo zones an uses celeborn tags to route the shuffle files to specific celeborn workers - it works without issues although it's slightly counter intuitive to set up initially. Currently it's split to multiple per-region clusters.

My general experience: as long as cluster is in same network (same region or DC) it's better to use one big cluster, and celeborn tags to isolate the load it provides better scaling and allows to fallback to empty tag to use whole cluster.

I've did not had issues with blending multiple concurrent spark jobs on same celeborn cluster. It handles mixed workloads very well and as long as they don't saturate disk space. CPU/MEMORY is rarely an issue as long as shuffle partitions are sensibly sized (64-512MB)

Was it worth for us to switch to celeborn from spark shuffles. YES the only bad thing is that we did not do it sooner. We have very agressive autoscaling for NM and long running jobs had regularly run into issues where nodes get either deprovisioned before shuffle data was consumed or cluster was scaled up and idling just to keep shuffle data available for couple writer threads. With celeborn we were able to decouple this completely and save a lot of $. Performance wise on ~8 celeborn workers we hit ~15-20GB/s read/write throughput for shuffles which is very respectable and scales very well as more nodes are added.

IMHO it's a no-brainer for any bigger spark deployment with autoscaling. For static size or small clusters might not be worth the time and cost.

[jsjasonseba]

Hi @jakubfijolek thanks for sharing this, super helpful.

I’m especially interested in this part:

3 static celeborn nodes + single master + 0-20 dynamic celeborn nodes triggered by autoscaler when large jobs are submitted.

Could you share a bit more detail on how you implemented the dynamic Celeborn workers?

For example:

• What event or signal do you use to trigger the autoscaler for the dynamic Celeborn nodes? Is it something KEDA-like based on metrics/events, such as pending Spark/YARN jobs or based on some submission logic?
• For scale-down, how do you coordinate the autoscaler with Celeborn decommissioning? Does the autoscaler wait until the Celeborn worker has fully drained / finished decommissioning before terminating the node, or do you use a fixed grace period before the node is removed?

We are evaluating a similar setup for Spark on Kubernetes, so I’m trying to understand how an autoscaler should interact with Celeborn’s worker registration and decommissioning lifecycle.

[jakubfijolek]

For scaleup i have 2 drivers.

1. As majority of my workloads are run over JDBC on fleet of long running thriftservers 1st signal is simply total celeborn cluster utilization (polled from celeborn master API). This is to avoid saturating particular celeborn cluster by one job and blocking everything. This assumes any particular shuffle will easily fit the cluster base capacity and covers the case where no-of-jobs * their shuffle requirements go above cluster size. Each sparkSQL query either way have their own LifecycleManager so as long as there is enough capacity when the query starts it should finish successfully

2. For jobs that I know will need a lot of shuffle (ex. insert into table A as select from table B with repartition) i either do it manually for one offs or implement as part of Airflow dag to send scaleup signal. In most of the cases shuffle space needed can be roughly estimated by dataset size + replication multiplier (if used) + overhead multiplier (hardcoded)

For scaledown signal is either utilization <threshold in 2nd case heavy job sends scaledown signal on exit.

Decomissioning
For now I'm simply using following sequence
Exclude workers to be scaled down, wait, trigger scaledown on ASG, decomission workers (ASG lifecycle hook), terminate instances.

I dont have this extensively tested and it's likely I'm running into cases where the node decomissions before data is fully expired but beauty of spark is that at worst case scenario it will do partial re-run of previous stage.
https://celeborn.apache.org/docs/latest/decommissioning/

[jsjasonseba]

Thanks for sharing, this makes sense.

I was mostly thinking about case 1, autoscaling from cluster utilization metrics. But your response gave me good insight into case 2, where some jobs can have much bigger shuffle needs than the others, so metric-based scaling may not be enough.

I also recently discovered this PR discussion related to HPA-based scaling, which might be relevant for a Kubernetes-based autoscaling of case 1 #2776 (comment)

I’m planning to try this on EKS and share what I find.