Apache Celeborn (Incubating) FAQ

[SteNicholas]

• Where should I ask questions/report bugs?

We encourage users to ask question in Mailing List and GitHub Discussions, and report bugs in GitHub Issues

• Are there IM discussion groups for Celeborn?

Yes, please join our WeChat group via Official Accounts(公众号) "Apache Celeborn孵化", or Slack Channel.
We encourage people to discuss in English, but Chinese is fine too.

• Why did my question get no responses?

MOSTLY because the question is not clear enough. How To Ask Questions The Smart Way or 提问的智慧 is a good principle for asking questions.

There are some NEGATIVE examples that won't get active responses.

• Celeborn Master/Worker does not work on local machine, is there a bug?
  (Celeborn Master/Worker 在本地机器上无法运行，是不是有 Bug？)

The Hadoop ecosystem has a long history of over a decade, it is composed of a bunch of components and is quite complex today, the "DOES NOT WORK" has countless possibilities.

• Have you checked Document - Quick Start to make sure your environment meets the Celeborn setup requirements?
• Have you read/analyzed the error messages and stacktrace before asking?
• Have you provided enough information for volunteers to help dig out? e.g.
  • components' version
  • configurations
  • stacktrace
  • logs(of each components, with context, not only the error snippets)
• How to reproduce the issue?

• Changed the code by self, or compiled by self, but does not mention that when asking questions.
  (自行修改源码、自行编译，但提问时未告知)

It absolutely misleads the volunteer who helps you dig out the issue, and wastes both time.

• If your self-compiled version does not work as expected, please try the OFFICIAL RELEASED version first to quickly identify if the issue is caused by compiling.
• If you changed the code, similar to the above case, please try the Official Released version first to quickly identify if the issue is caused by your change.
• Please provide your Baseline Version, Compiling Command, and "What's Your Change" with the question.

[SteNicholas]

The log of Celeborn Worker has failure that Sending ChunkFetchSuccess operation failed, chunk $streamChunkSlice. What's the root case of the failure?

Failure to send ChunkFetchSuccess is generally Caused by: java.io.IOException: Broken pipe, that is, the Spark Executor is disconnected. FetchHandler has a retry mechanism, therefore the failure could not affect the job execution and the problem could be ignored.

[SteNicholas]

The log of Spark Executor has exception that org.apache.celeborn.common.exception.CelebornIOException: Waiting timeout for task %s while limiting max in-flight requests to %s. How to troubleshoot the root cause?

Check the log of Spark Executor and find that the Epochs have a receive timeout. Meanwhile, check the log of Spark Driver and find that this Epochs Revive has no return value. The log of execution finally runs to the Reserve Slots, where the Reserve Slots succeed to print Reserve buffer success for shuffleId $shuffleId , otherwise fail to print Reserve buffers for $shuffleId still fail after retrying, clear buffers. It's found that this log has not been printed yet, which infers that LifecycleManager#reserveSlotsWithRetry causes this thread to hang up directly due to the exception. The thread pool does not configure the exception handler, which cause that the exception log could not be obtained and the root cause of the problem could not be investigated.

Therefore, batchHandleChangePartitionExecutors needs to set Uncaught Exception Handle via cherry pick #1962. Meanwhile, reduce the Split frequency and the number of Splits include:

• increase celeborn.worker.shuffle.partitionSplit.max, which default value is 2G.
• increase celeborn.client.shuffle.partitionSplit.threshold, which default value is 1G value and needs to be smaller than the above configuration.

[SteNicholas]

What are the reference materials of Apache Celeborn (Incubating)?

The reference materials of Apache Celeborn (Incubating) include:

• Article

  • https://mp.weixin.qq.com/s/r1mISsnoep58Q6H_SYSbRw
  • https://mp.weixin.qq.com/s/dvWf9xOOncKAhk71qY8I7A
  • https://mp.weixin.qq.com/s/-vhMrlgkpAB_WEqNZWlQ6A
  • https://mp.weixin.qq.com/s/cIwuiqc9NbQCAZE-WRbaPA
  • https://mp.weixin.qq.com/s/6ZNTOrWeaCCelU0lM41ROA
  • https://mp.weixin.qq.com/s/6ohh7COPSIJVa-W0H-LkZQ
  • https://mp.weixin.qq.com/s/tEWHwKaEfiOax-ArH24nCQ
  • https://mp.weixin.qq.com/s/Plka0G59N5cpXET8tYlEXg

• Video

  • https://www.bilibili.com/video/BV1SM4y1H7FB
  • https://www.bilibili.com/video/BV1Zw411e7nZ
  • https://www.bilibili.com/video/BV1G34y1N7NE
  • https://www.bilibili.com/video/BV1eQ4y1s72b

[waitinfuture]

When enable replication (spark.celeborn.client.push.replicate.enabled set true), the performance degrades dramatically compared to disable replication.

Check whether spark.celeborn.client.reserveSlots.rackaware.enabled is enabled. If do, try disable this config. If it helps, check whether Celeborn's workers spread evenly across racks, and also check the network latency across racks.