Discussed in #1445
Originally posted by ruslanen July 1, 2026
@Slach hi,
We use ClickHouse (different versions) + clickhouse-backup in production, on setups with a lot of
tables (for example: ~50K MergeTree + ~500K Join, ~954 GiB).
I noticed that hashing takes a really long time during backup.
Looks like fetching hash_of_all_files from system.parts runs together with FREEZE and they contend on some mutex inside ClickHouse - FREEZE jumps from ~100ms to ~5000ms per table because of it.
I tried just skipping that lookup and the difference is huge:
- with hashing: FREEZE ~5029ms,
create ~9.5h
- without: FREEZE ~116ms,
create ~26min
I know hash_of_all_files isn't useless - incremental create --diff-from-remote uses it to
dedup parts against the previous full backup, so skipping it on full backups would quietly hurt
future increments.
That's why I only did it behind an opt-in flag, not automatically.
Concretely, what I did:
- added a config field
clickhouse.skip_hash_of_all_files (env CLICKHOUSE_SKIP_HASH_OF_ALL_FILES);
- added a
--skip-hash-of-all-files CLI flag to create / create_remote / watch;
- gated the existing lookup on it in
AddTableToLocalBackup - one extra condition, so the default
behavior is unchanged:
// before
useHashOfAllFiles := version >= 19011000
// after
useHashOfAllFiles := version >= 19011000 && !b.cfg.ClickHouse.SkipHashOfAllFiles
When the flag is off (default) everything works exactly as today; when it's on, the per-part
hash_of_all_files read from system.parts is skipped and the FREEZE contention goes away.
My two main questions:
- Would you accept a PR with this config/flag as-is? If yes, which shape do you prefer:
- just keep the opt-in flag and document the diff-from-remote caveat,
- or keep the flag but warn/block when it's used with
--diff-from-remote?
- Or is there a better way to speed up the hashing itself - e.g. batch/defer the
system.parts
lookup so it doesn't fight with FREEZE - so no flag would be needed at all?
Discussed in #1445
Originally posted by ruslanen July 1, 2026
@Slach hi,
We use ClickHouse (different versions) + clickhouse-backup in production, on setups with a lot of
tables (for example: ~50K MergeTree + ~500K Join, ~954 GiB).
I noticed that hashing takes a really long time during backup.
Looks like fetching
hash_of_all_filesfromsystem.partsruns together withFREEZEand they contend on some mutex inside ClickHouse - FREEZE jumps from ~100ms to ~5000ms per table because of it.I tried just skipping that lookup and the difference is huge:
create~9.5hcreate~26minI know
hash_of_all_filesisn't useless - incrementalcreate --diff-from-remoteuses it todedup parts against the previous full backup, so skipping it on full backups would quietly hurt
future increments.
That's why I only did it behind an opt-in flag, not automatically.
Concretely, what I did:
clickhouse.skip_hash_of_all_files(envCLICKHOUSE_SKIP_HASH_OF_ALL_FILES);--skip-hash-of-all-filesCLI flag tocreate/create_remote/watch;AddTableToLocalBackup- one extra condition, so the defaultbehavior is unchanged:
When the flag is off (default) everything works exactly as today; when it's on, the per-part
hash_of_all_filesread fromsystem.partsis skipped and the FREEZE contention goes away.My two main questions:
--diff-from-remote?system.partslookup so it doesn't fight with
FREEZE- so no flag would be needed at all?