Add read-only waiting proof inspection command

[ellemouton]

What changed

Adds a temporary, read-only inspectwaitingproofs command for diagnosing lnd startup failures caused by records in the waitingproofs bucket.

The command:

• opens an offline channel.db copy with bbolt ReadOnly: true;
• reports the raw channel DB version key status, including a missing metadata/dbp key;
• reports the waiting-proof bucket state;
• identifies V1 records, V2 candidates, and startup-fatal decode records;
• reproduces and highlights invalid public key: unsupported format: 3d;
• distinguishes a structurally clean legacy V1 record from a typed-record mismatch;
• supports human-readable and JSON output;
• never repairs, deletes, or otherwise writes database content.

This is intentionally opened as a draft because it is primarily a temporary debugging helper for an active investigation.

Background

An lnd v0.21.1 node is crash-looping immediately after unlock with:

invalid public key: unsupported format: 3d

The leading byte of a legacy remote waiting proof is isRemote=0x01. The v0.21 typed decoder can interpret that same byte as the V2 proof type, then interpret shifted channel ID bytes as a MuSig2 nonce. If the resulting nonce starts with 0x3d, btcec.ParsePubKey returns the reported error.

The affected operator's inspection output showed:

db_version=unknown
waitingproofs_bucket=present
[decode_error] ... error="invalid public key: unsupported format: 3d" legacy=clean_legacy_v1 key_status="legacy_length_key"
[decode_error] ... error="unexpected EOF decoding AnnounceSignatures1"
summary total=2 v1_ok=0 v2_candidates=0 fatal=2 exact_unsupported_format_3d=1
verdict=exact_reported_crash_reproduced

The missing raw DB version key is important. lnd's FetchMeta returns the latest DB version when the metadata bucket exists but metadata/dbp is absent. That means startup logs can report the latest version even though the raw version key is missing, causing mandatory migrations such as migration 35 to be skipped. Locally, a fresh channeldb created through lnd's init path has metadata present and dbp absent, which reproduces this metadata condition.

With the latest version of this command, that first line is explicit:

db_version=unknown db_version_status=db_version_key_missing

For this failure mode, both 9-byte records are clean legacy V1 waiting proofs. The remote record reproduces the exact 0x3d crash through the V2 misdecode path; the local record enters the V1 path and fails with shifted EOF.

How to use

Stop lnd first and make an offline copy of channel.db:

cp --preserve=all \
  /path/to/lnd/data/graph/mainnet/channel.db \
  /tmp/channel.db.waitingproof-check

Inspect the copy:

chantools --nologfile inspectwaitingproofs \
  --channeldb /tmp/channel.db.waitingproof-check

For machine-readable output:

chantools --nologfile inspectwaitingproofs \
  --json \
  --channeldb /tmp/channel.db.waitingproof-check

Key verdicts:

• exact_reported_crash_reproduced: the store contains a record that produces the reported 3d error.
• waiting_proof_store_ruled_out: the bucket is absent or empty; inspect another startup pubkey path such as wtclient.db.
• other_startup_fatal_records_found: the store has fatal records, but they do not match the reported error.
• needs_full_v2_decode: a V2-looking record passed nonce parsing and requires the full v0.21 decoder.
• reported_crash_not_found: no record in this bucket explains the reported error.

For an exact match, legacy=clean_legacy_v1 with a 9-byte key means the value and key satisfy migration 35's legacy layout checks. A typed key/value type disagreement instead points toward a damaged or externally modified typed record.

Validation

• Added tests for the exact clean-legacy 0x3d reproduction.
• Added tests for missing metadata/dbp plus both local and remote legacy waiting proofs.
• Added a command-output test that verifies the human-readable lines for the operator-shaped DB.
• Added tests for v35 metadata, healthy typed V1 records, type-flipped typed records, unknown types, truncated records, and absent/empty buckets.
• Tests verify the database SHA-256 is unchanged after inspection.
• go test ./cmd/chantools -run 'TestInspectWaitingProofs' -count=1
• Built the command with Go 1.25.11.
• Locally reproduced the lnd metadata condition with a temporary test: has_metadata_bucket=true has_db_version_key=false on a fresh channeldb.

The full command-package suite has an unrelated pointer-address normalization failure in TestCompactDBAndDumpChannels on Darwin/arm64; the same failure reproduced before this update.

[ellemouton]

Temporary user instructions for the repairwaitingproofs workaround:

Temporary lnd startup repair for waitingproofs issue

This should only be run while lnd/litd is fully stopped.

1. Stop lnd/litd

Make sure lnd/litd is not running before touching channel.db.

2. Find your channel.db

The usual mainnet path is:

~/.lnd/data/graph/mainnet/channel.db

If you are using litd, Umbrel, Start9, Docker, or another packaged setup, your path may be different.

3. Make a manual backup

Before running the repair, copy channel.db somewhere safe.

Example:

cp ~/.lnd/data/graph/mainnet/channel.db \
  ~/channel.db.before-waitingproof-repair

The repair command also creates its own backup, but please keep this manual backup too.

4. Run the dry run first

Run:

./chantools --nologfile repairwaitingproofs \
  --channeldb ~/.lnd/data/graph/mainnet/channel.db

Expected output should include:

dry_run=true
db_version_status=db_version_key_missing
waitingproofs_bucket=present
summary total=2 clean_legacy=2 typed=0 other=0 would_migrate=2 migrated=0 deleted_legacy=0
verdict=would_repair

If the output does not look like this, stop here and send us the output.
Do not run with --commit unless the dry run says verdict=would_repair.

5. Run the repair

If the dry run output matches, run:

./chantools --nologfile repairwaitingproofs \
  --channeldb ~/.lnd/data/graph/mainnet/channel.db \
  --commit

Expected output should include:

dry_run=false
backup=/path/to/channel.db.repairwaitingproofs.<timestamp>.bak
db_version_status=db_version_key_missing
waitingproofs_bucket=present
summary total=2 clean_legacy=2 typed=0 other=0 would_migrate=2 migrated=2 deleted_legacy=2
verdict=repaired

Keep the backup file shown in the output.

6. Verify the repair

Run:

./chantools --nologfile inspectwaitingproofs \
  --channeldb ~/.lnd/data/graph/mainnet/channel.db

Expected output should include:

summary total=2 v1_ok=2 v2_candidates=0 fatal=0 exact_unsupported_format_3d=0
verdict=reported_crash_not_found

If the output still shows fatal=1 or fatal=2, stop and send us the output.

7. Start lnd/litd again

After verification passes, start lnd/litd again.

8. Keep all backups

Please keep:

- your manual backup
- the .bak file created by chantools
- the output from all three commands

until the node has restarted successfully.

Additional note:

This is a temporary repair for the waitingproofs records only. It does not modify the DB version metadata. That is intentional. The later official lnd fix should handle the missing DB version metadata safely.