Skip to content

Conversation

@EthanYuan
Copy link
Collaborator

@EthanYuan EthanYuan commented Apr 15, 2025

Problem:

The persisted bdk_wallet::ChangeSet (keychain) grows significantly over time, mainly due to local_chain history. This large size (more than 1 MB) hinders efficient synchronization with VSS.

Solution:

This PR implements periodic keychain compaction within the start_sync loop:

  1. Checks the persisted keychain size each loop iteration.
  2. If size exceeds KEYCHAIN_COMPACTION_SIZE_THRESHOLD_BYTES (256 KB):
    • The reason why it is not set to a smaller size, such as 128 KB or 64 KB, is that some super users, themselves, have a very large amount of unspent utxo. this optimal value needs to be further observed and adjusted.
    • Synchronously rebuilds the wallet state by creating a new Wallet instance and performing OnChainWallet::full_scan (using Esplora, stop_gap=50).
    • Persists the new wallet's state using storage.restore_changes, overwriting the old keychain. (restore_changes attempts backup).
    • Only if persistence succeeds, the in-memory wallet instance is replaced with the new one.

Test

  • local tests

keychian compaction log(10700 Bytes just for test):

2025-04-18 01:37:04.841 6bf4 INFO  [mutiny_core::onchain:882] Keychain size threshold exceeded 10700 Bytes, spawning simplified compaction task.

2025-04-18 01:37:04.843 6bf4 DEBUG [mutiny_core::onchain:948] PRE-COMPACTION size: 45044 bytes. Approx component sizes (bytes): LocalChain=44512, TxGraph=50, Indexer=88

2025-04-18 01:37:07.947 6bf4 DEBUG [mutiny_core::onchain:948] POST-COMPACTION size: 547 bytes. Approx component sizes (bytes): LocalChain=83, TxGraph=50, Indexer=20

2025-04-18 01:37:07.986 6bf4 INFO  [mutiny_core::onchain:927] Keychain compaction completed successfully.
2025-04-18 01:37:07.986 6bf4 INFO  [mutiny_core::onchain:928] Keychain compaction took 3 seconds

keychain backup:

image

@EthanYuan EthanYuan requested a review from Flouse April 15, 2025 12:47
@EthanYuan EthanYuan force-pushed the fix-keychain-size-issue-vss branch from 2679e9d to e99c765 Compare April 15, 2025 12:51
@Flouse Flouse requested a review from ShookLyngs April 16, 2025 00:49
@EthanYuan EthanYuan force-pushed the fix-keychain-size-issue-vss branch 2 times, most recently from 6ec40dc to 15282c1 Compare April 17, 2025 07:09
@EthanYuan EthanYuan force-pushed the fix-keychain-size-issue-vss branch from 15282c1 to 4597b31 Compare April 17, 2025 07:11
@EthanYuan EthanYuan marked this pull request as ready for review April 18, 2025 01:56
Co-authored-by: Flouse <1297478+Flouse@users.noreply.github.com>
@EthanYuan EthanYuan merged commit 2ea8a35 into utxostack:develop Apr 22, 2025
9 checks passed
@EthanYuan EthanYuan deleted the fix-keychain-size-issue-vss branch April 22, 2025 02:08
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants