Introduce `evicted-at`/`last-evicted` timestamps #1839

evanlinjin · 2025-02-16T09:13:28Z

Partially Fixes #1740.
Replaces #1765.
Replaces #1811.

Description

This PR allows the receiving structures (bdk_chain, bdk_wallet) to detect and evict incoming transactions that are double spent (cancelled).

We add a new field to TxUpdate (TxUpdate::evicted_ats), which in turn, updates the last_evicted timestamps that are tracked/persisted by TxGraph. This is similar to how TxUpdate::seen_ats update the last_seen timestamp in TxGraph. Transactions with a last_evicted timestamp higher than their last_seen timestamp are considered evicted.

SpkWithExpectedTxids is introduced in SpkClient to track expected Txids for each spk. During a sync, if any Txids from SpkWithExpectedTxids are not in the current history of an spk obtained from the chain source, those Txids are considered missing. Support for SpkWithExpectedTxids has been added to both bdk_electrum and bdk_esplora chain source crates.

The canonicalization algorithm is updated to disregard transactions with a last_evicted timestamp greater than or equal to their last_seen timestamp, except in cases where transitivity rules apply.

Notes to the reviewers

This PR does not fix #1740 for block-by-block chain source (such as bdk_bitcoind_rpc). This work is done in a separate PR (#1857).

Changelog notice

Add TxUpdate::evicted_ats which tracks transactions that have been replaced and are no longer present in mempool.
Change TxGraph to track last_evicted timestamps. This is when a transaction is last marked as missing from the mempool.
The canonicalization algorithm now disregards transactions with a last_evicted timestamp greater than or equal to it's last_seen timestamp, except when a canonical descendant exists due to rules of transitivity.
Add SpkWithExpectedTxids in spk_client which keeps track of expected Txids for each spk.
Change bdk_electrum and bdk_esplora to understand SpkWithExpectedTxids.
Add SyncRequestBuilder::expected_txids_of_spk method which adds an association between txids and spks.

Checklists

All Submissions:

I've signed all my commits
I followed the contribution guidelines
I ran cargo fmt and cargo clippy before committing

New Features:

I've added tests for the new feature
I've added docs for the new feature

Bugfixes:

This pull request breaks the existing API
I've added tests to reproduce the issue which are now passing
I'm linking the issue being fixed by this PR

This is a set of evicted txs from the mempool.

crates/chain/src/tx_graph.rs

The evicted-at and last-evicted timestamp informs the `TxGraph` when the transaction was last deemed as missing (evicted) from the mempool. The canonicalization algorithm is changed to disregard transactions with a last-evicted timestamp greater or equal to it's last-seen timestamp. The exception is when we have a canonical descendant due to rules of transitivity. Update rusqlite_impl to persist `last_evicted`. Also update docs: * Remove duplicate paragraphs about `ChangeSet`s. * Add "Canonicalization" section which expands on methods that require canonicalization and the associated data types used in the canonicalization algorithm.

The spk history returned from Electrum should have these txs present. Any missing tx will be considered evicted from the mempool.

* `TxGraph::try_list_expected_spk_txids` * `TxGraph::list_expected_spk_txids` * `IndexedTxGraph::try_list_expected_spk_txids` * `IndexedTxGraph::list_expected_spk_txids` Co-authored-by: valued mammal <[email protected]> Co-authored-by: Wei Chen <[email protected]>

Co-authored-by: Wei Chen <[email protected]>

Also add `detect_receive_tx_cancel` test. Also rm `miniscript` dependency. Update ci to rm `miniscript/no-std` for `bdk_esplora`. Co-authored-by: Wei Chen <[email protected]>

Methods `next_spk` and `iter_spks` are removed. These are superceded with `next_spk_with_expected_txids` and `iter_spks_with_expected_txids`.

evanlinjin · 2025-03-14T02:20:11Z

ut ACK 1eb0c32

Thanks for the fix and all the new docs, looks like a solid solution. Before (or after) this goes in it would be nice to have some confirmation from @ErikDeSmedt that it fixes #1740 for him (at least using electrum and esplora).

@notmandatory

The tests that we introduce in this PR are based on @ErikDeSmedt's tests (but we target for esplora/electrum in our PR here).

This PR does not fix evicted-tx-detection for bdk_bitcoind_rpc. That work is in a separate PR: #1857.

So I think it is okay to merge this without Erik's confirmation.

LLFourn

ACK 0a20724

LagginTimes

One last nit that I saw.

LagginTimes · 2025-03-14T04:51:04Z

crates/chain/src/indexed_tx_graph.rs

+    ///
+    /// This is used to fill [`SyncRequestBuilder::expected_spk_txids`](bdk_core::spk_client::SyncRequestBuilder::expected_spk_txids).
+    ///
+    /// The spk index range can be contrained with `range`.


Suggested change

/// The spk index range can be contrained with `range`.

/// The spk index range can be constrained with `range`.

I wonder how easy it would be to set up CI to check for spelling mistakes. Might save time.

ValuedMammal · 2025-03-16T20:07:55Z

ACK 0a20724

evanlinjin force-pushed the evicted_at branch 5 times, most recently from 6dc26c4 to 0fcc097 Compare February 21, 2025 10:48

LagginTimes force-pushed the evicted_at branch 2 times, most recently from 540a83f to 5107b3a Compare February 26, 2025 14:26

evanlinjin force-pushed the evicted_at branch 6 times, most recently from 5718a3e to 07903a8 Compare February 28, 2025 03:34

evanlinjin changed the title ~~Temporary PR: Evicted at~~ Introduce evicted-at/last-evicted timestamps Feb 28, 2025

evanlinjin marked this pull request as ready for review February 28, 2025 04:00

evanlinjin mentioned this pull request Feb 28, 2025

Introduce evicted-at/last-evicted timestamps #1811

Closed

8 tasks

evanlinjin assigned evanlinjin and LagginTimes Feb 28, 2025

evanlinjin added bug Something isn't working api A breaking API change labels Feb 28, 2025

evanlinjin added this to BDK Wallet Feb 28, 2025

evanlinjin requested review from LLFourn, LagginTimes and nymius February 28, 2025 04:01

evanlinjin force-pushed the evicted_at branch from 07903a8 to 9d6c795 Compare February 28, 2025 04:03

evanlinjin mentioned this pull request Feb 28, 2025

feat(rpc)!: Update Emitter::mempool to support evicted_at #1857

Merged

8 tasks

notmandatory moved this to Needs Review in BDK Wallet Feb 28, 2025

evanlinjin requested a review from notmandatory March 2, 2025 06:13

notmandatory removed this from BDK Wallet Mar 4, 2025

notmandatory added this to BDK Chain Mar 4, 2025

evanlinjin force-pushed the evicted_at branch from ecc6799 to e177e90 Compare March 14, 2025 00:01

feat(core): Add TxUpdate::evicted_ats

ae0336b

This is a set of evicted txs from the mempool.

evanlinjin force-pushed the evicted_at branch 2 times, most recently from 0f10b55 to 806a7d0 Compare March 14, 2025 00:40

evanlinjin requested a review from nymius March 14, 2025 00:41

LLFourn reviewed Mar 14, 2025

View reviewed changes

crates/chain/src/tx_graph.rs Outdated Show resolved Hide resolved

evanlinjin and others added 7 commits March 14, 2025 13:15

feat(core): Add expected txids to SyncRequest spks

b38569f

The spk history returned from Electrum should have these txs present. Any missing tx will be considered evicted from the mempool.

feat(electrum): Handle spks with expected txids

3ab4994

Co-authored-by: Wei Chen <[email protected]>

feat(esplora): Handle spks with expected txids

f42d5a8

Also add `detect_receive_tx_cancel` test. Also rm `miniscript` dependency. Update ci to rm `miniscript/no-std` for `bdk_esplora`. Co-authored-by: Wei Chen <[email protected]>

feat(core)!: Remove redundant SyncRequest methods

1f8fc17

Methods `next_spk` and `iter_spks` are removed. These are superceded with `next_spk_with_expected_txids` and `iter_spks_with_expected_txids`.

feat(examples): Update example crates to use expected_spk_txids

0a20724

evanlinjin force-pushed the evicted_at branch from 466fb3b to 0a20724 Compare March 14, 2025 02:16

LLFourn approved these changes Mar 14, 2025

View reviewed changes

LagginTimes reviewed Mar 14, 2025

View reviewed changes

evanlinjin merged commit f0faff6 into bitcoindevkit:master Mar 15, 2025
22 of 23 checks passed

github-project-automation bot moved this from Needs Review to Done in BDK Chain Mar 15, 2025

ValuedMammal mentioned this pull request Apr 3, 2025

Release wallet-1.2.0 #1923

Closed

41 tasks

This was referenced Apr 6, 2025

Socratic Seminar #30 Topics NashBitDevs/nashbitdevs.org#82

Closed

DC BitDevs 014 Topic Suggestion dcbitdevs/dcbitdevs.github.io#24

Closed

evanlinjin mentioned this pull request Apr 10, 2025

TxBuilder continously tries to spend missing output, Wallet unable to spend any funds bitcoindevkit/bdk_wallet#202

Closed

ValuedMammal mentioned this pull request Apr 12, 2025

deps!: update bdk_chain to 0.22.0 bitcoindevkit/bdk_wallet#208

Merged

6 tasks

notmandatory added this to the 2.0.0 milestone Apr 24, 2025

notmandatory added this to BDK Wallet Apr 24, 2025

notmandatory moved this to In Progress in BDK Wallet Apr 24, 2025

notmandatory removed this from BDK Wallet Apr 24, 2025

ValuedMammal mentioned this pull request Apr 30, 2025

Release bdk_chain v0.22.0 #1945

Closed

39 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Introduce `evicted-at`/`last-evicted` timestamps #1839

Introduce `evicted-at`/`last-evicted` timestamps #1839

Uh oh!

evanlinjin commented Feb 16, 2025 •

edited

Loading

Uh oh!

Uh oh!

evanlinjin commented Mar 14, 2025 •

edited

Loading

Uh oh!

LLFourn left a comment

Uh oh!

LagginTimes left a comment

Uh oh!

LagginTimes Mar 14, 2025

Uh oh!

evanlinjin Mar 14, 2025

Uh oh!

Uh oh!

ValuedMammal commented Mar 16, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

	/// The spk index range can be contrained with `range`.
	/// The spk index range can be constrained with `range`.

Introduce evicted-at/last-evicted timestamps #1839

Introduce evicted-at/last-evicted timestamps #1839

Uh oh!

Conversation

evanlinjin commented Feb 16, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Notes to the reviewers

Changelog notice

Checklists

All Submissions:

New Features:

Bugfixes:

Uh oh!

Uh oh!

evanlinjin commented Mar 14, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

LLFourn left a comment

Choose a reason for hiding this comment

Uh oh!

LagginTimes left a comment

Choose a reason for hiding this comment

Uh oh!

LagginTimes Mar 14, 2025

Choose a reason for hiding this comment

Uh oh!

evanlinjin Mar 14, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

ValuedMammal commented Mar 16, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

Introduce `evicted-at`/`last-evicted` timestamps #1839

Introduce `evicted-at`/`last-evicted` timestamps #1839

evanlinjin commented Feb 16, 2025 •

edited

Loading

evanlinjin commented Mar 14, 2025 •

edited

Loading