Skip to content

Conversation

@benaadams
Copy link
Member

Changes

  • More reduced version of Improve Block caching and reduce lookup allocations #8943
  • Introduce block hash -> block number cache
  • Don't update ClockCache concurrent dictionary if value the same
  • Don't update cache in Block/Header finder if it came from cache in first place
  • Use smaller keys for Block and Header cache

Types of changes

What types of changes does your code introduce?

  • Optimization

Testing

Requires testing

  • No

Copy link
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull Request Overview

This PR optimizes block-related caching mechanisms in Nethermind by introducing a hash-to-number cache, preventing unnecessary cache updates, and using more efficient cache keys.

Key changes:

  • Introduces a dedicated Hash256AsKey cache for block hash to block number mappings
  • Adds cache update avoidance logic in ClockCache when values are identical
  • Tracks whether data came from cache via fromCache parameter to prevent redundant cache operations
  • Replaces ValueHash256 with Hash256AsKey for more efficient cache key storage

Reviewed Changes

Copilot reviewed 12 out of 12 changed files in this pull request and generated 1 comment.

Show a summary per file
File Description
Nethermind.Taiko/L1OriginStore.cs Updates database lookup to use Hash256 instead of ValueHash256
Nethermind.Serialization.Rlp/KeyValueStoreRlpExtensions.cs Adds fromCache output parameter and refactors cache lookup logic to use Hash256AsKey
Nethermind.Facade/Simulate/SimulateDictionaryHeaderStore.cs Implements fromCache tracking for simulated header store
Nethermind.Facade/Simulate/SimulateDictionaryBlockStore.cs Implements fromCache tracking for simulated block store
Nethermind.Core/KeyValueStoreExtensions.cs Updates key generation methods to use in ValueHash256 for better performance
Nethermind.Core/Caching/ClockCache.cs Adds value comparison logic to avoid updating cache when value is unchanged
Nethermind.Blockchain/Headers/IHeaderStore.cs Adds fromCache output parameter to Get method signature
Nethermind.Blockchain/Headers/HeaderStore.cs Implements hash-to-number cache and integrates fromCache tracking
Nethermind.Blockchain/Blocks/IBlockStore.cs Adds fromCache output parameter to Get method signature
Nethermind.Blockchain/Blocks/BlockStore.cs Implements fromCache tracking and uses Hash256AsKey for cache
Nethermind.Blockchain/BlockTree.cs Uses fromCache to conditionally cache headers and blocks
Nethermind.Blockchain.Test/Blocks/HeaderStoreTests.cs Changes test variable declarations to use interface type

Copy link
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull Request Overview

Copilot reviewed 20 out of 20 changed files in this pull request and generated 2 comments.

@LukaszRozmej
Copy link
Member

LukaszRozmej commented Oct 28, 2025

Could we add:

  • ParentHash to the cache where we have blocks and numbers.
  • On Head change actively scan through cache, double check and ensure we have all the items for Blockhash opcode? (unless a long reorg you won't actually hit DB!)

@benaadams
Copy link
Member Author

Made new bugs :(

@benaadams benaadams requested a review from asdacap October 29, 2025 02:32
@benaadams
Copy link
Member Author

  • On Head change actively scan through cache, double check and ensure we have all the items for Blockhash opcode? (unless a long reorg you won't actually hit DB!)

Not sure how this would help? (adding 256 blockhash refreshes to every block)

@benaadams benaadams merged commit fe8adcc into master Oct 29, 2025
141 of 143 checks passed
@benaadams benaadams deleted the block-caching branch October 29, 2025 10:04
benaadams added a commit that referenced this pull request Oct 29, 2025
* Don't recache if from cache

* Improve ClockCache fast path

* Use smaller keys for Block and Header caches

* Pass via in

* Add number cache

* formatting

* sp

* Actually use cache

* Remove duplicated method

* Tweak

* Change hash compare

* Break test fix speed

* Feedback

* Test fix

* null refs

* Fall back to prior implementation

* Restore test

* Less test changes

* Also do reorged block

* null ref

* Speed up forkchoice
benaadams added a commit that referenced this pull request Nov 4, 2025
benaadams added a commit that referenced this pull request Nov 11, 2025
LukaszRozmej pushed a commit that referenced this pull request Nov 12, 2025
* Revert "Improve Block related caching (#9571)"

This reverts commit fe8adcc.

* merge fix
kamilchodola added a commit that referenced this pull request Nov 13, 2025
* Change IPrecompile to use Result<byte[]> allowing to return errors from precompiles (#9561)

* Change IPrecompile to use Result<byte[]>

* whitespace

* fix benchmark

* fix tests

* Move back DataGasCost to simple long

* Some fixes

* fixes

* refactor

* expose precompile error from EVM

* ModExpPrecompile should fail on run not on DataGasCost

* fixes

* Fix tests and bubble up precompile errors

* fix build

* Another try

* Error(2)

* Choose substate error

* fixes

* Simplify handling errors

* fix PairingCheckPrecompile

* Add syntatic suggar, while preserving lazy evaluation

* fix

---------

Co-authored-by: lukasz.rozmej <[email protected]>

* Update outdated link (#9654)

* Feature/xdc prevent finalized reorg (#9655)

* override in blocktree

* fix

* unittest

* format

* fix

* Fix

* remove test

* cleanup

* Do not insert block if invalid fork

* format

* review comments

* format

* Update src/Nethermind/Nethermind.Xdc/XdcBlockTree.cs

Co-authored-by: Lukasz Rozmej <[email protected]>

---------

Co-authored-by: Lukasz Rozmej <[email protected]>

* fix: DynamicCallWithInput to skip value for DELEGATECALL/STATICCALL (#9672)

* fix: DynamicCallWithInput to skip value for DELEGATECALL/STATICCALL

* update test

* Route ColumnDb.Remove through Set(null) to use wrapper semantics (#9636)

* Changes for test blockchain (#9680)

* test blockchain changes

* format

* Improve diag Docker container (#9635)

* Fix bump-version action and adds reviewers as default (#9557)

* feat: add reviewers to the bump-version action

* feat: use checkout action

* Fix free disk space check for non existent directories (HealthChecks) (#9666)

* Create non existent db directory to get drive info

* Add db folder to ignored to pass tests

* Update OP Superchain chains (#9675)

Co-authored-by: emlautarom1 <[email protected]>
Co-authored-by: Lautaro Emanuel <[email protected]>

* Remove Nethermind.Analytics (#9683)

* Feature/block production blob limit (#9686)

* Introduce BlockProductionBlobLimit to limit number of blobs picked when producing block

* Add MaxBlobCount calculation tests

* whitespace

* Rename

* Update src/Nethermind/Nethermind.Config/IBlocksConfig.cs

Co-authored-by: Ahmad Bitar <[email protected]>

---------

Co-authored-by: Ahmad Bitar <[email protected]>

* Fix resource leak in HttpJsonRpcSubmitter (#9687)

Update HttpJsonRpcSubmitter.cs

* Remove unused BlockFilter.StartBlockNumber and simplify CreateBlockFilter (#9630)

* Remove unused BlockFilter.StartBlockNumber and simplify CreateBlockFilter

* fix ci

* formatting

* Revert "Improve Block related caching (#9571)" (#9688)

* Revert "Improve Block related caching (#9571)"

This reverts commit fe8adcc.

* merge fix

* Update Dockerfiles (#9689)

Co-authored-by: rubo <[email protected]>

* Fix Chainspec for Taiko-Hoodi (#9699)

fix transition for taiko-hoodi

* Xdc : Add XdcTestBlockchain for Module testing (#9596)

* start

* more dependencies

* build

* rename

* voting

* vote task

* bit of refactor

* invoke block produced

dont seal

* refactor context

* format

* timeout missing

* block producer

* format

* basic incomplete setup

* test

* block production test

* format

* merge cleanup

* merged

* use the constant

* use timestamp from attributes

* Merged

* Merged fix

* cleanup

* fixes

* cleanup

* commit cert before vote

propose in background

* merge and refactor

* rename

* format

* comment

* refactor to continuous loop

* remove TCM

* Format

* fix

* draft implmentation

* cleanup

* handle regenesis block in XdcTestBlockchain

* refactor to inherit from TestBlockchain

* register FromXdcContainer

* handle _fromContainer from base

* attempt rotating signers

* test fix

* test fix

* test and container setup fix

* decoder fix

* bug fixes

* init QC

* test xdc sealer rotation

* bug fixes and working now

* Format

* Update packages.lock.json

* remove

* total difficulty fix

* format

* override in blocktree

* fix

* unittest

* format

* fix build issues

* fix

* Fix

* remove test

* fixes

* format

* cleanup

* undo merge and cleanup usings

* add custom genesis builder for xdc

* format whitespaces

* Do not insert block if invalid fork

* format

* reorg test

* revert block processor

* revert cloning of header in PrepareBlockForProcessing

* reorgs tests working

* fix bad db

* fix seal validator

* fix

* format

* implement add block without commiting qc and generate extra keys for testing

* fix and refactor

* optional params

* fix

* one fix and a bit of refactor

* reorg test

* rename

* return block instead

* reorg test

* fixes

* dont set author

* 10 seconds per add block

* test blockchain changes

* format

* fixes for hotstuff module

* fix test block producer

* fixes for epoch master calculation

* format

* rename

* rollback QC test blockchain order

* snapshotmanager tests fixed

* format

* cleanup

* remove TestBlockchainTester

* cleanup

* small fix in header validator test

---------

Co-authored-by: ak88 <[email protected]>
Co-authored-by: cicr99 <[email protected]>

* XDC - Snapshot Manager refactor (#9695)

refactor of snapshot manager

* Xdc : Port BlockInfoValidator Module Tests  (#9650)

* start

* more dependencies

* build

* rename

* voting

* vote task

* bit of refactor

* invoke block produced

dont seal

* refactor context

* format

* timeout missing

* block producer

* format

* basic incomplete setup

* test

* block production test

* format

* merge cleanup

* merged

* use the constant

* use timestamp from attributes

* Merged

* Merged fix

* cleanup

* fixes

* cleanup

* commit cert before vote

propose in background

* merge and refactor

* rename

* format

* comment

* refactor to continuous loop

* remove TCM

* Format

* fix

* draft implmentation

* cleanup

* handle regenesis block in XdcTestBlockchain

* refactor to inherit from TestBlockchain

* register FromXdcContainer

* handle _fromContainer from base

* attempt rotating signers

* test fix

* test fix

* test and container setup fix

* decoder fix

* bug fixes

* init QC

* test xdc sealer rotation

* bug fixes and working now

* Format

* Update packages.lock.json

* remove

* draft impl of blockinfo validator tests

* total difficulty fix

* format

* override in blocktree

* fix

* unittest

* format

* fix build issues

* fix

* Fix

* remove test

* fixes

* format

* cleanup

* undo merge and cleanup usings

* add custom genesis builder for xdc

* format whitespaces

* Do not insert block if invalid fork

* format

* reorg test

* revert block processor

* revert cloning of header in PrepareBlockForProcessing

* reorgs tests working

* fix bad db

* fix seal validator

* fix

* format

* implement add block without commiting qc and generate extra keys for testing

* fix and refactor

* optional params

* fix

* one fix and a bit of refactor

* reorg test

* rename

* return block instead

* reorg test

* fixes

* fix failing test

* dont set author

* 10 seconds per add block

* test blockchain changes

* format

* ws fixes

* fixes for hotstuff module

* fix test block producer

* fixes for epoch master calculation

* format

* rename

* rollback QC test blockchain order

* CLEANUP

* remove unused

---------

Co-authored-by: ak88 <[email protected]>
Co-authored-by: cicr99 <[email protected]>

* Unit tests for commit in XDC QC manager (#9700)

unittests

* Xdc : Port HeaderValidator Module Tests  (#9617)

* start

* more dependencies

* build

* rename

* voting

* vote task

* bit of refactor

* invoke block produced

dont seal

* refactor context

* format

* timeout missing

* block producer

* format

* basic incomplete setup

* test

* block production test

* format

* merge cleanup

* merged

* use the constant

* use timestamp from attributes

* Merged

* Merged fix

* cleanup

* fixes

* cleanup

* commit cert before vote

propose in background

* merge and refactor

* rename

* format

* comment

* refactor to continuous loop

* remove TCM

* Format

* fix

* draft implmentation

* cleanup

* handle regenesis block in XdcTestBlockchain

* refactor to inherit from TestBlockchain

* register FromXdcContainer

* handle _fromContainer from base

* attempt rotating signers

* test fix

* test fix

* test and container setup fix

* draft port of some HeaderValidatorTests

* fix build issue

* extra tests

* decoder fix

* handle merge issues

* bug fixes

* init QC

* test xdc sealer rotation

* bug fixes and working now

* Format

* Update packages.lock.json

* remove

* fix build issues

* more build fixes

* total difficulty fix

* format

* override in blocktree

* fix

* unittest

* format

* fix build issues

* fix

* Fix

* remove test

* fixes

* format

* cleanup

* undo merge and cleanup usings

* add custom genesis builder for xdc

* format whitespaces

* Do not insert block if invalid fork

* format

* reorg test

* revert block processor

* revert cloning of header in PrepareBlockForProcessing

* reorgs tests working

* fix bad db

* fix seal validator

* fix

* format

* implement add block without commiting qc and generate extra keys for testing

* fix and refactor

* optional params

* fix

* one fix and a bit of refactor

* reorg test

* rename

* return block instead

* minor changes and fixes

* reorg test

* fixes

* bring back cache field in sealValidator

* dont set author

* 10 seconds per add block

* test blockchain changes

* format

* ws fixes

* fixes for hotstuff module

* fix test block producer

* fixes for epoch master calculation

* format

* rename

* rollback QC test blockchain order

* cleanup

---------

Co-authored-by: ak88 <[email protected]>
Co-authored-by: cicr99 <[email protected]>

* Update base-sepolia testnet matrix (#9705)

chore: update base-sepolia testnet matrix

* Comment out Linea in workflows (#9706)

chore: comment linea on our workflows

---------

Co-authored-by: Alexey Osipov <[email protected]>
Co-authored-by: lukasz.rozmej <[email protected]>
Co-authored-by: Jaye Turner <[email protected]>
Co-authored-by: ak88 <[email protected]>
Co-authored-by: Snezhkko <[email protected]>
Co-authored-by: radik878 <[email protected]>
Co-authored-by: Ruben Buniatyan <[email protected]>
Co-authored-by: Marcos Antonio Maceo <[email protected]>
Co-authored-by: Damian Orzechowski <[email protected]>
Co-authored-by: core-repository-dispatch-app[bot] <173070810+core-repository-dispatch-app[bot]@users.noreply.github.com>
Co-authored-by: emlautarom1 <[email protected]>
Co-authored-by: Lautaro Emanuel <[email protected]>
Co-authored-by: Ahmad Bitar <[email protected]>
Co-authored-by: maradini77 <[email protected]>
Co-authored-by: GarmashAlex <[email protected]>
Co-authored-by: Ben {chmark} Adams <[email protected]>
Co-authored-by: Diptanshu Kakwani <[email protected]>
Co-authored-by: Ayman Bouchareb <[email protected]>
Co-authored-by: ak88 <[email protected]>
Co-authored-by: cicr99 <[email protected]>
Co-authored-by: Carmen Irene Cabrera Rodríguez <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

6 participants