-
Notifications
You must be signed in to change notification settings - Fork 623
Improve Block related caching #9571
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Pull Request Overview
This PR optimizes block-related caching mechanisms in Nethermind by introducing a hash-to-number cache, preventing unnecessary cache updates, and using more efficient cache keys.
Key changes:
- Introduces a dedicated
Hash256AsKeycache for block hash to block number mappings - Adds cache update avoidance logic in
ClockCachewhen values are identical - Tracks whether data came from cache via
fromCacheparameter to prevent redundant cache operations - Replaces
ValueHash256withHash256AsKeyfor more efficient cache key storage
Reviewed Changes
Copilot reviewed 12 out of 12 changed files in this pull request and generated 1 comment.
Show a summary per file
| File | Description |
|---|---|
| Nethermind.Taiko/L1OriginStore.cs | Updates database lookup to use Hash256 instead of ValueHash256 |
| Nethermind.Serialization.Rlp/KeyValueStoreRlpExtensions.cs | Adds fromCache output parameter and refactors cache lookup logic to use Hash256AsKey |
| Nethermind.Facade/Simulate/SimulateDictionaryHeaderStore.cs | Implements fromCache tracking for simulated header store |
| Nethermind.Facade/Simulate/SimulateDictionaryBlockStore.cs | Implements fromCache tracking for simulated block store |
| Nethermind.Core/KeyValueStoreExtensions.cs | Updates key generation methods to use in ValueHash256 for better performance |
| Nethermind.Core/Caching/ClockCache.cs | Adds value comparison logic to avoid updating cache when value is unchanged |
| Nethermind.Blockchain/Headers/IHeaderStore.cs | Adds fromCache output parameter to Get method signature |
| Nethermind.Blockchain/Headers/HeaderStore.cs | Implements hash-to-number cache and integrates fromCache tracking |
| Nethermind.Blockchain/Blocks/IBlockStore.cs | Adds fromCache output parameter to Get method signature |
| Nethermind.Blockchain/Blocks/BlockStore.cs | Implements fromCache tracking and uses Hash256AsKey for cache |
| Nethermind.Blockchain/BlockTree.cs | Uses fromCache to conditionally cache headers and blocks |
| Nethermind.Blockchain.Test/Blocks/HeaderStoreTests.cs | Changes test variable declarations to use interface type |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Pull Request Overview
Copilot reviewed 20 out of 20 changed files in this pull request and generated 2 comments.
|
Could we add:
|
|
Made new bugs :( |
Not sure how this would help? (adding 256 blockhash refreshes to every block) |
062c906 to
d332636
Compare
d332636 to
2651c16
Compare
* Don't recache if from cache * Improve ClockCache fast path * Use smaller keys for Block and Header caches * Pass via in * Add number cache * formatting * sp * Actually use cache * Remove duplicated method * Tweak * Change hash compare * Break test fix speed * Feedback * Test fix * null refs * Fall back to prior implementation * Restore test * Less test changes * Also do reorged block * null ref * Speed up forkchoice
This reverts commit fe8adcc.
* Change IPrecompile to use Result<byte[]> allowing to return errors from precompiles (#9561) * Change IPrecompile to use Result<byte[]> * whitespace * fix benchmark * fix tests * Move back DataGasCost to simple long * Some fixes * fixes * refactor * expose precompile error from EVM * ModExpPrecompile should fail on run not on DataGasCost * fixes * Fix tests and bubble up precompile errors * fix build * Another try * Error(2) * Choose substate error * fixes * Simplify handling errors * fix PairingCheckPrecompile * Add syntatic suggar, while preserving lazy evaluation * fix --------- Co-authored-by: lukasz.rozmej <[email protected]> * Update outdated link (#9654) * Feature/xdc prevent finalized reorg (#9655) * override in blocktree * fix * unittest * format * fix * Fix * remove test * cleanup * Do not insert block if invalid fork * format * review comments * format * Update src/Nethermind/Nethermind.Xdc/XdcBlockTree.cs Co-authored-by: Lukasz Rozmej <[email protected]> --------- Co-authored-by: Lukasz Rozmej <[email protected]> * fix: DynamicCallWithInput to skip value for DELEGATECALL/STATICCALL (#9672) * fix: DynamicCallWithInput to skip value for DELEGATECALL/STATICCALL * update test * Route ColumnDb.Remove through Set(null) to use wrapper semantics (#9636) * Changes for test blockchain (#9680) * test blockchain changes * format * Improve diag Docker container (#9635) * Fix bump-version action and adds reviewers as default (#9557) * feat: add reviewers to the bump-version action * feat: use checkout action * Fix free disk space check for non existent directories (HealthChecks) (#9666) * Create non existent db directory to get drive info * Add db folder to ignored to pass tests * Update OP Superchain chains (#9675) Co-authored-by: emlautarom1 <[email protected]> Co-authored-by: Lautaro Emanuel <[email protected]> * Remove Nethermind.Analytics (#9683) * Feature/block production blob limit (#9686) * Introduce BlockProductionBlobLimit to limit number of blobs picked when producing block * Add MaxBlobCount calculation tests * whitespace * Rename * Update src/Nethermind/Nethermind.Config/IBlocksConfig.cs Co-authored-by: Ahmad Bitar <[email protected]> --------- Co-authored-by: Ahmad Bitar <[email protected]> * Fix resource leak in HttpJsonRpcSubmitter (#9687) Update HttpJsonRpcSubmitter.cs * Remove unused BlockFilter.StartBlockNumber and simplify CreateBlockFilter (#9630) * Remove unused BlockFilter.StartBlockNumber and simplify CreateBlockFilter * fix ci * formatting * Revert "Improve Block related caching (#9571)" (#9688) * Revert "Improve Block related caching (#9571)" This reverts commit fe8adcc. * merge fix * Update Dockerfiles (#9689) Co-authored-by: rubo <[email protected]> * Fix Chainspec for Taiko-Hoodi (#9699) fix transition for taiko-hoodi * Xdc : Add XdcTestBlockchain for Module testing (#9596) * start * more dependencies * build * rename * voting * vote task * bit of refactor * invoke block produced dont seal * refactor context * format * timeout missing * block producer * format * basic incomplete setup * test * block production test * format * merge cleanup * merged * use the constant * use timestamp from attributes * Merged * Merged fix * cleanup * fixes * cleanup * commit cert before vote propose in background * merge and refactor * rename * format * comment * refactor to continuous loop * remove TCM * Format * fix * draft implmentation * cleanup * handle regenesis block in XdcTestBlockchain * refactor to inherit from TestBlockchain * register FromXdcContainer * handle _fromContainer from base * attempt rotating signers * test fix * test fix * test and container setup fix * decoder fix * bug fixes * init QC * test xdc sealer rotation * bug fixes and working now * Format * Update packages.lock.json * remove * total difficulty fix * format * override in blocktree * fix * unittest * format * fix build issues * fix * Fix * remove test * fixes * format * cleanup * undo merge and cleanup usings * add custom genesis builder for xdc * format whitespaces * Do not insert block if invalid fork * format * reorg test * revert block processor * revert cloning of header in PrepareBlockForProcessing * reorgs tests working * fix bad db * fix seal validator * fix * format * implement add block without commiting qc and generate extra keys for testing * fix and refactor * optional params * fix * one fix and a bit of refactor * reorg test * rename * return block instead * reorg test * fixes * dont set author * 10 seconds per add block * test blockchain changes * format * fixes for hotstuff module * fix test block producer * fixes for epoch master calculation * format * rename * rollback QC test blockchain order * snapshotmanager tests fixed * format * cleanup * remove TestBlockchainTester * cleanup * small fix in header validator test --------- Co-authored-by: ak88 <[email protected]> Co-authored-by: cicr99 <[email protected]> * XDC - Snapshot Manager refactor (#9695) refactor of snapshot manager * Xdc : Port BlockInfoValidator Module Tests (#9650) * start * more dependencies * build * rename * voting * vote task * bit of refactor * invoke block produced dont seal * refactor context * format * timeout missing * block producer * format * basic incomplete setup * test * block production test * format * merge cleanup * merged * use the constant * use timestamp from attributes * Merged * Merged fix * cleanup * fixes * cleanup * commit cert before vote propose in background * merge and refactor * rename * format * comment * refactor to continuous loop * remove TCM * Format * fix * draft implmentation * cleanup * handle regenesis block in XdcTestBlockchain * refactor to inherit from TestBlockchain * register FromXdcContainer * handle _fromContainer from base * attempt rotating signers * test fix * test fix * test and container setup fix * decoder fix * bug fixes * init QC * test xdc sealer rotation * bug fixes and working now * Format * Update packages.lock.json * remove * draft impl of blockinfo validator tests * total difficulty fix * format * override in blocktree * fix * unittest * format * fix build issues * fix * Fix * remove test * fixes * format * cleanup * undo merge and cleanup usings * add custom genesis builder for xdc * format whitespaces * Do not insert block if invalid fork * format * reorg test * revert block processor * revert cloning of header in PrepareBlockForProcessing * reorgs tests working * fix bad db * fix seal validator * fix * format * implement add block without commiting qc and generate extra keys for testing * fix and refactor * optional params * fix * one fix and a bit of refactor * reorg test * rename * return block instead * reorg test * fixes * fix failing test * dont set author * 10 seconds per add block * test blockchain changes * format * ws fixes * fixes for hotstuff module * fix test block producer * fixes for epoch master calculation * format * rename * rollback QC test blockchain order * CLEANUP * remove unused --------- Co-authored-by: ak88 <[email protected]> Co-authored-by: cicr99 <[email protected]> * Unit tests for commit in XDC QC manager (#9700) unittests * Xdc : Port HeaderValidator Module Tests (#9617) * start * more dependencies * build * rename * voting * vote task * bit of refactor * invoke block produced dont seal * refactor context * format * timeout missing * block producer * format * basic incomplete setup * test * block production test * format * merge cleanup * merged * use the constant * use timestamp from attributes * Merged * Merged fix * cleanup * fixes * cleanup * commit cert before vote propose in background * merge and refactor * rename * format * comment * refactor to continuous loop * remove TCM * Format * fix * draft implmentation * cleanup * handle regenesis block in XdcTestBlockchain * refactor to inherit from TestBlockchain * register FromXdcContainer * handle _fromContainer from base * attempt rotating signers * test fix * test fix * test and container setup fix * draft port of some HeaderValidatorTests * fix build issue * extra tests * decoder fix * handle merge issues * bug fixes * init QC * test xdc sealer rotation * bug fixes and working now * Format * Update packages.lock.json * remove * fix build issues * more build fixes * total difficulty fix * format * override in blocktree * fix * unittest * format * fix build issues * fix * Fix * remove test * fixes * format * cleanup * undo merge and cleanup usings * add custom genesis builder for xdc * format whitespaces * Do not insert block if invalid fork * format * reorg test * revert block processor * revert cloning of header in PrepareBlockForProcessing * reorgs tests working * fix bad db * fix seal validator * fix * format * implement add block without commiting qc and generate extra keys for testing * fix and refactor * optional params * fix * one fix and a bit of refactor * reorg test * rename * return block instead * minor changes and fixes * reorg test * fixes * bring back cache field in sealValidator * dont set author * 10 seconds per add block * test blockchain changes * format * ws fixes * fixes for hotstuff module * fix test block producer * fixes for epoch master calculation * format * rename * rollback QC test blockchain order * cleanup --------- Co-authored-by: ak88 <[email protected]> Co-authored-by: cicr99 <[email protected]> * Update base-sepolia testnet matrix (#9705) chore: update base-sepolia testnet matrix * Comment out Linea in workflows (#9706) chore: comment linea on our workflows --------- Co-authored-by: Alexey Osipov <[email protected]> Co-authored-by: lukasz.rozmej <[email protected]> Co-authored-by: Jaye Turner <[email protected]> Co-authored-by: ak88 <[email protected]> Co-authored-by: Snezhkko <[email protected]> Co-authored-by: radik878 <[email protected]> Co-authored-by: Ruben Buniatyan <[email protected]> Co-authored-by: Marcos Antonio Maceo <[email protected]> Co-authored-by: Damian Orzechowski <[email protected]> Co-authored-by: core-repository-dispatch-app[bot] <173070810+core-repository-dispatch-app[bot]@users.noreply.github.com> Co-authored-by: emlautarom1 <[email protected]> Co-authored-by: Lautaro Emanuel <[email protected]> Co-authored-by: Ahmad Bitar <[email protected]> Co-authored-by: maradini77 <[email protected]> Co-authored-by: GarmashAlex <[email protected]> Co-authored-by: Ben {chmark} Adams <[email protected]> Co-authored-by: Diptanshu Kakwani <[email protected]> Co-authored-by: Ayman Bouchareb <[email protected]> Co-authored-by: ak88 <[email protected]> Co-authored-by: cicr99 <[email protected]> Co-authored-by: Carmen Irene Cabrera Rodríguez <[email protected]>
Changes
Types of changes
What types of changes does your code introduce?
Testing
Requires testing