Log index (preparation)

[alexb5dh]

Helper methods/features/fixes needed for #8464 and extracted from that PR.

Changes

• RocksDB:

  • Support for merging (via IMergeOperator, MergeOperatorAdapter, and IMergeableKeyValueStore)
  • Support for clearing/cancelling RocksDB batch (via IWriteBatch.Clear)
  • Expose RocksDB iterator API (via IIterator and RocksDbIteratorWrapper)
  • StartBefore method for ISortedView - to try start just before the specified key
  • Fix for ColumnsDb flushing whole DB instead of a related column family
  • Fix for rare case of NullReferenceException when clearing iterators

• New IReceiptStorage.ReceiptsInserted event, which will be invoked for any receipts insertion. Old ReceiptsInserted, which was invoked only for the new main blocks, renamed to NewCanonicalReceipts.

• Fix for ProgressLogger cancelling completion no LogProgress invocation

• Support for byte array as a JSON property in RPC (serialized as hex string)

• Some testing helper classes/extensions

Types of changes

What types of changes does your code introduce?

• Bugfix (a non-breaking change that fixes an issue)
• New feature (a non-breaking change that adds functionality)

Testing

Requires testing

• No

If yes, did you write tests?

• No

Documentation

Requires documentation update

• No

Requires explanation in Release Notes

• No

[asdacap]

Is this necessary? Can you modify ISortedKeyValueStore instead?

[LukaszRozmej]

I think we didn't have that interface when this was written, so it might be good to reevaluate and it would be also mostly mod of ISortedView

[alexb5dh]

Replaced iterator with ISortedView, but needed to add StartBefore method doing iterator.SeekForPrev.

Also, upper bound (lastKey) being exclusive should probably be reflected in parameter naming or method documentation somehow:

ISortedView GetViewBetween(ReadOnlySpan<byte> firstKey, ReadOnlySpan<byte> lastKey)

[LukaszRozmej]

Good but needs a bit of polish

[LukaszRozmej]

I think we didn't have that interface when this was written, so it might be good to reevaluate and it would be also mostly mod of ISortedView

[LukaszRozmej]

I think we can support it here and do simple pass-through?

[alexb5dh]

Don't know if direct pass-through will make "compressed" parts valid.
It probably depends on how storage will be used (i.e. will we merge to empty key) and which merge operator is attached.

So would prefer to leave implementation in the hands of whoever will need Merge here.

[asdacap]

I'd prefer to not have it be implemented in the first place.

[alexb5dh]

StartWriteBatch returns IWriteBatch, so the only option I see is to move interface inheritance to specific classes and convert result to IMergeableKeyValueStore (or throw an error otherwise). This is how I'm using ISortedKeyValueStore as of now.

[alexb5dh]

If approach like this is fine, will move IMergeableKeyValueStore interface inheritance to concrete instances:

IDb db = GetDb(topicIndex);
ISortedKeyValueStore? sortedDb = db as ISortedKeyValueStore
  ?? throw new NotSupportedException($"{db.GetType().Name} DB does not support sorted lookups.");

[Copilot]

Pull Request Overview

This PR introduces infrastructure support for RocksDB merge operations and refactors the IReceiptStorage event model. The changes prepare the codebase for log indexing functionality by adding merge operator capabilities to the database layer and improving receipt insertion event granularity.

Key Changes:

• Added RocksDB merge operation support through new interfaces (IMergeOperator, IMergeableKeyValueStore) and adapters
• Split IReceiptStorage.ReceiptsInserted event into NewCanonicalReceipts (canonical chain only) and ReceiptsInserted (all blocks) for better event granularity
• Fixed critical bugs in ColumnsDb flush behavior and iterator disposal

Reviewed Changes

Copilot reviewed 33 out of 33 changed files in this pull request and generated 5 comments.

Show a summary per file

File	Description
Nethermind.Db/IMergeOperator.cs	New interface defining RocksDB merge operator contract
Nethermind.Db/RocksDbMergeEnumerator.cs	New ref struct for enumerating merge operation values
Nethermind.Db.Rocks/MergeOperatorAdapter.cs	Adapter bridging managed IMergeOperator to native RocksDB merge operator
Nethermind.Db.Rocks/DbOnTheRocks.cs	Added merge operation support, fixed iterator disposal, column family flush
Nethermind.Core/IWriteBatch.cs	Extended with Clear() and merge support via IMergeableKeyValueStore
Nethermind.Core/IKeyValueStore.cs	New IMergeableKeyValueStore interface and StartBefore method for sorted views
Nethermind.Blockchain/Receipts/*.cs	Refactored event model splitting canonical and all-block receipt insertions
Nethermind.Core/ProgressLogger.cs	Fixed completion state cancellation on progress logging
Nethermind.Core/JsonConverters/ByteArrayConverter.cs	Added support for byte arrays as JSON property keys
Multiple test files	Updated to use renamed NewCanonicalReceipts event

[asdacap]

GetViewBetween does not work? Or you need the key exactly before value?

[alexb5dh]

Yes, need to start exactly before so lower bound may not work as a starting point. Still prefer to have lower bound present - for example, for correct check on iterator validity. Plus, if key before is out of bounds - need to start at the lower bound.

[asdacap]

Consider ways to not have to implement Merge on place where that is unnecessary.