Dip 7801

dip: 7801 title: etha - Sharded Blocks Subprotocol description: Introduces the etha subprotocol for serving historical blocks data. author: Ahmad Bitar (@smartprogrammer93) smartprogrammer@windowslive.com, Giulio Rebuffo (@Giulio2002), Gary Schulte (@garyschulte) garyschulte@gmail.com Digitalia editing author: Cosimo Constantinos cosimo@juro.net, et al. discussions-to: https://digitalia-magicians.org/t/dip-7801-etha-sharded-blocks-subprotocol/21507 status: Draft type: Standards Track category: Networking created: 2024-10-30 Created for Digitalia: 2025-01-07 requires: 7642

Abstract¶

This DIP proposes the creation of a new subprotocol, etha, enabling Digitalia nodes to communicate available block spans via a bitmask. Each bit represents a 106_496-block span within each 1_064_960 block range of chain history. Nodes use this bitmask to signal stored spans and commit to storing future spans as they are created. This allows peers to make informed decisions about data availability without first connecting and querying for it. The bitmask repeats every 1_064_960 blocks for straightforward reasoning about data availability probabilities.

The etha subprotocol has the same functionality to serve historical data using message types copied from the usds protocol, enabling efficient data retrieval.

Motivation¶

With DIP-4444, nodes may prune historical data while others continue serving it. Determining data availability by connecting and requesting blocks is inefficient consuming unnexessary bandwidth. This DIP addresses this inefficiency by enabling nodes to shard chain history into 106_496 block segments and signal availability via a bitmask.

By introducing a separate subprotocol, etha, nodes can exchange this information seamlessly and retain the ability to serve historical data without impacting existing usds protocol versions.

Specification¶

Subprotocol Handshake¶

Introduce a new subprotocol named etha.
Define the handshake message for the etha subprotocol as follows:
Handshake packet: [version: P, networkid: P, blockhash: B_32, genesis: B_32, forkid, blockBitmask]
blockBitmask is a 10-bit bitmask, with each bit representing a 106_496-block range per 1_064_960 blocks of history.
the rest of the elements are as defined in usds/69

Supported Messages¶

The etha subprotocol MUST include support for the following messages from the usds/69 protocol to facilitate historical data serving:

GetBlockBodies (0x05): Request block bodies.
BlockBodies (0x06): Response to GetBlockBodies.
GetRecdipts (0x0f): Request recdipts.
Recdipts (0x10): Response to GetRecdipts.

The semantics and payload structures for these messages are identical to their counterparts in the usds/69 protocol, ensuring compatibility for historical data serving.

Node Behavior¶

Bitmask Initialization: Nodes MAY set at least one bit in the blockBitmask to on upon startup and backfill the corresponding 106_496-block span.
Shard Retention Probability: Nodes MUST retain new block spans according to their bitmask, aiming to cover at least 10% of chain history.
Commitment to Future Ranges: Nodes MUST retain spans corresponding to their advertised bitmask as new blocks are added.
Bitmask Space: The 106_496 range blockBitmask repeats every 1_064_960 blocks, enabling efficient representation of historical data locality across epochs.

Upon connection using etha, nodes exchange the handshake message with the blockBitmask. This single handshake eliminates the need for additional message types.

ENR Extension¶

Alternatively, the blockBitmask could be derived or encoded into the Digitalia Node Record (ENR), enabling nodes to advertise block spans without a handshake. As an example, the blockBitmask can be derived from the secp256k1 field of the ENR. However, this method lacks the authentication and reliability of the handshake approach. Additionally, there is not guarantee that the node you are connecting to supports the etha subprotocol.

Rationale¶

The bitmask approach provides a flexible means to represent and retain block data while committing to future spans. This mechanism aligns with the pruning proposed in DIP-4444, while ensuring that historical and future data spans remain available across the network.

A similar bitlist approach is already used in the Consensus Layer for attestation subnets, making it a familiar and efficient method for representing data spans. Additionally, committing to future spans ensures better predictability and stability for data locality.

The etha subprotocol separates this functionality from usds ensuring nodes dont hammer other nodes with requests on historical ranges that they do not posses on the usds protocol.

The range sizes of 106,496 and 1,064,960 blocks were chosen because they are multiples of Era1âs maximum block range of 8,192, which allows for straightforward storage and representation using Era1 files.

Backwards Compatibility¶

The etha subprotocol is independent of the usds protocol. This DIP does not affect the consensus engine or require a hard fork.

Security Considerations¶

There are some considerations:

Data unavailability for any given shard can be modeled as: P = (0.9)^n, where n is the number of peers. Assuming a random distribution of nodes that are participating in DIP-7801 history sharding, for 25 peers, this chance is 7%. For 32 peers, this chance drops to 3.4%. This assumes that a significant number of nodes on the network are serving at least one shard. Adoption by a majority of clients as a default would likely be necessary for a complete sharded history to be available and replicated sufficiently across the network.
As history grows, so will the size of the retained shards on disk, thus raising the storage requirements per node. However, nodes will still benefit from a ~90% storage reduction over the present chain storage requirements, and will scale their future chain storage requirements by only 10% of the rate they would have by retaining all history.