Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update sync pipelines to continue from era1 import #8403

Open
wants to merge 35 commits into
base: main
Choose a base branch
from
+34 −19
Open

Update sync pipelines to continue from era1 import #8403

Changes from all commits
Commits
Show all changes
35 commits
Select commit Hold shift + click to select a range
c777ae6
Refactor to use clock to validate expiry
Matilda-Clerke Jan 30, 2025
d27a5a6
7935: Finish unit testing E2StoreReader
Matilda-Clerke Feb 12, 2025
b7a359d
Build out E2StoreReader and associated files
Matilda-Clerke Feb 19, 2025
da718f0
Spotless
Matilda-Clerke Feb 19, 2025
c19aa23
Add javadoc to new util module classes
Matilda-Clerke Feb 20, 2025
2d99f07
Merge branch 'main' into 7935-add-e2storereader
Matilda-Clerke Feb 21, 2025
8b67d5f
Implement Era1 block import sub command
Matilda-Clerke Feb 25, 2025
bb1c713
Remove unnecessary e2 and era file stuff
Matilda-Clerke Feb 26, 2025
f2060aa
Rename files to be era1 specific
Matilda-Clerke Feb 26, 2025
079b40c
Rename e2 package to era1
Matilda-Clerke Feb 26, 2025
816e45a
Spotless
Matilda-Clerke Feb 26, 2025
dd16c26
Merge branch 'refs/heads/7935-add-e2storereader' into 7935-add-era1-f…
Matilda-Clerke Feb 26, 2025
6dea2f3
Rename Era1StoreReaderListener to Era1ReaderListener
Matilda-Clerke Feb 26, 2025
b9564d5
Merge branch 'refs/heads/7935-add-e2storereader' into 7935-add-era1-f…
Matilda-Clerke Feb 26, 2025
37c257e
Clean up after merge
Matilda-Clerke Feb 26, 2025
26fd7b5
Set up Era1BlockImporter to enable FAST sync and variations upon besu…
Matilda-Clerke Feb 27, 2025
3afd859
Use bouncycastle for little endian to long conversion
Matilda-Clerke Feb 27, 2025
abbe9af
Rename slot related variables to blockIndex
Matilda-Clerke Feb 27, 2025
f921fc0
Spotless
Matilda-Clerke Feb 27, 2025
a387b33
Merge branch 'refs/heads/7935-add-e2storereader' into 7935-add-era1-f…
Matilda-Clerke Mar 2, 2025
93afe39
Fix broken unit test
Matilda-Clerke Mar 3, 2025
11d4885
Fix AT compilation
Matilda-Clerke Mar 3, 2025
aa9a08b
Fix javadoc
Matilda-Clerke Mar 3, 2025
6e9043d
Merge branch 'main' into 7935-add-era1-format-to-blocks-importer
Matilda-Clerke Mar 4, 2025
0d2385c
Rework Era1BlockImporter to use appropriate data directory. Add unit …
Matilda-Clerke Mar 5, 2025
3730494
Fix javadoc
Matilda-Clerke Mar 5, 2025
dfd937d
changelog
Matilda-Clerke Mar 5, 2025
692ae2c
Add era1 files to .gitattributes as binary file type
Matilda-Clerke Mar 5, 2025
a3f926a
Merge branch 'main' into 7935-add-era1-format-to-blocks-importer
Matilda-Clerke Mar 6, 2025
1f3b8d7
Remove pivot block header file creation from Era1BlockImporter
Matilda-Clerke Mar 6, 2025
16f3351
Modify SnapDownloaderFactory to allow snap sync after importing era1 …
Matilda-Clerke Mar 11, 2025
db7be9e
Modify Snap, checkpoint, and fast sync downloader factories to allow …
Matilda-Clerke Mar 12, 2025
1f50a1d
Merge branch 'main' into 7935-update-sync-pipelines-to-continue-from-…
Matilda-Clerke Mar 12, 2025
b731b08
Check if world state root hash is still the genesis state root before…
Matilda-Clerke Mar 13, 2025
77da0e9
Fix broken unit tests
Matilda-Clerke Mar 13, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
11 changes: 6 additions & 5 deletions ...va/org/hyperledger/besu/ethereum/eth/sync/checkpointsync/CheckpointDownloaderFactory.java
View file
Original file line number Diff line number Diff line change
@@ -15,7 +15,6 @@
package org.hyperledger.besu.ethereum.eth.sync.checkpointsync;

import org.hyperledger.besu.ethereum.ProtocolContext;
import org.hyperledger.besu.ethereum.core.BlockHeader;
import org.hyperledger.besu.ethereum.eth.manager.EthContext;
import org.hyperledger.besu.ethereum.eth.sync.PivotBlockSelector;
import org.hyperledger.besu.ethereum.eth.sync.SyncMode;
@@ -92,11 +91,13 @@ public static Optional<FastSyncDownloader<?>> createCheckpointDownloader(
address ->
snapContext.addAccountToHealingList(
CompactEncoding.bytesToPath(address.addressHash())));
} else if (fastSyncState.getPivotBlockHeader().isEmpty()
&& protocolContext.getBlockchain().getChainHeadBlockNumber()
!= BlockHeader.GENESIS_BLOCK_NUMBER) {
} else if (!protocolContext
.getWorldStateArchive()
.getWorldState()
.rootHash()
.equals(GENESIS_STATE_ROOT)) {
LOG.info(
"Checkpoint sync was requested, but cannot be enabled because the local blockchain is not empty.");
"Checkpoint sync was requested, but cannot be enabled because the world state root hash exists.");
return Optional.empty();
}

14 changes: 9 additions & 5 deletions ...ava/org/hyperledger/besu/ethereum/eth/sync/fastsync/worldstate/FastDownloaderFactory.java
View file
Original file line number Diff line number Diff line change
@@ -14,8 +14,8 @@
*/
package org.hyperledger.besu.ethereum.eth.sync.fastsync.worldstate;

import org.hyperledger.besu.datatypes.Hash;
import org.hyperledger.besu.ethereum.ProtocolContext;
import org.hyperledger.besu.ethereum.core.BlockHeader;
import org.hyperledger.besu.ethereum.eth.manager.EthContext;
import org.hyperledger.besu.ethereum.eth.sync.PivotBlockSelector;
import org.hyperledger.besu.ethereum.eth.sync.SyncMode;
@@ -48,6 +48,8 @@
public class FastDownloaderFactory {

protected static final String FAST_SYNC_FOLDER = "fastsync";
protected static final Hash GENESIS_STATE_ROOT =
Hash.fromHexString("0xd7f8974fb5ac78d9ac099b9ad5018bedc2ce0a72dad1827a1709da30580f0544");
Comment on lines +51 to +52
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should not this depend on the actual network or genesis used, so the value can change, maybe @matkt has suggestion on how to check if the world state is empty

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good point. I'm not sure anyone really cares about the ability to import era1 files on sepolia, but it could be useful for testing. Afaik, we don't really know which network the era1 files we're importing are for, so I could add --network and --data-path to allow people to better specify what they're doing. The network option would allow us to just check a different genesis state root, but alternative ways of checking whether a sync is already completed might be preferable

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

to check if the initial sync is done you can use

besu/ethereum/eth/src/main/java/org/hyperledger/besu/ethereum/eth/sync/state/SyncState.java

Line 319 in b7a0b91

public boolean isInitialSyncPhaseDone() {

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yes clearly it's not good like that. So when we are passing this code the state is not empty and has the genesis modification already ?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

if it is the case why not using ?

Hash genesisStateRoot = protocolContext.getBlockchain().getGenesisBlock().getHeader().getStateRoot()


private static final Logger LOG = LoggerFactory.getLogger(FastDownloaderFactory.class);

@@ -83,11 +85,13 @@ public static Optional<FastSyncDownloader<?>> create(
fastSyncStateStorage.loadState(ScheduleBasedBlockHeaderFunctions.create(protocolSchedule));

if (!syncState.isResyncNeeded()
&& fastSyncState.getPivotBlockHeader().isEmpty()
&& protocolContext.getBlockchain().getChainHeadBlockNumber()
!= BlockHeader.GENESIS_BLOCK_NUMBER) {
&& !protocolContext
.getWorldStateArchive()
.getWorldState()
.rootHash()
.equals(GENESIS_STATE_ROOT)) {
LOG.info(
"Fast sync was requested, but cannot be enabled because the local blockchain is not empty.");
"Fast sync was requested, but cannot be enabled because the world state root hash exists.");
return Optional.empty();
}

11 changes: 6 additions & 5 deletions .../src/main/java/org/hyperledger/besu/ethereum/eth/sync/snapsync/SnapDownloaderFactory.java
View file
Original file line number Diff line number Diff line change
@@ -15,7 +15,6 @@
package org.hyperledger.besu.ethereum.eth.sync.snapsync;

import org.hyperledger.besu.ethereum.ProtocolContext;
import org.hyperledger.besu.ethereum.core.BlockHeader;
import org.hyperledger.besu.ethereum.eth.manager.EthContext;
import org.hyperledger.besu.ethereum.eth.sync.PivotBlockSelector;
import org.hyperledger.besu.ethereum.eth.sync.SyncMode;
@@ -87,11 +86,13 @@ public static Optional<FastSyncDownloader<?>> createSnapDownloader(
address ->
snapContext.addAccountToHealingList(
CompactEncoding.bytesToPath(address.addressHash())));
} else if (fastSyncState.getPivotBlockHeader().isEmpty()
&& protocolContext.getBlockchain().getChainHeadBlockNumber()
!= BlockHeader.GENESIS_BLOCK_NUMBER) {
} else if (!protocolContext
.getWorldStateArchive()
.getWorldState()
.rootHash()
.equals(GENESIS_STATE_ROOT)) {
LOG.info(
"Snap sync was requested, but cannot be enabled because the local blockchain is not empty.");
"Snap sync was requested, but cannot be enabled because the world state root hash exists.");
return Optional.empty();
}

17 changes: 13 additions & 4 deletions .../test/java/org/hyperledger/besu/ethereum/eth/sync/fastsync/FastDownloaderFactoryTest.java
View file
Original file line number Diff line number Diff line change
@@ -22,8 +22,10 @@
import static org.mockito.Mockito.verify;
import static org.mockito.Mockito.when;

import org.hyperledger.besu.datatypes.Hash;
import org.hyperledger.besu.ethereum.ProtocolContext;
import org.hyperledger.besu.ethereum.chain.MutableBlockchain;
import org.hyperledger.besu.ethereum.core.MutableWorldState;
import org.hyperledger.besu.ethereum.eth.manager.EthContext;
import org.hyperledger.besu.ethereum.eth.sync.PivotBlockSelector;
import org.hyperledger.besu.ethereum.eth.sync.SyncMode;
@@ -33,6 +35,7 @@
import org.hyperledger.besu.ethereum.mainnet.ProtocolSchedule;
import org.hyperledger.besu.ethereum.trie.diffbased.bonsai.storage.BonsaiWorldStateKeyValueStorage;
import org.hyperledger.besu.ethereum.trie.forest.storage.ForestWorldStateKeyValueStorage;
import org.hyperledger.besu.ethereum.worldstate.WorldStateArchive;
import org.hyperledger.besu.ethereum.worldstate.WorldStateKeyValueStorage;
import org.hyperledger.besu.ethereum.worldstate.WorldStateStorageCoordinator;
import org.hyperledger.besu.metrics.SyncDurationMetrics;
@@ -94,6 +97,15 @@ public void setup(final DataStorageFormat dataStorageFormat) {
}
when(worldStateKeyValueStorage.getDataStorageFormat()).thenReturn(dataStorageFormat);
worldStateStorageCoordinator = new WorldStateStorageCoordinator(worldStateKeyValueStorage);

WorldStateArchive worldStateArchive = mock(WorldStateArchive.class);
MutableWorldState mutableWorldState = mock(MutableWorldState.class);
when(protocolContext.getWorldStateArchive()).thenReturn(worldStateArchive);
when(worldStateArchive.getWorldState()).thenReturn(mutableWorldState);
when(mutableWorldState.rootHash())
.thenReturn(
Hash.fromHexString(
"0xd7f8974fb5ac78d9ac099b9ad5018bedc2ce0a72dad1827a1709da30580f0544"));
}

@ParameterizedTest
@@ -149,8 +161,7 @@ public void shouldNotThrowIfSyncModeChangedWhileFastSyncComplete(
@SuppressWarnings("unchecked")
@ParameterizedTest
@ArgumentsSource(FastDownloaderFactoryTestArguments.class)
public void shouldNotThrowWhenFastSyncModeRequested(final DataStorageFormat dataStorageFormat)
throws NoSuchFieldException {
public void shouldNotThrowWhenFastSyncModeRequested(final DataStorageFormat dataStorageFormat) {
setup(dataStorageFormat);
initDataDirectory(false);

@@ -171,8 +182,6 @@ public void shouldNotThrowWhenFastSyncModeRequested(final DataStorageFormat data
syncState,
clock,
SyncDurationMetrics.NO_OP_SYNC_DURATION_METRICS);

verify(mutableBlockchain).getChainHeadBlockNumber();
}

@ParameterizedTest