🐼🌿 We're a bamboo-munching, crypto-loving team of pandas who've traded the forest for the Ethereum blockchain! We were born during #TestingTheMerge
- Tweets 219
- Following 8
- Followers 2,488
- Likes 126
🚨🚨 Pectra is scheduled for Mainnet activation on May 7 @ 10:05:11 UTC!
We’ve launched a new website just in time! Featuring your go-to hub with a personalized fork checklist and key moments to watch for as the upgrade unfolds.
Link below👇👇
14
44
182
18,621
Help make the dataset more robust!
More information here: ethpandaops.io/posts/contrib…
Vape cabal? $1.5B? Libra? Trench warfare?
Brother what are you talking about? I'm busy watching the Ethereum beacon chain in the @ethPandaOps lab
4
11
90
12,244
Electra was successfully activated on Hoodi last week. Our latest analysis dives in to EIP7549 and compares attestion packing performance before and after the fork.
Link below!👇👇👇
3
17
88
5,852
Status update on Pectra testnets:
- If your testing doesn't need a validator: Use Sepolia (e.g: EIP-7702 wallets)
- If your testing needs a validator: See the rest of the thread
- If you don't rely on any external infra, use devnet-6: pectra-devnet-6.ethpandaops.…
Thread: (1/5) 🧵
6
21
77
30,452
Its just a chill testnet with over a million validators
The new testnet Hoodi is live! The testnet is aimed at being a stable location for staking operators and infra providers to test on.
Holesky Validators and Staking providers (deprecated)
Sepolia: App and tool dev
Hoodi: Validators and Staking providers
blog.ethereum.org/2025/03/18…
2
11
75
3,915
Status update:
- Holesky is still stable and finalizing
- There are some epochs with lower participation as people migrate their hacky setups back to stable ones
- Holesky can be used for pectra validator testing again
- Solo stakers, please use Ephemery: ephemery.dev
Holesky was non-finalizing for almost two weeks, this meant the nodes had a lot of states to keep around. Once finality happened, a lot of the states need pruning. Some changes made to the hacky branches seem to take client offline.
They're slowly coming back online now!
4
11
69
6,262
Hoodi is a new testnet we plan on launching 17th March. The network will have 1M validators and be the new location for validator related testing. We aim to keep the network as stable as possible, in order for staking providers to have a stable staging env
github.com/eth-clients/hoodi
ACDE Recap
A new testnet, Hoodi, is going live Monday to wrap up Pectra testing, if you need to test validator exits, be on the lookout for it! Everything else can be tested on Sepolia & Holesky.
Pectra will be scheduled 30+ days after Hoodi forks successfully, pending infra and client testing.
Fusaka planning will run in parallel, with a deadline of March 24 to propose EIPs, and a tentative date of April 10 for a scope freeze 🫡
Full recap on @EthMagicians, link below 👇
1
13
67
7,090
🚨 Announcement! We're opening up our Xatu data pipeline to the community!
If you're running a beacon node this is your opportunity to help accelerate Ethereum development.
Read more: ethpandaops.io/posts/contrib…
3
23
62
10,320
We have just finished testing dencun features on a mainnet shadowfork. We have seen blob inclusion from each EL/CL pair as well as blobs appearing in blocks built by MEV relayers.
1
10
58
3,121
🐼Announcement time!
We're making our Xatu data open source and publicly available! 🚀🚀🚀
Our goal is to enable researchers, developers, and enthusiasts to drive Ethereum's continued growth and success.
Read more: ethpandaops.io/posts/open-so…
3
11
59
15,144
Holesky was non-finalizing for almost two weeks, this meant the nodes had a lot of states to keep around. Once finality happened, a lot of the states need pruning. Some changes made to the hacky branches seem to take client offline.
They're slowly coming back online now!
1
7
61
14,034
The hoodi testnet finalized pectra!
The initial testing looks great, however the full suite of validator tests will take atleast a day to complete.
Tune in to ACD to hear more! :D
1
14
59
4,448
Announcing Tracoor!
Primarily created as a tool for incidents on the Ethereum network, Tracoor enables developers to concentrate on resolving issues as quickly as possible.
Read more: ethpandaops.io/posts/tracoor…
5
16
58
15,421
If you are running a holesky validator and struggling to get your beacon node up and running feel free to use our public beacon api endpoint hosted at:
beacon.holesky.ethpandaops.i…
We got some big chungus machines behind this load balancer!
Let’s get holesky finalized again!
2
22
44
16,346
Our newest blog post introduces Assertoor, a new end-to-end, cross-client integration testing tool that operates at a higher abstraction level than Hive.
Check out how it works and how we're using it to do automated testing on full scale ethereum testnets:
ethpandaops.io/posts/asserto…
2
11
42
14,603
Have you ever wondered how to easily build on top of a kurtosis package?
We built an L2 devnet on kurtosis just to showcase that functionality! You can find a deeper dive on the topic here: ethpandaops.io/posts/kurtosi…
1
1
40
14,423
If you've got a devcon ticket - You've got devnet funds. Claim them and be the first to build a EIP-7702 enabled smart wallet or try out a range of other changes coming to Ethereum soon! :D
The Mekong Testnet was just announced - and thanks to @_pk910_ you can already claim funds on it with your @EFDevcon ticket via @ZupassProject here:
devcon-mekong-faucet.pk910.d…
You can also get funds on sepolia and holesky with the ticket.
blog.ethereum.org/2024/11/07…
3
5
36
8,928
EthPandaOps pivots to new challenges. We think GitChain is the way forward. To learn more check out our latest blog post below.
ethpandaops.io/posts/april-f…
3
5
39
3,031
Over the past few months, the ethPandaOps team has been more active in maintaining the Kurtosis ethereum-package. We're excited to share that we will be taking full ownership of the package on 13th June by moving it to our Github org
This is a breaking change so read below!
1/3
We're excited to announce that Kurtosis is now fully open-source under the Apache 2.0 license! 🦢✨
(1/4)
1
7
28
3,756
Bit late to the game, but here's our Pectra retrospective: ethereum-magicians.org/t/pec…
1
8
36
3,067
Are you ready for devcon week?
Don't forget to try the Mekong testnet :D
mekong.ethpandaops.io/
2
7
30
9,201
The Ethereum Dencun fork went live successfully earlier this week on Mainnet! We’ve been using our monitoring tool called Xatu to collect data on network health, find the first (of many) full blogposts here: ethpandaops.io/posts/dencun-…
🧵
We'd like to introduce our new website ethpandaops.io - your gateway to all things EthPandaOps and a way to keep up to date with projects we’re developing! 🌿
1
10
35
6,709
Pectra lands on Mainnet in a few hours! Wondering where to watch the fork unfold? Well we've got a new section in The Lab to monitor block production that might do the job 😏😏😏
Just in time for Pectra, we've been cooking up a live view of Ethereum block production over in The Lab.
Watch the lifecycle of a slot unfold as blocks travel through relays (or are built locally 👑), land on nodes and then receive attestations 👀
👇👇👇
1
4
32
1,856
We'd like to introduce our new website ethpandaops.io - your gateway to all things EthPandaOps and a way to keep up to date with projects we’re developing! 🌿
1
9
30
19,210
🚨 3 new Xatu tables just dropped! 🚨
Fresh from the Consensus Layer P2P network, these tables give a rare view of Ethereum's inner workings. We can't wait to see the insights the community will uncover.
Read more: ethpandaops.io/posts/xatu-co…
2
14
27
7,605
Attacknet, a tool we collaborated with @trailofbits on is out! It gives us the ability to perform chaos engineering on local devnets, allowing us to simulate edge cases before they happen on mainnet.
Find more info here: ethpandaops.io/posts/attackn…
Today we're releasing Attacknet, a new tool in the blockchain security arsenal. Built in collaboration with the @Ethereum Foundation, it uses Chaos Engineering to test the most challenging network conditions imaginable for fault tolerance blog.trailofbits.com/2024/03…
2
7
28
11,767
Just when you thought you already had your blog post for the week from the Pandas🐼here is another one!
This post is a deep dive to Kurtosis and how local devnets can make your life easier too.
ethpandaops.io/posts/kurtosi…
@KurtosisTech
2
7
26
8,871
We also have a new website design with a focus on the tooling we develop as well using them. Over the next weeks, we plan on fleshing out our Node operator, Client dev and Data analyst guides - so you have everything you need when you start digging into Ethereum.
🚨🚨 Pectra is scheduled for Mainnet activation on May 7 @ 10:05:11 UTC!
We’ve launched a new website just in time! Featuring your go-to hub with a personalized fork checklist and key moments to watch for as the upgrade unfolds.
Link below👇👇
2
1
24
2,537
📢📢 New data time!
Using @notnotstorm's excellent Cryo tool we've started publishing 16(!) new canonical execution tables to our public parquet files.
Learn more: ethpandaops.io/posts/xatu-ex…
1
6
22
2,471
Latest post: a retrospective on EIP7691. We take a look back to see how 6/9 blobs is performing, and then glance forward to a potential 60M gas limit increase.
Will @BarnabasBusa's RFC1149 based internet connection fly under the radar?👀
Some light reading for your Friday! Up today is a retrospective on EIP7691, which increased blob throughput. Did it perform as we expected? And what does this tell us about a potential 60M gas limit?
Read on 👇
1
5
33
12,740
TLDR:
- EIP-7702 or any EL related testing: Use Sepolia
- Validator testing: Wait for Holesky finality (till 12th March) or use Holesky shadowfork once announced (after 12th March)
- MEV relays: Use Sepolia (reach out to get validator traffic)
(5/5)
1
3
15
2,627
1,006,660,619,135 🎉
We've hit 4 commas of Ethereum attestations in our @ClickHouseDB cluster!
Some stats:
- 3 networks
- 1 trillion rows
- 14TB compressed
- 500TB uncompressed 😮
Want to dig in to the data? We publish it all here:
ethpandaops.io/data/xatu/
3
14
3,274
As L2s start using Blobs on testnets, they might need a CL API loadbalancer. We've developed and have been running Dugtrio on our fleet and are quite happy with it!
github.com/ethpandaops/dugtr…
Take it for a spin and let us know if you find missing features!
1
3
15
1,650
The kind of tools we're building to help make decisions more confidently on ethereum :D
Comparing a node subscribed to all attnets (left) vs a node subscribed to the default amount (right). There's a lot more aggregate traffic since Dencun
All p2p traffic:
- w/ 2 attnets: 750kB/s
- w/ 64 attnets: 2.5MB/s total
Attestations range from 60% to 90% of the node traffic
1
15
633
Check out the cool stats collected from Xatu!
notes.ethereum.org/@ethpanda…
If you love data, have a look at the EF Data visualization challenge running now: esp.ethereum.foundation/data…
Help us make sense of the upgrade :D
Blobs are live on Ethereum Mainnet!
The Dencun fork finalized without issues and the network has been running stably since then. We can see healthy blob propagation statistics (expect a deeper analysis over the next weeks).
Congrats to all the client teams and contributors!
6
12
1,762
We are currently trying to bring finality back to Holesky, If you are unable to sync a node then please point your validators to the beacon RPC here: holesky-rescue.ethpandaops.i…
If you need help finding peers, use this: notes.ethereum.org/@ethpanda…
(3/5)
1
10
613
TLDR:
Hoodi: Stable validator testing env
Sepolia: Stable app/smart contract dev testing env
Holesky: Supported for now, yet to decide longer term
Note, we may move to a 3-4y lifecycle for testnets in order to ensure we can repurpose networks based on changing network needs!
2
10
706
If we are unable to gain finality on Holesky by 12th March, We will be spinning up a Holesky shadowfork that will last until 31st March. All validator related testing can then move to this shadowfork, more information will be shared once the network is up.
(4/5)
1
9
745
Shadowforks are useful in order to get realistic network workloads gossiped from mainnet. We combined those transactions with blob transactions, and still saw very few missed slots. Super solid performance from each client team. Successful test.
9
296
Holesky was the intended location for validator testing. The bugs in the network live, but non-finalizing. This implies validators actions cannot be performed (Even if you can continue using the chain).
(2/5)
1
7
620
Our first eg is called optimism-package, and deploys the optimism L2 on top of an Ethereum L1.
You can find the package here: github.com/ethpandaops/optim…
This experiment also taught us a lot about how L2s work in general and we're hoping its not the last L2 package in kurtosis
1
4
375
So experiment with kurtosis packages as importable modules and tell us what you think!
Hint: It should be easy to build pre-confirmations infra on top :D
5
277
We go into the concept of importing kurtosis packages in this blogpost, showing how easy it is to extend the ethereum-package for your L2/tooling needs. You can always expect the L1 to just exist by calling a function, reducing maintenance cost for your testing and developing!
1
4
302
We also have a checklist ready for node operators, and a list of things our team will be looking out for at fork time (and how to follow along!)
ethpandaops.io/posts/pectra-…
3
370
If your validator or node was affected, then clear the CL datadir and checkpoint sync from the last finalized state: checkpoint-sync.holesky.ethp…
The node should be back up in no time!
holesky-rescue.ethpandaops.i… will continue to work if you need a backup.
2
2
4
601
While this stability is great news for now, we're still coming up with a plan on where/how to allow people to test effectively without being delayed by large queues. More information will be shared the next days!
3
512
Some questions will be answered as Dencun matures over time:
- Is there a higher reorg rate from the increased bandwidth❓
- Is there an impact to attestation effectiveness❓
- Does it impact geographically distributed validators more❓
- Can we safely bump the blob parameters❓
1
4
193
You can find more about our architecture and the kind of data we run here: notes.ethereum.org/@ethpanda…
Reach out if you're a researcher and the data is interesting to you :)
1
4
239
If you'd like to contribute your data (thank you!!), please fill out the form here: forms.gle/S7g5g8nB8aGG8aTX6
Access is currently limited as we scale up our data pipeline!
1
4
870
Huge props to our pandas @samcmAU and @mattevansnz for the lift <3
4
310
If you ever had a question about what tools we use to observe, test, validate or deploy - you will find the answer on our website. If you have ideas for topics you want us to cover, send them across :)
1
3
349
The dataset contains a wealth of information about the Ethereum network, including detailed data on beacon chain events, mempool activity, and canonical chain events.
Check out all the available data here: ethpandaops.io/data/xatu
1
2
3
839
Privacy is our priority. Contributooors are able to choose how much metadata they provide. Additionally, certain metadata fields are obfuscated such that an event cannot be mapped back to an individual node on the network.
1
3
408
We are currently running instances for Mainnet, Sepolia & Holesky, but long term we hope to foster a set of community-run instances similar to checkpoint sync endpoints.
If you'd like to run an instance, check out the README to get started: github.com/ethpandaops/traco…
2
386
We'll share more detailed next steps once the network is fully stable again! Stay tuned for now!
3
398
The tool was used during the testing phase for Dencun, largely to ensure that the network would mostly be stable even if blobs were delayed. We were also able to reproduce the Prysm mainnet incident from May 2023 using this tool.
1
2
97
oo we haven't checked this in a while, it could be nimbus-nethermind? Also the Java clients allow you to tune memory used. Maybe @EthereumOnARM has a more educated answer?
1
2
113
Fuelled by Ethereum's ethos of openness and collaboration, we're thrilled to open up this dataset to everyone.
We can't wait to see your creations and insights!
With love,
EthPandaOps ❤️
2
331
Sepolia continues to be the stable location for app and smart contract devs. It has a small validator size, faucets, rpc support and everything needed for app devs.
1
2
346
The original issues that happened here can't happen on mainnet. However we will have a retrospective next week on other fixes.
We will be doing mainnet shadow forks to catch any other bugs
1
2
45
This is not a checkpoint sync node its a node you can point your validator to, you can pass --beacon-rest-api-provider
20
Events like when the node saw a block, or when a chain reorg happens, are crucial for core devs when choosing protocol parameters. We intend to make this process as easy as possible for all involved!
1
2
171
Increasing the blob count has been a hot topic recently (especially with home stakers) so we're opening up our data collection pipeline.
This will let community members contribute data for research purposes that was previously impossible to collect and analyze.
1
2
233
The 50th percentile of attestation arrivals remains the same as before the fork. This is a good indicator for a healthy upgrade.
1
1
131
Note: Testnet results are not 1:1 mappings to mainnet. They do a good job of showing us how changes perform compared to their past state of course.
1
346
If you're on the lookout for an EL loadbalancer, try out DShackle: github.com/emeraldpay/dshack…
1
135
Please update your URLs to target new releases once the migration has been made. (github.com/ethpandaops/ether…) Alternatively, you can clone the last release (3.1.) in the current organization, and continue using the package in its frozen state.
2/3
1
1
196
We're excited to take continue building on this package to enable easy, fully featured local devnets!
3/3
1
167
The exit queue of Holeksy is filled with slashed and validators < ejection balance. This amounts to almost 1M validators, The remaining ~700k will stay online and their balances should increase as they perform duties.
1
1
592
Holesky will continue to be supported in the short term. As the exit queue is filled for a long duration, it has limited usefulness for pectra community testing. We may repurpose it as a core-dev testnet, using it to test non-finality and gas limit increases, more info to come!
1
1
476
CPU/Memory usage increase was marginal, the network usage also went up (but still under the 5Mbps range)
1
1
244
TLDR:
- Successful fork ✅
- Blobs arrive within an acceptable time ✅
- Slight delay in attestation arrivals due to the addition of blobs ✅
- No massively noticeable increase in block arrival times ✅
1
1
130
It appears that block propagation is ~10% worse when a slot has 6 blobs instead of 1 blob. This is still well under the 4s mark, but we are clearly getting closer to the attestation deadline and more analysis might be needed.
1
1
135
Most blobs seem to arrive at the ~2.5s mark, with the p95 values on our sentries getting close to ~4s. The p95 value is concerning as 4s is the attestation deadline, but we might need to collect data over a longer duration.
1
1
167
Reach out to @trailofbits or @thebensams or any of the ethpandaops folks to talk about the tool!
And thanks to the team at @KurtosisTech for all of their support in making this happen
1
140
Keep up to date with our analyses on our new website, ethpandaops.io . If you’re a researcher interested in the Data Challenge for EIP4844, please reach out to ethpandaops(at)ethereum.org for access to the data!
More details here: esp.ethereum.foundation/data…
1
1
172
It appears that the time difference from the first blob seen to the last blob seen in a slot is quite narrow. This is quite good news as a higher latency would indicate some routing issues on the p2p layer.
1
1
153
The best part: If you're application is built using kurtosis and the ethereum-package, you'd be able to inherit the ability to perform chaos engineering on your package too!
The tool is built to be modular, so help us add support for your favorite L2 or application :D
1
1
168
