In that empty ecosystem of yours, there were only those who farmed airdrops.
You did PR by using them. Now you call them bots. Disgrace
On December 16, an inscription event on zkSync Era caused a large increase in transactions over a period of 38 hours. For nearly 14 hours straight, the network handled ~150 TPS — peaking at 187 TPS — with an average tx cost of ~$0.12.
Here are the details and our learnings ⬇️
Inscription events have recently stress-tested almost every chain, leading to outages of some networks, as well as massive increases in transaction fees. While zkSync Era performed well and showed resilience amidst increased transaction load, we learned a lot from the event.
Despite a new ATH in TPS and sustaining low gas fees, Saturday’s event uncovered three areas we'll immediately look to improve.
📍Database performance
One of the system’s primary databases was configured with fewer total connections than needed during prolonged ~150 TPS. This led to 15 minutes of downtime, which was fixed immediately once the team re-started the database. A monitoring error (now fixed) was responsible for us not identifying the exact problem immediately.
The system uses read-replica databases to distribute query load. In the midst of the event, we saw a replication lag on one of these replicas, which led to some APIs operating in a degraded state. Unfortunately this affected some of the external partners that were running RPC endpoints too. We’ve already adjusted our load distribution to better protect the database.
📍Transaction UX errors
Certain wallets showed that a tx failed when it actually succeeded; we uncovered that this was because of a difference in behaviour in how our API responded (under certain conditions) compared to the Ethereum JSON-RPC API. This issue is now fixed and deployed to production.
📍 Lagging Block Explorer
The block explorer couldn’t keep up at ~150 TPS and many users took this as a sign that the network was delayed, even though that wasn’t the case and transactions were continuing to go through - the explorer was just slow to index them. We immediately deployed a banner that warned users of extra traffic and advised users they could look up transactions with the transaction hash. We are expediting the work to improve our indexer’s performance.
Why are these events significant?
Ethereum and ZK technology continue to lay the foundation for a new Internet of Value. This weekend served as an important stress-test and a big milestone on the way of bringing Ethereum to the next billion people. We want to be transparent about the issues we saw, but there was also a huge amount that went well, with the system processing ~6M txs with an average tx cost of $0.12!