Prysm, the Ethereum consensus client, revealed the impact of the incident that occurred on December 4th in a post-mortem (post-event). This happened the day after the Fusaka update was rolled out. One of these consequences was the loss of 382 ETH from rewards to Ethereum validators.
According to the Prysm report, “Nearly all nodes (on the client, but not on the broader Ethereum network) experienced resource exhaustion events when attempting to process a particular certificate. Prysm was unable to respond to the validator’s request in a timely mannerresulting in blocks and certificates being omitted.
Prism incident facts
Incidents regarding Ethereum clients continued from that period (epoch) 411439 to 411480, adding a total of 42 periods.
The same report shows that A total of 248 blocks were lost in the node registry. This client was unable to synchronize with the rest of the nodes on the general network.
The specific problem was that Prysm’s attestation validation logic was computationally intensive (reprocessing blocks and transitions from previous epochs). This caused resource depletion. Under these conditions, many Prysm nodes would lag or crash, losing synchronization with the rest of the network.
However, this did not affect all Ethereum nodes. Other consensus clients continued to operate, keeping the network up and running.
Data shows that network participation during this event was as low as 75%. The validator lost approximately 382 ETH in verification rewards due to its inability to operate properly during the resource exhaustion event.
Bugs identified on GitHub
Prysm provides a more complete technical explanation of what happened to Ethereum validators. reveal it This bug was found in repository pull request 15965.
A Prysm beacon node received a certificate from a node that may be out of sync with the network. These certificates referenced the roots of blocks from previous eras. To verify them, Prysm tried to recreate a beacon state that was compatible with the view of the chain of asynchronous nodes. This resulted in multiple processing of blocks from previous epochs and costly recomputation of epoch transitions.
Security analyst at Prysm, an Ethereum consensus client.
Ethereum was saved by customer diversity
He also revealed that Diversity of Ethereum Validation Clients Reduced Impact of Incidents: «Customer diversity prevented a noticeable impact on Ethereum users. If clients occupy more than a third of the network, finality is temporarily lost and more blocks can be lost. If the client has errors and occurs more than two-thirds of the time, an invalid chain can be terminated,” they said from Prysm.
However, they reveal that another authentication client, Lighthouse, may account for more than 56% of the network, which is dangerously close to the threshold. A client error can result in an invalid chain being terminated (finally processed on the blockchain).
According to the incident documentation, client versions v7.0.1 and v7.1.0 include a long-term fix for the bug that caused the loss of validators’ rewards.