Block Finalization Stall
Incident Report for Avalanche
Resolved
This incident has been resolved.
Posted Feb 23, 2024 - 17:43 UTC
Update
A significant % of stake weight has upgraded to AvalancheGo@v1.11.1 and block finalization is stable. We will continue monitoring for further regressions.
Posted Feb 23, 2024 - 17:36 UTC
Monitoring
Block finalization on the Primary Network is beginning to resume. Stability will fluctuate as validators continue updating to AvalancheGo@v1.11.1.

You can monitor block finalization here: https://subnets.avax.network/c-chain
Posted Feb 23, 2024 - 16:36 UTC
Identified
🚨🚨 Attention Avalanche Validators 🚨🚨

Please upgrade your node to AvalancheGo to v1.11.1: https://github.com/ava-labs/avalanchego/releases/tag/v1.11.1

This release disables logic added in v1.10.18 that lead to validators sending an excessive amount of gossip to each other.

Avalanche Validators provision a stake-weighted bandwidth allocation for each peer and this buggy logic led to each node saturating their allocation with useless transaction gossip. This dynamic prevented pull queries issued by the validator from being processed in a timely manner and led to consensus stalling (as no polls were being handled).

As soon as a sufficient amount of stake upgrades to this release, consensus should return to normal.
Posted Feb 23, 2024 - 15:59 UTC
Investigating
Developers across the community are currently investigating a stall in block finalization that is preventing blocks from being accepted on the Primary Network.

Updates will be posted here as the issue is investigated.
Posted Feb 23, 2024 - 11:30 UTC
This incident affected: Network (Mainnet).