Final update for the evening. The chain seems to be progressing. As more validators update and reload from a snapshot, we should continue to see them come back online. Core devs are moving to continuous monitoring of the Consensus Group and block production.
Reminder, please see announcements in Discord channels if you operate a chain following node. Hotspot owners do not need to make any adjustments and PoC activity should return to the state it was prior to this halt after several hours.
A postmortem report will be prepared and shared as soon as we can prepare it later this week.
Posted Aug 09, 2022 - 08:02 UTC
Monitoring
Update: Blocks are moving again but the number of transactions in each block is low (about 10% of expected txns). We still have not seen a reward block, so we're continuing to monitor.
Updates for ETL and Node are currently being tested. Validators have their update already.
Posted Aug 09, 2022 - 05:09 UTC
Update
Update: We have an emergency release going out to Validators (and another release for ETL, Node, and Router operators shortly).
The emergency release includes a new snapshot that all Validators will sync to to realign ledgers (there was a ledger drift).
Update again soon.
Posted Aug 09, 2022 - 03:04 UTC
Update
Update: Just setting some expectations here. We expect this to be a long chain halt (hours +) and there will be several emergency releases going out to Validators, Node, and ETL operators. Announcement on when that's available will be made separately. Chain is halted, not dead.
Again, pending transactions will not go through, rewards are affected as the chain is not moving, but it's affecting everyone equally.
Posted Aug 09, 2022 - 00:37 UTC
Update
Update: We have halted again at 1,469,988 and getting more clarity at the root cause. We believe receipts in blocks cannot be absorbed. More details coming as soon as we have a plan.
Posted Aug 08, 2022 - 22:45 UTC
Identified
Update: The core team has asked all Validators currently in the Consensus Group to restart their Validators in order to resume block production.
Posted Aug 08, 2022 - 20:34 UTC
Update
Update: We're still looking, but have a potential solution percolating. In the mean time, I want to also mention that device packets are still sending on the network despite the chain halt due to the fault-tolerant system the core team has put in place.
Posted Aug 08, 2022 - 20:18 UTC
Update
Some nodes are able to progress to block 1469953 but not all. This issue may auto-resolve but the core team is still investigating.
Posted Aug 08, 2022 - 19:52 UTC
Investigating
The core team is investigating the chain halt that occurred at block height 1469951 as a result of activating the latest chain variable.
All funds are safe, but pending transactions will be delayed until we can resolve this halt. Rewards will be delayed until the chain can move forward.