Have also noticed loss of incoming peers and pauses in transactions at the 48h mark as we experienced previously. Nodes that have been updated to 1.23 experience increasing instability on relays after updating to 1.23 the longer they run 15h or longer from my experience after this time nodes have started to "lag" and lose incoming connections which continues to increase until they are restarted.

About 20% of reporters dropped out (200+ reporters right now)

Bug still there in 1.25.1. If I understand this correctly, its related to reward processing. Won't this just keep getting worse as we get more wallets staking?

This becomes worse, we have to address this asap! 6mins without blocks and 100% usage.

Yep, could be please do something about this heavy calculation?

Yes, this is almost certainly due to the reward calculation. We are actively testing out some options (such as this) so that we can address this soon.

Hopefully, the epoch cutover calculations can be similarly spread out with the pulse technique.

Hopefully, the epoch cutover calculations can be similarly spread out with the pulse technique.

indeed, that's on our radar as well (it's the stake distribution calculation needed for the snapshot that is consuming the resources). the pulse technique is certainly one candidate solution.

My relays are losing a considerable amount of peers at the 48h time mark. The network behaves really unstable at this time and several SPOs (using decent hardware specs) have reported losing blocks because of this

Red Alert!! 5 minutes without blocks today.

Our fix for this has been merged in the ledger repo (IntersectMBO/cardano-ledger#2142), but we are still working on integrating the changes into the node.

Today, we have seen 10+ minutes.

With just 22% of nodes upgraded to 1.26.1 the issue was greatly improved on the last run through this:

This was the epoch before on 1.25.1

I consider this thus fixed and close the issue.