This Derby game got a lot of attention over the past couple days. Here's some explanation as to what's going on.
First, my results from running from the US midwest:
1052ms, 1068ms, 1261ms, 1100ms, 1118ms, 995ms, 937ms, 1090ms, 995ms, 985ms (average is 1060ms)
Some Monad implementation decisions:
1. Monad has a vote pacing mechanism which sets a minimum time between validators voting to every 500ms. This was added so that 2/3 of the validators can't outrun the other 1/3. Before this was implemented, Monad actually ran too fast in test environments - some validators weren't able to participate in voting because they couldn't keep up (maybe due to lower hardware specs, or due to distance from the majority, etc). There are also some theoretical timing games that could be played if a large portion of nodes are Byzantine.
Monad could run as fast the other L1s where stake is concentrated in a region (these have been discussed many times before) if this mechanism was removed. This was just a conservative decision to maximize stability and participation. It can certainly be revisited, to lower the minimum time.
2. Monad RPC reports tx inclusion only after one round of voting has occurred. This is also a conservative decision to reduce the chance of falsely reporting inclusion when there is a fork. This is resolved by the recent Monad BFT paper. Once that's implemented, RPC can be updated to give inclusion guarantee earlier. (Note that the regional L1s are achieving finality earlier due to #1 above.)
So the flow is:
1. web browser sends tx to rpc node
2. rpc node sends tx to next leader
3. leader sends proposal to validators
4. validators send votes to next leader
5. leader sends next proposal to validators (this includes the votes on that tx inclusion)
6. validators send to rpc nodes
7. web browser queries rpc nodes
Because of the above pace mechanism, it's going to be slightly over 500ms just to round-trip one proposal. Also, the tx on average will arrive to the leader in the middle of the window. The sum of these two on average is going to be a bit more than 750ms. Part of the additional latency of < 200-250ms is likely due to the additional hops. The team is actively working on investigating the additional latency (wanted to share my thoughts early).
Next steps:
1. Rollout the new consensus algorithm Monad BFT
2. Consider reducing or removing the vote pacing mechanism
3. Find and resolve the other latency inefficiencies
Other comments:
1. Not concerned with competing with centralized L2s which are providing preconf/miniblocks - Monad is intended to be a globally decentralized network
2. Not concerned with competing with regional L1s who run fast just due to the majority of stake being close to each other latency-wise
3. Some issues were likely just RPC proximity - for example we believe the Australia test was hitting Japan RPC nodes so several of the hops above just increase significantly (so we need better RPC node distribution and geo routing) - we'll follow up on this
4. Some issues were likely RPC rate-limiting or other issues. We'll follow up on this.
5. As I've mentioned in some podcasts, we didn't spend as much time on RPC as we should have, so there's been some hiccups in testnet related to RPC. I don't think that played a role here, but noting just for completeness.
6. For some apps, pipelining makes sense and can hide latency that gives a similar ux for a globally decentralized network as a single server.