Hyperliquid has partially addressed the concerns I raised two days ago. @pejisdev has mempool data from a node and ran some great analyses on how the the chain has behaved in the past few days, with a little input from me. TLDR: cancels are fixed, modifies still broken.
The team at @HyperliquidX took rapid action to address the prioritization of cancels. They were seemingly able to do so without a network upgrade which is really interesting - there must be a mechanism to control certain parameters/behaviours of the chain without a redeploy of the binaries.
Unfortunately ALO modifies still seem to lag when the chain is congested. This is problematic because it incentivizes market makers to cancel + post instead of modifying under load, which increases the load further.
Overall, very pleased to see how responsive the team at HL has been to this issue and others that have been flagged on discord as they continue to scale the chain, and am confident they can continue to deliver and improve.
Under load, Hyperliquid cancels were losing to orders signed after them. Thanks to @FrothyBeverage the issue went public, and the goated Hyperliquid team was fast to fix it.
We then worked together to go deeper and see whether anything else had broken.
Modifies were hit too, to the point where it is now faster to cancel and place a new order than to modify one. Here is the data.
Paper, code and data, so anyone can check it from Hyperliquid's public archive, or straight from the histograms we collected on the @femtosxyz node:
Paper link : https://t.co/QdwvPOSjev
Decentralization and open nodes let anyone check data like this independently. We see that as a strength, not a weakness, next to other perp DEXs. Thanks to Hyperliquid for making it all available.
Hyperliquid.
@HyperliquidX's IOC speed-bump mechanism has a flaw. Normally designed to protect makers in the interests of a more liquid market, it breaks down in periods of high activity. When blocks become full, pending modifies and cancels queue up for later blocks, so their latency increases. However, IOCs behave as though they have a separate blockspace dedicated for them, and their latency barely budges. You can see this in the lower left chart, showing latency of different order types during 19 activity bursts.
The result is that under relatively modest load (25-30k tps), IOC + 8 bps (priority fee) orders can pick off resting liquidity before it has a chance to get out the way. Meanwhile, under higher load (35-40k tps), even IOCs without any priority fee can pick off resting liquidity.
This, in my opinion, is antithetical to the ethos and design of the ALO and cancel prioritization mechanism and needs investigation by the team. It exists to reduce toxicity from HFT takers, and is breaking down precisely when most needed.
It may also be exploitable. If IOCs really are insulated from the congestion affecting cancels and modifies, a malicious user could spam orders, delaying makers’ cancels while their own IOCs go through, and taking the liquidity left stale.
One single example that occurred on AVAX today caused 1.3M$ of volume to be adversely selected for a total loss of ~13k$ over the subsequent 10 seconds. This affected us materially. Faced with such episodes of adverse selection, spreads and liquidity are undoubtedly worse than they could be otherwise.
Would appreciate if anyone with a contact at HL could share this with the team.
@0xC0RT yea it's part of the current status quo and makers will be adjusting with it in mind (or rather, in the data), but it could be improved so everyone (except the toxic takers) is better off
It is important to clarify that this is a fixable technical issue on Hyperliquid. The exchange has on the whole been wildly successful at pioneering a fair and efficient market design that benefits smaller makers and retail compared to the more traditional style.
@HyperliquidX's IOC speed-bump mechanism has a flaw. Normally designed to protect makers in the interests of a more liquid market, it breaks down in periods of high activity. When blocks become full, pending modifies and cancels queue up for later blocks, so their latency increases. However, IOCs behave as though they have a separate blockspace dedicated for them, and their latency barely budges. You can see this in the lower left chart, showing latency of different order types during 19 activity bursts.
The result is that under relatively modest load (25-30k tps), IOC + 8 bps (priority fee) orders can pick off resting liquidity before it has a chance to get out the way. Meanwhile, under higher load (35-40k tps), even IOCs without any priority fee can pick off resting liquidity.
This, in my opinion, is antithetical to the ethos and design of the ALO and cancel prioritization mechanism and needs investigation by the team. It exists to reduce toxicity from HFT takers, and is breaking down precisely when most needed.
It may also be exploitable. If IOCs really are insulated from the congestion affecting cancels and modifies, a malicious user could spam orders, delaying makers’ cancels while their own IOCs go through, and taking the liquidity left stale.
One single example that occurred on AVAX today caused 1.3M$ of volume to be adversely selected for a total loss of ~13k$ over the subsequent 10 seconds. This affected us materially. Faced with such episodes of adverse selection, spreads and liquidity are undoubtedly worse than they could be otherwise.
Would appreciate if anyone with a contact at HL could share this with the team.
@drwconvexity The frequent batch auction design that the blockchain lends itself to is actually very efficient, Hyperliquid specifically having enjoyed great market microstructure for a long time. Blockchain matching can also be more resilient to failures (e.g. hardware).
@0xniko0x priority fees can also get you to earlier blocks, since the speed-bump mechanism normally delays you to a later block than you could otherwise have been included in
@0xniko0x yeap within a block it's all cancels then all modifies then all takers or smth like that. Within those categories it depends on submission time and priority fee.
@quant_xbt Is there a meaningful improvement possible? I connect to multiple public endpoints and have had a good experience given appropriate logic to manage disconnects etc. Hard to know what being 0.x ms faster might help with but I doubt its much on hl
@Mookafish Since radiation scales wirh the fourth power, I wonder if the engineers considered using heat pumps - increasing radiator temp by only 50 degrees could halve their size
@bookdepth@todd_cfa The precedent is concerning and I agree with this, however the specific case revolved around a pattern of manipulative behavior. The point being that he intended to manipulate with and trade against his public statements, it was no accident or innocent change of thesis/timing