thx4play

@heneseyyy

Joined July 2012

4.9K Following

473 Followers

810 Posts

heneseyyy retweeted

josh lee

@dogemos

almost 3 years ago

A personal retrospective on celestia launch What went well, what didn't go so well, and what to improve for Keplr and Osmosis 👇 Celestia was a bombshell of a launch. After a multiyear bear market, it was refreshing to see metrics fly through the roof. Celestia launch was the largest event since 2021 where people who have never used the cosmos stack, used it for the first time. First impressions are important. I agree that the first impression of the Osmosis / Keplr was likely not as impressive as people expected. Here’s a little bit of a background of what went well, what didn’t go so well, and how it can be improved for the future. 🥰 What went well 🥰 1. The Celestia mainnet launch itself. The @CelestiaOrg team did an amazing job of building innovative software, and launching the Celestia mainnet with a set of genesis validators. While the incentivized testnet was probably one of the most resource intensive testnet I’ve seen in a while, seeing mainnet create blocks flawlessly as genesis time approached proved that the team did a great job of testing the software and prepping the validators. 2. Airdrop distribution. I’d argue that the Celestia airdrop was one of the best crafted airdrops in recent network launches. While likely not perfect, they did a good job of filtering out potential sybils, distributing to a wide (and very strategic) group of stakeholders ranging from stakers, early adopters, researchers, code contributors, etc. The airdrop claims process could have been better (i.e. requiring multiple addresses for multiple criteria eligibility, inevitable geoblocking, etc), but all in all, the distribution was well thought out and created a strong pre-launch community. 3. Node endpoint infrastructure. I know I’ll get shit for saying endpoint infrastructure went ‘well’, but I genuinely think the @keplr_infra team did a fantastic job here. We saw the largest spike of traffic in Keplr history. We were expecting about 3x the usual traffic, but saw over 10x. The team scaled endpoints in 10-15 minutes. And note that not only did we have to scale Celestia endpoints, but also every other natively integrated chain’s endpoints because opening a wallet refreshes balances for ALL chains. Considering the monumental task they faced in the midst of immense pressure, I’d still rate this a success. 4. Skip Protocol’s swap API @SkipProtocol Swap API is the backbone of Keplr Swaps. It was able to detect and service Celestia and Osmosis quickly, then add the new liquidity pools that were created shortly after. The process was largely automated, which gives me confidence that this will scale well as we see more chains/assets launch in the future. I want to emphasize that the Keplr Swap issues you saw had little to do with Skip, but rather a more fundamental issue with Cosmos fee logic(discussed below). 😬 What didn’t go so well 😬 1. Keplr multichain UX Despite our infra stack handling the influx of users fairly well, a more fundamental issue was Keplr’s multichain design. More specifically the ‘manage chain visibility’ feature. We implemented this because we saw multiple users highlighting how manually selecting a chain to see a balance (a la Keplr v1) was neither user friendly nor scalable–to which we agreed. However, the new UX pattern is extremely confusing for unfamiliar users. It’s hard to notice when a new chain is available. It’s annoying to enable a chain’s visibility to each account separately (like me who have 20+ accounts in my wallet). Overall, I still think the idea of a chain being less prominent as a point of interaction in a wallet is a direction we’ll continue to move towards. However, it’s good intention that was poorly executed. We will reexamine if there’s a better way to go about it–where non-Cosmos users can understand at a first glance. Let me know if you have ideas. 2. Keplr’s error message needs to be improved Things break. That’s inevitable. However, what isn’t inevitable is providing a better way of letting users know any information on what broke, why it broke (it may not be a Keplr issue, but a chain issue, Ledger issue, endpoint issue, etc), and how you can potentially fix it. The challenge of a wallet is that you get error messages from external sources. Chains spit out error messages. Endpoints spit out error messages. Ledger hardware wallets spit out error messages. However, as a user–you don’t know which of these things the error came from. This hopefully helps us as well, where we can spend less time attempting to replicate bug reports. We’ll work on implementing better error handling, and friendlier messages. More contextual information on the error, and steps you may be able to take to make it work (if possible). 3. Cosmos fee/mempool logic. IBC as a protocol was not the issue. Relayers were not the issue. ‘Better IBC’ doesn’t fix this. Better relayer software doesn’t fix this. Self-relaying doesn’t fix this. Only better fee logic and transaction filtering does. I want to emphasize that since there seems to be misunderstandings on how IBC works. The reason why you saw so many IBC transfers fail was a more fundamental reason: Osmosis (and probably all other Cosmos chains today) did not have a smarter fee logic and transaction filtering. For a successful IBC transfer to occur, relayers need to move packets of information between chains, and submit it to each chain as a transaction. The issue was that the chain was so bottlenecked by various arbitrage bots essentially spamming the chain with their transactions, that even relayers, a special class of citizens that Osmosis depends on as a key component of operations, were not able to submit transactions. Preventing an event like yesterday from happening is not by changing how IBC works, but by fixing how the chains handle transactions (mempool, fees, messages, etc). Unfortunately, Cosmos has, in my opinion, a fairly outdated model of handling transaction priority. It’s further exacerbated by the fact that transaction fees are parameters set individually by the validators. While the intention was to allow validators to be selective about tx fees they are willing to process a transaction for, and for there to be a competitive market amongst validators around it, the reality is that there is no ‘priority’. This means chains treat an invalid arb transaction at the same importance as a crucial IBC packet being submitted by a relayer. Furthermore, some of these bots submitted unreasonably large gas amounts for their transaction (i.e. 20x beyond what was needed). Even if these transactions fail (i.e. due to it being invalid, prices fluctuating, etc), it still took up blockspace (which has a limitied amount of gas per block). This meant a failed arb transaction took the space for what could have been 20 user transactions/relayer transactions. Osmosis has a parameter to selectively apply a different gas price for arb transactions (atomic transactions where the input and output denom is the same, but takes multiple swaps in between) and a separate fee parameter for transactions with an exceptionally large gas amount. However, these parameters have to be individually applied by validators which increased the coordination and communication cost. I’d still like to acknowledge the many Osmosis validators who’ve been incredibly quick to respond and apply these recommendations as needed. Ultimately, there needs to be long term solutions. The following are some things being worked on which hopefully would permanently resolve these issues from happening. - Better fee handling (a la EIP-1559) which dynamically adjusts to blockspace demand - Skip’s more intelligent block building (where hopefully you can give relayers submitting a packet a higher priority over other transactions) - Zaki’s ideas to even remove the mempool from CometBFT to a more intelligent mempool (see https://t.co/5GkcrfxYTN) - and likely many more which I’ve probably failed to include (please feel free to reply below!) (bonus) IBC timeouts and cancellability While I mentioned that IBC didn’t fail, I do think there are ways IBC UX could be improved in scenarios such as yesterday. IBC timeouts were set too short (i.e. 10 mins), which meant even if the transaction could have gone through if it took 15 minutes, any transaction over 10 minutes were considered invalid. We think improvements such as increasing the timeout while also allowing explicit cancellation of pending IBC transactions, retry logics, etc would help massively in making IBC transactions not feel like ‘IBC purgatory’. ---------------- I’d like to point out that a lot of these problems are problems that exist in all of Cosmos. It came to light because of the scale of transactions we ran into yesterday, but it could happen to any chain. The bad news is that many people may have had a bad first impression on IBC / Cosmos stack. For that, I apologize. I genuinely hope you stick around, because when it works well (which is like 99% of the time), @cosmosibc is one of the most magical cross-chain experience you can have in all of crypto! The good news is that now that we know what the point of failure is for IBC to work well (the fee/tx handling logic), we can work to fix it! And once this is fixed, it's a lesson no other Cosmos chains have to go through. Cosmos has had its fair share of chaotic failures. (Remember Terra? 😉) But as an ecosystem and as a technology, it continues to iterate for the better. ✨ Huge shout out to all the @osmosis validators, @crypto_crew (the relayer heroes), and the osmolabs + keplr + osmosis support team members who worked tirelessly to resolve these issues.

dogemos's tweet photo. A personal retrospective on celestia launch

What went well, what didn't go so well, and what to improve for Keplr and Osmosis 👇

Celestia was a bombshell of a launch. After a multiyear bear market, it was refreshing to see metrics fly through the roof. Celestia launch was the largest event since 2021 where people who have never used the cosmos stack, used it for the first time.

First impressions are important. I agree that the first impression of the Osmosis / Keplr was likely not as impressive as people expected.

Here’s a little bit of a background of what went well, what didn’t go so well, and how it can be improved for the future.

🥰 What went well 🥰

1. The Celestia mainnet launch itself.

The @CelestiaOrg team did an amazing job of building innovative software, and launching the Celestia mainnet with a set of genesis validators. While the incentivized testnet was probably one of the most resource intensive testnet I’ve seen in a while, seeing mainnet create blocks flawlessly as genesis time approached proved that the team did a great job of testing the software and prepping the validators.

2. Airdrop distribution.

I’d argue that the Celestia airdrop was one of the best crafted airdrops in recent network launches. While likely not perfect, they did a good job of filtering out potential sybils, distributing to a wide (and very strategic) group of stakeholders ranging from stakers, early adopters, researchers, code contributors, etc. The airdrop claims process could have been better (i.e. requiring multiple addresses for multiple criteria eligibility, inevitable geoblocking, etc), but all in all, the distribution was well thought out and created a strong pre-launch community.

3. Node endpoint infrastructure.

I know I’ll get shit for saying endpoint infrastructure went ‘well’, but I genuinely think the @keplr_infra team did a fantastic job here.

We saw the largest spike of traffic in Keplr history.

We were expecting about 3x the usual traffic, but saw over 10x.

The team scaled endpoints in 10-15 minutes. And note that not only did we have to scale Celestia endpoints, but also every other natively integrated chain’s endpoints because opening a wallet refreshes balances for ALL chains.

Considering the monumental task they faced in the midst of immense pressure, I’d still rate this a success.

4. Skip Protocol’s swap API

@SkipProtocol Swap API is the backbone of Keplr Swaps. It was able to detect and service Celestia and Osmosis quickly, then add the new liquidity pools that were created shortly after. The process was largely automated, which gives me confidence that this will scale well as we see more chains/assets launch in the future. I want to emphasize that the Keplr Swap issues you saw had little to do with Skip, but rather a more fundamental issue with Cosmos fee logic(discussed below).

😬 What didn’t go so well 😬

1. Keplr multichain UX

Despite our infra stack handling the influx of users fairly well, a more fundamental issue was Keplr’s multichain design. More specifically the ‘manage chain visibility’ feature.

We implemented this because we saw multiple users highlighting how manually selecting a chain to see a balance (a la Keplr v1) was neither user friendly nor scalable–to which we agreed. However, the new UX pattern is extremely confusing for unfamiliar users. It’s hard to notice when a new chain is available. It’s annoying to enable a chain’s visibility to each account separately (like me who have 20+ accounts in my wallet).

Overall, I still think the idea of a chain being less prominent as a point of interaction in a wallet is a direction we’ll continue to move towards. However, it’s good intention that was poorly executed. We will reexamine if there’s a better way to go about it–where non-Cosmos users can understand at a first glance. Let me know if you have ideas.

2. Keplr’s error message needs to be improved

Things break. That’s inevitable.

However, what isn’t inevitable is providing a better way of letting users know any information on what broke, why it broke (it may not be a Keplr issue, but a chain issue, Ledger issue, endpoint issue, etc), and how you can potentially fix it.

The challenge of a wallet is that you get error messages from external sources. Chains spit out error messages. Endpoints spit out error messages. Ledger hardware wallets spit out error messages. However, as a user–you don’t know which of these things the error came from. This hopefully helps us as well, where we can spend less time attempting to replicate bug reports.

We’ll work on implementing better error handling, and friendlier messages. More contextual information on the error, and steps you may be able to take to make it work (if possible).

3. Cosmos fee/mempool logic.

IBC as a protocol was not the issue. Relayers were not the issue.

‘Better IBC’ doesn’t fix this. Better relayer software doesn’t fix this. Self-relaying doesn’t fix this. Only better fee logic and transaction filtering does.

I want to emphasize that since there seems to be misunderstandings on how IBC works.

The reason why you saw so many IBC transfers fail was a more fundamental reason: Osmosis (and probably all other Cosmos chains today) did not have a smarter fee logic and transaction filtering.

For a successful IBC transfer to occur, relayers need to move packets of information between chains, and submit it to each chain as a transaction. The issue was that the chain was so bottlenecked by various arbitrage bots essentially spamming the chain with their transactions, that even relayers, a special class of citizens that Osmosis depends on as a key component of operations, were not able to submit transactions.

Preventing an event like yesterday from happening is not by changing how IBC works, but by fixing how the chains handle transactions (mempool, fees, messages, etc).

Unfortunately, Cosmos has, in my opinion, a fairly outdated model of handling transaction priority.

It’s further exacerbated by the fact that transaction fees are parameters set individually by the validators. While the intention was to allow validators to be selective about tx fees they are willing to process a transaction for, and for there to be a competitive market amongst validators around it, the reality is that there is no ‘priority’. This means chains treat an invalid arb transaction at the same importance as a crucial IBC packet being submitted by a relayer.

Furthermore, some of these bots submitted unreasonably large gas amounts for their transaction (i.e. 20x beyond what was needed). Even if these transactions fail (i.e. due to it being invalid, prices fluctuating, etc), it still took up blockspace (which has a limitied amount of gas per block). This meant a failed arb transaction took the space for what could have been 20 user transactions/relayer transactions.

Osmosis has a parameter to selectively apply a different gas price for arb transactions (atomic transactions where the input and output denom is the same, but takes multiple swaps in between) and a separate fee parameter for transactions with an exceptionally large gas amount. However, these parameters have to be individually applied by validators which increased the coordination and communication cost. I’d still like to acknowledge the many Osmosis validators who’ve been incredibly quick to respond and apply these recommendations as needed.

Ultimately, there needs to be long term solutions. The following are some things being worked on which hopefully would permanently resolve these issues from happening.

- Better fee handling (a la EIP-1559) which dynamically adjusts to blockspace demand
- Skip’s more intelligent block building (where hopefully you can give relayers submitting a packet a higher priority over other transactions)
- Zaki’s ideas to even remove the mempool from CometBFT to a more intelligent mempool (see https://t.co/5GkcrfxYTN)
- and likely many more which I’ve probably failed to include (please feel free to reply below!)

(bonus) IBC timeouts and cancellability

While I mentioned that IBC didn’t fail, I do think there are ways IBC UX could be improved in scenarios such as yesterday. IBC timeouts were set too short (i.e. 10 mins), which meant even if the transaction could have gone through if it took 15 minutes, any transaction over 10 minutes were considered invalid.

We think improvements such as increasing the timeout while also allowing explicit cancellation of pending IBC transactions, retry logics, etc would help massively in making IBC transactions not feel like ‘IBC purgatory’.

----------------

I’d like to point out that a lot of these problems are problems that exist in all of Cosmos. It came to light because of the scale of transactions we ran into yesterday, but it could happen to any chain.

The bad news is that many people may have had a bad first impression on IBC / Cosmos stack.

For that, I apologize. I genuinely hope you stick around, because when it works well (which is like 99% of the time), @cosmosibc is one of the most magical cross-chain experience you can have in all of crypto!

The good news is that now that we know what the point of failure is for IBC to work well (the fee/tx handling logic), we can work to fix it! And once this is fixed, it's a lesson no other Cosmos chains have to go through.

Cosmos has had its fair share of chaotic failures. (Remember Terra? 😉) But as an ecosystem and as a technology, it continues to iterate for the better.

✨ Huge shout out to all the @osmosis validators, @crypto_crew (the relayer heroes), and the osmolabs + keplr + osmosis support team members who worked tirelessly to resolve these issues.

406

147K

thx4play

@heneseyyy

Who to follow

Last Seen Users on Sotwe

Trends for you

Most Popular Users