First, if you aren't aware, you can read the gummiworm specification here:
https://t.co/3dEdmR9ElP
Our head node had memory limit caps on various processes. This was done to ensure the machine stayed responsive under load, so we could still connect and debug any issues.
Unfortunately, they were set far too low for the main consensus process, At just 2gb instead of the intended 12gb. Typo fixed.π
Shortly before 13:00 UTC, the main consensus process restarted because of this memory cap and the ongoing load from the trading bots.
That caused the coil peers, which double-check our work and provide multisignatures to "confirm" the network activity, to lose their connection and silently fail to reconnect. First bug filed and fixed.
By the time we identified this, it had been roughly 40 minutes since the disconnection, and the coil peer started downloading the soft-confirmed blocks that had been produced. Normally, this process happens in real time, as blocks are produced, and it easily keeps pace. But with the large backlog, it would have taken 2 hours to download all of the soft-confirmed blocks, as it downloaded them 1 by 1 and serially, rather than pipelined. Second bug filed, we'll work on improving the performance of that download.
We wrote a quick tool to speed that up out of band, bulk-downloading the blocks and inserting them into the database.
Finally, that revealed an inefficiency in confirming large block stacks. One of the benefits of Gummiworm is that the coil peers can catch up thousands of blocks in a single confirmation. If the head sees a surge of traffic while the coil network is signing a previous stack, the coil can catch up all at once.
However, the code that generates these hard confirmations and the cryptographic commitments was accidentally quadratic in size, re-starting the problem. Third bug filed and top priority today.
Until we get that optimization down, we've turned down the traffic. Rather than running at 1000 transactions per second, we're running at an average of 100 transactions per second to ensure competition traffic is unaffected.
We'll ramp that back up once we land the performance fixes in the hard confirmation code!
And we're back!
https://t.co/blhvUO53yN
Sorry for the hiccup folks, we're monitoring closely and things seem healthy! Several deposits that were attempted during the outage failed without funds ever leaving user wallets as designed, and since recovery, several users have been able to deposit successfully.
Wouldn't be a testnet without these kinds of valuable insights that prepare us for a mainnet launch!
If you're interested in the gory technical details, you can find them in the π§΅ below!
We've locked in the seeds for the simulation; for the sake of verifying the competition data after the fact, here are three hashes which pin down the behavior:
Simulation git commit: b7c19e366757f8f26eace6232d30d11a81e175de
World commitment:
7a2e3a9874916670578430fba8ffb7414cff4fbf5f11e0236d55ecb1816cdbdb
News commitment:
f3eddfe8829a24ec8c451f3b4f0737fe6b0c20d830a691466ae85c620fcdd306
The Sugar Rush testnet and trading competition starts in just 40 minutes, at 13:00 UTC!
https://t.co/blhvUO53yN
You can deposit now, and you'll receive a bundle of fantasy tokens to trade with at the bell. Our automated bots are already making trades, so you can start watching the price!
Deposits are 10 mainnet ADA and take 15 minutes. You can withdraw at any time (and quit the competition), and you'll get your 10 ADA back regardless at the end of the competition.
We DO NOT recommend using hardware wallets for this testnet.
Note: In order to allow the team to deploy fixes and improvements during the competition, we have planned for 1 hour of downtime at 02:00 UTC each day, and a potential 4 hour downtime at the start of week 2. If there's nothing new to deploy, trading will continue as normal.
In preparation for the trading contest on Monday, we've prepared some SDKs and documentation for the programmers who want to try their hand at automated trading!
Dig in and start building:
πhttps://t.co/YmwnbQFxeZ
The Sugar Rush Testnet launches Monday!
Sundae Labs is sponsoring over $5k in prizes for a 2-week skill-based trading competition on the Sugar Rush testnet!
Official rules below π https://t.co/OuEA9LA6Pn
Think you actually know how to trade at max speed?
Sugar Rush is an upcoming CLOB DEX, powered by Gummiworm. In order to launch our testnet, we're sponsoring $5k USD in prizes towards a 2 week trading competition.
ποΈ Aug 24 - Sept 6
Full rules and details dropping soon.π
Sundae Labs showed up big at Buidler Fest Argentina π¦π·
Our CTO, Pi Lanningham (@Quantumplation), breaks down our latest projects:
β’ Midnight Capacity Exchange
β’ an early build of Gummiworm, a Cardano L2 solution
β’ and a first sneak peak at the @SugarRushDEX, powered by Gummiworm