Lighter is an app-specific Ethereum L2, so it operates a sequencer that processes every order, cancel, deposit, withdraw, etc., subject to published rules that are encapsulated in the ZK cicruits and must be followed

The sequencer ultimately relies on hardware running on a cloud provider, co-located close to where most of our traders are located geographically. The cloud provider provisions CPUs, databases, and other resources for the operation of the Lighter sequencer

In the hours preceding the outage, the markets were getting increasingly volatile, culminating with an extreme period of volatility between 5PM and 6PM EST, a period many market observers described as the most volatile in 10 years

This resulted in TPS for the overall Lighter network that the sequencer has to process to exceed 10X its normal TPS, in some blocks as much as 100X

To be clear, in a world of infinite hardware supply, the Lighter system can support arbitrarily high TPS, it's just a matter of having enough machines to sequence the transactions and, in a separate process, generate proofs of the correctness of transactions

Therefore, as we do not want to pass on arbitrarily high costs to our trading customers all the time, the logic for scaling machines up or down has been somewhat conservative in terms of costs. And, importantly, the database resources used by the sequencer has been modest

During the process of launching public mainnet last week on October 1, our team understood that the database resources will likely struggle with the growth that will come from the launch, and had a plan in place to upgrade the DB, however it kept getting pushed to avoid downtime

It is not lost on us that by trying to solve for the perfect time to upgrade the infrastructure, we missed doing so before the period of extreme market volatility on October 10. The only silver lining is that the DB held up during the worst of the market moves and only gave up

This will be an important lesson for the engineering team to not let "the perfect be the enemy of the good" when addressing infrastructure.

Because of how the different components of the system are interconnected, it turns out that while upgrading the database while the system is healthy can take 10-15 minutes, upgrading during an unhealthy period does not have a predictable ETA.

For that reason, once the outage did start at 10:30 PM ET, the decision was made to bring back the existing DB and get it to work instead of immediately upgrading in the middle of the outage that degraded other parts of the system too

After some very hard work by the engineering team during the middle of Friday night / Saturday morning, the system came back online at 3AM and other components including the pricing and risk engines were live too. We had to be careful to make sure there were no bad orders or

Meanwhile, the customer support team had been in touch with traders on Discord, TG, and X. Many were affected by the outage in being unable to get in and out of positions that they intended to. That said, thankfully this 4.5 hour stretch happened after markets started to calm

Among other tools we used to track who was affected and how, a form was circulated to the trading community to fill out how someone was affected. If you were affected and still haven't filed a ticket or filled out the form, please do (links in Discord)

While we took steps to improve the health of the system, and now it has been back online for almost 24 hours, the main step will be to finally perform the infrastructure upgrade - this will happen at 7AM EST (a little over 5 hours from now), planned downtime of 10-15 min

As this outage was clearly due to mistakes on our end, we are planning to compensate traders. In addition to the work of the customer support team, the data science team has been analyzing which traders had losses and/or liquidations during this outage window

On Monday, we will announce the details of the mass compensation event based on that data and reported issues, likely a combination of stablecoins and points distributed in a fair manner. A smaller part of that compensation will also go to those affected by the unrelated LLP

Some initial stats we can share: during the 4.5 hour outage window, 2008 traders lost more than 1K, 367 traders lost more than 10K, and 38 traders lost more than 100K. Liquidation fees of over 1K were paid by 30 traders, and none over 10K. We are also considering losses and

There are a lot of important lessons for us here and we thank the community of our traders for their support and understanding of growing pains of scaling. We always welcome feedback, whether specific to your trades or generally about our protocol and product.

/fin End of message

@Lighter_xyz Massive respect to the @Lighter_xyz team 🕯️ Handled everything fast, transparent, and fair. If anything, this showed how solid the team really is, the future’s brighter than ever 🔥

@Lighter_xyz good transparency, but tell those vc and insiders to give more money to engineering team so they get better hardware so something like this doesnt happen again..! Glighter🕯️

@Lighter_xyz Love to hear this. Lighter is on early stage. A lot of rooms to improve. Even Hyperliquid randomly went offline, just 1-2 months ago. For CEX, we are familiar with Binance and before that, Bitmex outages.

@Lighter_xyz Looking at the overall situation, this, although it was the first serious outage, was not a serious problem It was quick, did not result in additional losses for users, and has already been resolved and learned You can't avoid these events, but it depends on how you respond,

@Lighter_xyz Appreciate the transparency fr every project should do post mortems like this

@Lighter_xyz 🫡🫡

@Lighter_xyz GLighter!

@Lighter_xyz In 🕯️ we trust

@Lighter_xyz 🕯️

@Lighter_xyz

@Lighter_xyz gLighter

@Lighter_xyz 🕯️ ty

@Lighter_xyz 👀

@Lighter_xyz 🕯️

@Lighter_xyz What doesn’t kill you makes you stronger. We will see a much greater Lighter soon!

@Lighter_xyz Bullish as ever🗿🕯️