Lisk launch post-mortem: What went right, what went wrong

Yesterday, at exactly 8pm UTC, we published Lisk v0.3.0 and announced the launch of Lisk. What a launch that was! In this post-mortem we bring light to the problems we faced, explain why our web login isn’t/wasn’t accessible, and clear up misunderstandings.

First of all, I’d like to apologise for the rough launch and the harm that the DDoS attack caused. Since the launch, we have been working tirelessly to ensure that all our systems are working perfectly again. We simply didn’t anticipate this overwhelming demand and trading chicanery.

The only problem we still face at this moment are our web logins. The reason is the same as yesterday: huge amount of traffic and ongoing DDoS attacks. However, everything else is working very well. Users can install Lisk on various operation systems like Windows, Mac OS, and Linux. Local client installations are functional and the Lisk network is alive and healthy.

The current client is aimed at technical users, but it’s relatively easy to install it with our guides. The best option is to simply use the Docker installation if you are on Windows or Mac. After we have guaranteed the network’s stability and optimised a few important Lisk modules (e.g., peer communication); we will look into options to provide a one-click Lisk client.

Timetable

  • 8pm: Released Lisk v0.3.0 on GitHub with the source code attached.
  • 8pm: Launched login.lisk.io to the public.
  • 8pm: Released Lisk v0.3.0 binary builds on our downloads.
  • 8pm: login.lisk.io was under immediate DDoS attack with 50,000 concurrent connections and gigabytes of traffic.
  • 8:05pm-5am: We tried to mitigate the DDoS attack as best as possible by putting up login.lisk.io mirrors and hardening our server configurations.
  • ca. 8:45pm-12:45pm: downloads.lisk.io was under DDoS attack, which caused the binary installation script to stop working. The script downloads the Lisk binaries from the download server.
  • 10:38pm: We fixed and released the Lisk v0.3.0 docker version.

Our infrastructure

We have our entire infrastructure on Microsoft Azure. The nodes are extremely fast and high quality, unfortunately the deployment often takes quite long.

Each genesis delegate is running on its own node with 2 cores and 7GB RAM. That means for the genesis delegates alone, we are maintaining 101 individual nodes. These nodes are running in a secured, isolated environment with no API access.

Initially, login.lisk.io was running in a cluster environment which can be easily expanded with more nodes if necessary. It started including five nodes with 8 cores and 28GB RAM each.

In front of the cluster environment is our proxy server (load balancer). It automatically selects the node the user connects to and is dividing the work-load equally among them.

What happened

Unfortunately, the proxy server was attacked directly. That means even if we put up more nodes for the cluster from the beginning, there would have been no difference.

The attack hit us hard. There were 50,000 concurrent connections pointed towards our login.lisk.io cluster, generating terabytes of traffic. Several thousand concurrent connections pointed to the downloads.lisk.io server as well. This was the reason why no one was able to access our web login the whole night or install our binary client between 8:45pm-12:45pm.

Since then, we have been enhancing our node configurations to ensure that they can handle more traffic. We removed the cluster environment (because only the proxy server was attacked anyway) and added more independent nodes (listed below).

Please note, there are still ongoing DDoS attacks towards our web logins and download server but they have become much weaker than last night. We are still at 1000’s of concurrent connections though.

The mistakes we made

  • We didn’t setup a proper anti-DDoS strategy with a specialised team.
  • We didn’t have the docker container ready at launch time. We configured the network the whole day and we unfortunately ran out of time.

What went right and where we stand

  • Nobody lost any LSK. The glitch where transactions don’t get any confirmations is a web login issue, which occurs due to the heavy load. Lisk holders won’t have this problem on their local client.
  • The whole network didn’t fork at all, it’s running very smoothly since launch. It had only one small hiccup where no new blocks came for a few minutes, but it corrected itself.
  • The DDoS attacks pointed to our web logins doesn’t affect the stability of our network and blockchain at all. They also get weaker every hour.
  • All problems only occur on our web login due to the DDoS attacks. Everything else is running smoothly.
  • We have setup a multitude of mirror web logins. login01.lisk.io, login02.lisk.io, login03.lisk.io, login04.lisk.io, login05.lisk.io, login06.lisk.io, login08.lisk.io
  • UPDATE: We now have added a proper cluster to login.lisk.io, please try it out if you had problems until now.

The next steps

  • We turned our blockchain explorer online again, on a more powerful server.
  • We properly configured a cluster of 5 high-performance nodes for login.lisk.io again, this time with a more powerful proxy node and a better anti-DDoS configuration. Meanwhile, we get more messages that the mirror web logins work.

Frequently Asked Questions

  • My transactions don’t get any confirmations.

Please install the Lisk client locally on your computer, let it completely sync, and try it again. If the problem occurred locally, please restart the client, let it completely sync, and try it again.

  • login.lisk.io is down, can I use other nodes?

UPDATE: We now have added a proper cluster to login.lisk.io, please try it out if you had problems until now.

Please try one of our mirror web logins. login01.lisk.io, login02.lisk.io, login03.lisk.io, login04.lisk.io, login05.lisk.io, login06.lisk.io, login08.lisk.io

We don’t guarantee the security of community-hosted nodes, which by definition are not under our control. Please use them at your own risk.

  • I get the error “Account has no LSK”, how can I send all my LSK at once?

If you want to transfer all your LSK from one address to another you have to subtract the 0.1 LSK fee per transaction. That means if you own 1 LSK in your account, you can only send up to 0.9 LSK.

Team funds

Our LSK funds are all untouched and no LSK were sent to any exchange.

In the following weeks we will announce new Lisk advisers, who are major figures in the blockchain industry. They have been advising us in the background since March and are extremely helpful. In March, we also decided to grant them LSK from the adviser funds. Due to tax reasons, it was necessary to transfer the LSK before any trading activity started.

The transferred LSK do not belong to the Lisk team any more and we hold no responsibilities for them.

Lisk is still Lisk

Even though the web logins are not always accessible right now, this doesn’t change what Lisk is. Blockchain applications are bigger than ever before. Lisk needs a lot of work, yes. However, we are well funded and will continue this journey for many years into the future. This is the first day Lisk has been live, and we have thousands of days ahead of us, proving what Lisk really can achieve.

Building blockchain applications is our dream and won’t give up because our web login wasn’t accessible for a day or two. We will fight for Lisk and our vision, the vision that everyone has unlimited access to all kinds of applications from any device they want. We believe our cutting-edge technology platform is on course to disrupt the $77 billion app industry.

In the coming weeks we will announce a lot! An updated whitepaper, a detailed view of our vision, new advisers, Lisk updates, and more! We will be at many conferences. For example, we will be in Berlin at BlueYard together with all major blockchain startups or in China at the Global Blockchain Summit.

This is just the beginning.

Kind regards,

The Lisk Team