503 Service Unavailable
Incident Report for Textalk Webshop

Major outage on December 15

On Friday, December 15, we had major disruptions in our webshop platform. The internal memory of our load balancer was flooded and the system became slow and unresponsive. Our backup load balancer was not activated, which meant we could not deliver data on the requests.

Attack by a network of bots

Based on the logs, we could see that a sudden burst of requests in very short intervals came from all around the world. The origin was Wordpress sites, and we suspect that a botnet was used to carry out the attack. This botnet exploited weaknesses in Wordpress installations.

Aftermath

We managed to get our redundancy (the backup load balancer) started, which could handle traffic again. We replaced our load balancer and upgraded it's memory to handle extreme situations. We have also configured the system to ensure that redundancy is activated earlier.

Posted about 1 month ago. Dec 18, 2017 - 11:22 CET

Resolved
The system has been running on optimal performance during the night. Today the team will implememt additional fail safe machanisms to ensure this problem never arises again.
Posted about 1 month ago. Dec 16, 2017 - 06:36 CET
Update
System seems stable. We will continue monitoring through out the night.
Posted about 1 month ago. Dec 15, 2017 - 23:47 CET
Monitoring
A fix has been applied. Monitoring the system
Posted about 1 month ago. Dec 15, 2017 - 22:13 CET
Identified
We have identified the problem as an issue with our traffic load balacer. We have several people working to resolve the problem.
Posted about 1 month ago. Dec 15, 2017 - 21:43 CET
Investigating
Servers are currently not responding
Posted about 1 month ago. Dec 15, 2017 - 21:33 CET