Website Outage for rchain.coop on SquareSpace - PostMortem

What: rchain.coop website was temporariliy unresponsive

When: Jan 2nd, 2018: 10:07 am
Why: RChain was plugged on CoinMarketCap by Ian Balina - and we received over 18K new visits to the site.
Technical Cause: Still looking into but .. either high burst of traffic triggered event in intrusion mitigation system and a back-off period began or high iowait existed on host due to lack of resources.

Event log

10:06 am: Kitblake reports in Discord #admin channel that a Tg user is reporting that Rchain.coop is down and confirms outage. Notifies Patrick727

10:07 am: Nash Foster sends telegram notification to Medha Parlikar (Unlicensed) : RChain website down

10:20 am: Patrick727 confirms that the site is back up.


Info


site: https://www.rchain.coop/
squarespace.com uses a load-balancer or ADC(Application Delivery Controller) (probably NGINX). I would think it caches site in ADC and plays it from memory but maybe not

I can't see any global outages at that time
https://status.squarespace.com/

but maybe it just got reported later as in excerpt from page


Resolved - This incident has been resolved. 
Jan 2, 13:37 EST
Monitoring - A fix has been implemented and we are monitoring the results. 
Jan 2, 13:26 EST
Investigating - We are currently investigating an issue with some images not displaying correctly. 
Jan 2, 13:00 EST

Mitigation & Avoidance

Never put anything into rchain ecosystem without monitoring from a centralized source of truth.
Coordinate in future press releases to pre-allocate extra resources (automated scaling can take a little time) and have watch performance monitoring closely for fast reaction if their is an issue. 


Crypto is hot. Expect a lot of interest in RChain especially in what and how we are going about it. 

  • We are moving site to our own services to ensure premium service.
  • Always cache queries in ram.  No spindle disk or even SSD. Use NVME. You can rip Wordpress to static site as well.

ref: https://support.squarespace.com/hc/en-us/articles/206545657-My-site-is-loading-slowly

ref: https://support.squarespace.com/hc/en-us/articles/217999797-Traffic-Overview-Analytics