cloudblog
2016/01/21
January 21, 2016
3 min read

WSO2 Cloud Incident Report: Jan 21, 2016

WSO2 Cloud faced a serious degradation of service due to DNS failing to resolve addresses related to WSO2 Cloud URLs on 21st January 2016. Here is the incident report and actions taken. Start time: January 21, 2016 0923 PST Recovery time: January 21, 2016, 1001 PST Impact:
  • Reachability to all WSO2 Cloud functionalities was disrupted.
  • Partial service downtime due to DNS resolve failures for some parts of the world.
  • There was a 38-minute gateway downtime during this incident : https://uptime.cloud.wso2.com/
Root cause: Verification failure between WSO2 name-service provider and ICANN has resulted in disabling of WSO2 domains. Since WSO2 Cloud is run under a subdomain of wso2.com WSO2 Cloud was also impacted with the domain disabling. It is observed that this has not affected to some regions because caching servers were still resolving for those regions. Actions:
  • As soon as our monitoring tools alerted us regarding the outage we escalated it to our infrastructure team.
  • They managed to resolve the situation by contacting the relevant parties and getting the services up.
  • Infrastructure team updated the contact details for the internet domain to ensure that verification calls always reach the team.