Yesterday, Credible fully understands was challenging for you, your Agency, care providers, and clients. We fully understand the criticality of our software and the role it plays with your Agency.
Starting at approximately 10:45 a.m. yesterday, Credible experienced an issue that severely affected our environment. Based on our real time monitoring system that showed an abnormal and severe spike and volume of traffic - all the hallmarks of a DDOS attack, we concluded it to be so. However upon further investigation, it was a false positive and merely a symptom of another issue. Please rest assured that we spared no expense, effort and retained experienced outside experts to find the root cause. That said, the source of the event was one of our main storage arrays becoming unresponsive due to several factors - suboptimal de-duplication of data, storage misallocations, write cache errors, and now adjusted monitoring levels. If only one of these factors occurred it would have been transparent, however because they transpired simultaneously - it adversely affected performance and availability.
We truly apologize for the service interruption and the time it took to recover - please understand that we needed to be thorough, take great care to safeguard the integrity of the data, and needed to determine performance would be consistent once the environment is operational.
We are committed to providing the best user experience, and we aren't stopping here. Credible will be engaging specific vendors to further review the environment to identify any nascent exposures, as well as working together to collapse restart times and processes. This may require us to replace or upgrade certain key components to mitigate any vulnerabilities and ensure quality of services. To focus on hardening our production environment, we will be postponing our ICD - 10 release on May 21st to May 28th. We will send out the proper notification so you may prepare for any disruptions.
No comments:
Post a Comment