Announcement to Site Administrators

Hello,


I wanted to personally reach out and keep you informed on the status of what has transpired over the past few days. 


First and foremost, I want to recognize that this is a significant inconvenience, and the timing is less than ideal for all parties involved. Once our Technology Management team is able to complete a full-system restore, I can begin to coordinate discussions with them to get more in-depth detailed information about this issue. I will then be issuing a root cause notification to explain the cause and the steps we will be taking to mitigate potential future situations, improvements and remediation plans moving forward. Fortunately, we believe this to be an incredibly rare occurrence, as it should be. 


WHAT IS HAPPENING?

  • The current state of the production outage: the SIS services were restored (minus the reporting server - still ongoing) at around 4 AM CST today. 
  • Over early morning hours and all day yesterday, our Technology Management & Engineering and Data Center teams were diligently working to get the SIS back online but ran into issues that delayed this process. The Unplanned Production Outage - 12-SEP-2018 details status updates as they were made available and the majority of the critical issues were fixed early this morning (minus the reporting server - still ongoing).
  • The Data Center, that maintains the physical hardware for the SIS, started work early this morning on restoring the reporting cluster, which is required to bring up the reports server. I have been informed that this is expected to take all day. Updates will be provided as they become available.
  • After the Data Center completes restoring the reporting server, the Technology Management & Engineering team will take over and begin work to bring the reporting databases online (i.e. final configuration and making sure the application infrastructure services are acceptable). Updates will be provided as they become available.
  • Unfortunately, we were only able to recover data from 11-SEP-2018 ~8 PM MT. We were unable to recover transaction logs after 11-SEP-2018 ~8 PM MT, therefore any data after this date/time up until the outage that occurred on 12-SEP-2018 ~4:34 PM CST will be lost.


DETAILS ABOUT WHAT IS OCCURRING

At the moment, I believe the most pressing matter is the loss of data between 11-SEP-2018 ~8 PM MT (9 PM CST) thru 12-SEP-2018 ~4:34 PM CST. After several hours and attempts to recover the data from the corrupt drives, our Technology Management & Engineering and Data Center teams were unsuccessful. This log data loss occurred due in part to the normal established SIS backup process. Transaction log backups are paused between 8 PM and 1130 PM and a backup copy of the database file is then set to kick-off during that window. It just so happened that the failure occurred at the exact moment the backup was set to run. Regrettably, any data your users removed, entered or manipulated between these dates/times can’t be recovered; there is nothing more that we can do. 


Furthermore, we still have the ongoing issue with the reporting server. As you may have already noticed, the SIS Home screen (e.g. dashboards screen) may display the message - "The destination in the SIS you are trying to access cannot be reached". This is due to the report server that is still offline and the dashboard widgets run off that reporting server. Please let your users know should they inquire about the issue. Additionally, the reporting server is the engine that drives the Reports module in the SIS. What this means is that all TopView reports, TopView Custom Reports, TopView Scheduled Reports, the Ad Hoc reporting tool to build TopView reports, and the use of Quick Query (with custom fields) services will not be available until the reporting server is restored and back online. I am working in tandem with the Technology Management & Engineering teams to gather updates every few hours. I will make all attempts to keep you all informed.  


Please know that we are actively continuing to monitor the SIS environment accordingly. However, as stated before, monitoring all potential issues may not be possible, and it is in those instances that we rely on our Student Solution Community to contact us should you experience a connectivity problem or issue. Users always have the option to submit a support ticket via Freshdesk or by sending an email directly to: support@epsupport.freshdesk.com with a detailed description of the issue at-hand.


HOW YOU CAN GET HELP

As always, you can always reach out to me directly via email or via phone. Also, I am actively updating the SIS community forums with regular updates as they come available, so be diligent and stay informed by visiting the announcements and known issues forums.


Thank you for your attention and understanding. 


All the best,


Mark Aguero
Product Manager, Student Solution
Campus Management Corp.
5201 Congress Ave., Boca Raton, Florida 33487