GlobalNOC shines during disaster preparedness tests

What does GlobalNOC do when disaster strikes?  To find out, staff successfully completed an exercise to see how well they handled potentially crippling scenarios.

At 10am ET on November 9, 2013, the Systems Engineering Team shut down systems at IUPUI’s Informatics & Communications Technology Complex (ICTC) building, home to IU’s primary 24/7/365 Network Operation Center (NOC).  To simulate a complete loss and total inaccessibility, the shutdown affected all systems in the ICTC – including the NOC, data center, and all supporting technology.

Technicians designed the disaster contingency exercise to find out:

  1. If staff members could execute disaster recovery without instruction from management or supervisors
  2. If the IUPUI library provided an adequate contingency site 
  3. How much time was required for operations to rollover to a contingency location
  4. How well Bloomington and remote staff assumed full operational responsibility
  5. The potential impact to customers

As the exercise progressed, critical alarms prompted an operational transfer to the IU Bloomington NOC, followed by an evacuation of the ICTC facility.  In just 10 minutes, exercise participants were able to set up a contingency operation at the IUPUI library.  In turn, the Bloomington NOC assumed full responsibility without any lag in operations. The exercise concluded successfully at 1pm, and all GlobalNOC operations returned to normal.

GlobalNOC has a critical mission, and its monitoring systems must be available 100% of the time to sustain email, phone and overall network health continuity in the event one of the sites is lost.

All tests proceeded without a hitch, and GlobalNOC directors have already provided budgetary approval for areas noted for improvement. Disaster preparedness upgrades should be complete by the end of May 2014 – well ahead of the next Disaster Recovery Test during the fourth quarter of 2014.

Read the full exercise report.