Data centre outage hits all Queensland hospitals

Queensland’s hospitals and health services are without access to key systems and have been forced in some instances to revert to paper-based processes after a routine upgrade in the health department’s core Brisbane data centre went awry.

On Wednesday afternoon, Queensland Health issued a public advisory explaining that a data centre issue meant that all the state’s hospitals were turning to “contingency plans” to maintain services in the absence of a number of clinical systems.

“No patients are being turned away from hospitals and surgeries are occurring as normal. Routine downtime procedures have been implemented to ensure patient safety," the advisory read.

“All hospital and health services have experienced some level of interruption to services but all have developed contingency plans to ensure services to patients continue."

Queensland Health’s acting chief technology officer Paul Carroll told iTnews the outage had resulted from a problematic software upgrade to storage controllers in the department’s data centre located inside the Royal Brisbane and Women’s Hospital at Herston.

“We were conducting a software upgrade to our storage controllers across our three data centres,” Carroll said.

While working on upgrading a third pair of disk arrays, Carroll said the team encountered a memory issue, so started to roll back the upgrade and revert to the original state. Problems arose when they tried to bring the machines back into the cluster, and the whole storage area network (SAN) started to experience issues.

Carroll said between 30 and 40 important Queensland Health systems are connected to the Herston SAN, accounting for the the widespread impact of the outage, which hit at about 11:00am Queensland time.

He described the sequence of events as “an extremely rare occurrence”.

“This sort of change should not impact the facility in this way," Carroll said.

The worst affected systems were those which are not yet fully redundant, including the encryption service Queensland Health uses on its discharge summaries, and the endoscopy system used across 33 hospitals in the state.

“Not everything we have is fully redundant yet. This is something we are working towards as part of our ongoing work program,” Carroll said.

The outage did not affect backed-up systems like the Hospital Based Corporate Information System (HBCIS), which Carroll described as the department’s “most important”.

The disk arrays are now back up and running, but Carroll and his team will spend Wednesday night and some of Thursday testing the integrity of data in the systems and making sure clinical users are 100 percent sure about what has been recorded and what will need to be updated manually.

He said he would not bring systems back online until business owners say it is safe to do so.

“We need to make sure those business owners know what information is up to date and which information they now have to go back and re-enter," he said.

“There is plenty of work to be done overnight."