Health criticised for error-prone Australian vaccination data

Australia’s Covid-19 vaccine rollout was hampered by poor data quality and oversight in the IT systems used by the Department of Health and Aged Care to manage vaccinations.

According to a report published by the Australian National Audit Office yesterday, data quality was a major issue, with error rates in some systems as high as 14 percent.

Data accuracy was hampered by the large number of users of the systems, and a lack of oversight by Health.

“Health has not formally reviewed the data entered into any of the systems … This has resulted in undetected and undisclosed inaccuracies in the data”, the auditor said.

After the delays experienced in vaccine approval, Australia began engaging the industry in late 2020, allocating funds in the December MYEFO  that year.

In May 2021, Services Australia rehosted the Australian Immunisation Register (AIR) onto a private cloud, to cope with expected demand.

Other systems were outsourced to various providers: Salesforce ran the Vaccine Administration System (VAS) which handled ordering; Salesforce and AWS operated the Covid-specfic Covid-19 Vaccine Administration System CVAS; and Accenture managed reporting dashboards. 

In February 2021, Accenture also implemented the delivery-side Vaccine Data Solution (VDS) which handled logistics and reported vaccine coverage.

The audit is critical of Health’s management of the data project: “Health put in place effective monitoring and reporting arrangements using the best available data.

“However, it did not undertake sufficient reporting against targets, and it does not have adequate assurance over the completeness and accuracy of the data and third-party systems”.

The auditor expressed concern for confidentiality, integrity and availability of data.

“Health relies on point of time assessments, contractual obligations and management statements from entities. These are not sufficient to demonstrate that IT controls have been implemented and were operating effectively over the vaccine rollout.

“This increases the risk that third party providers do not have appropriate IT controls in place over the security of this data,” the report stated.

AIR and CVAS were singled out as being error-prone.

“Health does not gain assurance about the quality of the data it uses for monitoring and reporting immunisation coverage and has not clarified responsibility for data quality [in the AIR]”, the report stated.

The lack of any data review “has resulted in undetected and undisclosed inaccuracies in the data, particularly in the AIR and CVAS systems which are distributed systems and require manual input from many users.”

As a result: “The most recent National Centre for Immunisation Research and Surveillance study on AIR data in 2018 identified an error rate of 14 percent.

“Despite known inaccuracies in the collection of data, Health has used the available data as the most reliable data source for planning and supporting the vaccine rollout. In its use of this data, Health has identified anomalies and rectified these through its providers.

“However, Health has not quantified the inaccuracies in its internal or external data processes in the period examined.”

The ANAO recommended that the department “ensure it regularly obtains and reviews assurance over the data quality and IT controls in place in externally managed systems on a risk basis, including IT security, change management and batch processing”, and Health has agreed.