CSIRO's 'cloud right program' curbs instance, cost sprawl

The CSIRO is the first government agency in the country to provide managed cloud accounts with all three major providers to keep the storage and compute needs of its researchers in check.

It represents the first phase of the peak science and research organisation’s 'cloud right program', which aims to centralise cloud provisioning oversight while improving access to on-premises resources.

Cloud platforms team lead Brendan Speet, who manages the program, told a HashiCorp secure public cloud webinar the work was born from a need to secure the management layer and provide the guardrails for cloud consumption.

“Research, by its nature, is very difficult to articulate the computing requirements [for] upfront,” he said, adding that “capacity management, skillset requirements and the time to delivery for research workloads” has only become more challenging.

“More often than not, when we were engaging with research projects to gather their requirements, the response that we were getting was 'we just need storage and compute, how much we don't know', 'we just need to test something', or 'we just need to build a prototype'."

Speet said that researchers had been used to having the management layer taken care of by CSIRO's central IT departments, which worked “reasonably well within the walls of … on-premises networks”.

But with the arrival of cloud, researchers across the organisation's 10 business units found they could gain faster access to compute for their projects, though often did so without much regard for security or management.

“It didn’t take long for simple demonstrations that were spun up on developer cloud accounts to turn into production workloads, or researchers to start running multiple projects out of a single cloud account,” Speet said.

“Many a time, we’ve engaged with research projects that were using cloud. We found that if there was a team member within the research project that had the ability to spin up an EC2 instance, they were now the self-appointed tech lead for a project.

“Users were starting to spin up EC2 in the same way they would VMs on-premises, not understanding what the shared responsibility was in the context of cloud, especially when it came to security.”

Cloud right

The CSIRO established the cloud right program in 2018 to streamline the process that researchers take to consume cloud, including securing the management layer and providing the necessary guardrails. 

Speet said phase one of the program, which is almost complete, “focused on the management layer”, which was “need[ed] to start getting control over federated billing and chargeback”. 

He said this has provided centralised billing aligned against projects to enable better forecasting of cloud costs; savings on central agreements, including better understanding of compute and services in use; and an automated chargeback process to remove cloud administration responsibilities from researchers.

It has also involved defining the security requirements for cloud, including around authentication and authorisation, and an automation and configuration baseline to streamline the process of accessing cloud.

“We needed to remove duplication and complexity, speed up the development time to production with appropriate templates, simplify solution building collaboration, [and] provide a simpler method to scale up users to provision resources across multiple vendors,” Speet said.

As part of the third-party toolset requirements for the program, the cloud right team implemented HashiCorp's open source security and infrastructure-as-code tools Vault and Terraform Enterprise.

Speet said the organisation settled on the tools as they were gaining momentum in the development community as a more standardised approach to managing and provisioning infrastructure-as-code deployment, along with managing secrets.

“It was very clear we needed a managed Vault and Terraform implementation to overcome the complexity and duplication of deploying multiple vaults, along with the need to centralise the management of state files,” he said.

Working with the three major cloud providers, CSIRO has developed a common automation and configuration baseline for Amazon Web Services, Microsoft Azure and Google Cloud Platform.

This baseline configuration is deployed “to each and every cloud account that gets handed out”, which researchers can access by submitting a custom-built cloud request form that automatically generates a service desk request.

“From my understanding, CSIRO is the first government agency in Australia to provide managed cloud across the three major vendors, alongs with the implementation of Terraform and Vault,” Speet added.

Next steps

Having laid the foundations, Speet said the team is now starting to build "modules for approved solutions to remove duplicated effort and complexity."

“This will increase security, improve the build time to production and allow researchers to concentrate on getting back to doing just research,” he said.

“But we have to be realistic. 'ClickOps' [cloud providers' default screen of clickable options for service configuration] isn’t going to be going away anytime soon for most researchers, as a full digital transformation will take time.”

Speet said there is also a need to “concentrate on building internal communities to encourage more collaboration around the best usage patterns of cloud, and the importance of secrets management”.