Macquarie Group moves to cloud-based event brokers

Macquarie Group has shifted its event platform - used to exchange actions between parties and systems globally to enable a transaction - to run in the public cloud.

In an event-driven architecture, an event is “a change in state, or an update, like an item being placed in a shopping cart on an e-commerce website”, according to AWS. For Macquarie, an event could be an interbank transfer or a stock trade, for example.

An event broker - which is either middleware software, an appliance or a service - sits between event producers and consumers, routing the necessary event data between the parties to make the transaction go through.

Macquarie Group engineering director Ankil Patel said in a blog post that the group had moved from running “hardware appliances sitting in on-premises data centres [to] software event brokers residing in the cloud.”

Though it looked at other options, Macquarie ultimately decided to stick with its existing event platform vendor Solace, and simply use a different form factor of Solace’s event broker technology.

“With Macquarie’s goal of moving 100 percent of its infrastructure to the cloud by end of 2022, it became apparent that the future of the event platform was not hardware appliances sitting in on-premises data centres, but rather in software event brokers residing in the cloud,” Patel said.

“Adding to the importance: The Solace appliances were reaching end-of-life, which gave our small migration team a time frame of just 11 months.”

Macquarie decided to use Solace event brokers hosted inside of a Docker container. 

“The container seemed to be a perfect fit, given the eventual destination of Macquarie’s Amazon Web Services’ (AWS) virtual private cloud,” Patel said.

“Moving from the hardware appliance to a Docker container also meant we could start taking an infrastructure-as-code (IaC) approach. 

“We built a completely automated IaC pipeline that would deploy a cluster of highly available brokers in AWS using Macquarie’s cloud deployment platform (Arturo). 

“These pipeline configurations (such as CloudFormation templates for containers deployed to AWS), dictate the broker and cloud configuration, making sure that it is consistent for every deployment.”

Macquarie migrated from hardware to software event brokers one business unit at a time, which took a total of nine months.

“For each migration, we used Arturo to deploy the new cloud brokers in AWS,” Patel said.

“Then at the given time, applications sending and receiving events from the brokers switched from on-premises broker to cloud brokers.”

One of the advantages of running software-based event brokers is every business division now has its own.

“While the physical event broker appliances were powerful and capable, for cost-efficiency several lines of business shared a single broker,” Patel said.

Sharing an appliance made it difficult to perform maintenance, since different businesses had different availability requirements.

“Moving to software-based brokers means that each line-of-business has their own dedicated broker resources,” Patel said.

“As a result, maintenance windows can be customized to their individual needs … but because the individual brokers are bound together into an event mesh, the global connectedness remains.”

As more of Macquarie’s application and system landscape is re-platformed to run in the cloud, hosting the eventing in the same places promise to improve performance and reduce costs.

“Numerous applications that send and receive events now sit in the same cloud,” Patel said.

“With less network distance to span, events move even faster between applications, spending less time on the wire, pushing even more responsiveness.

“Events now don’t need to enter and leave the cloud as frequently to reach their destination. That reduces ingress and egress charges, a significant cost associated with cloud.”

Patel said there are also security benefits, given that progressively “less interaction with applications outside our virtual private cloud” is required as more of the environment becomes cloud-hosted.

Patel added that Macquarie is “the first financial institution to establish a Solace-based event platform running exclusively in public cloud”.