Wall Street regulator the Financial Industry Regulatory Authority (Finra) has moved 90 percent of its data to the cloud with Amazon Web Services (AWS) in an unprecedented migration for the normally conservative financial services industry.
Following the flash crash in 2010 the US Securities and Exchange Commission (SEC) started working with Finra on a controversial plan to monitor and analyse trading activity more closely. Finra is subsequently building one of the biggest databases on the planet.
Speaking at AWS re:Invent in Las Vegas this week, Steve Randich, CIO at Finra said the watchdog will store and process information relating to “all the exchanges and financial services firms that do securities trading”. This includes the 12 exchanges such as Nasdaq and the New York Stock Exchange, and 3,700 broker dealers including the likes of Goldman Sachs and Merrill Lynch.
Finra then examines these market events to create a picture of the whole market. Its teams of analysts and data scientists analyse the data to look for market manipulation, such as insider trading, layering, cross product manipulation, front running and increasingly innovative fraudulent activities.
Finra began their migration from appliance storage in their own data centres to AWS at the beginning of 2014 and completed phase one in July 2016.
The data coming in peaks at 75 billion events per day, which is more than Visa and MasterCard process in six months, according to Randich. Finra now stores more than 20 petabytes of data, predominantly in AWS S3 and Glacier. It spins up 30,000 EC2 nodes per day at peak times, with 50 percent having a life cycle of under two hours to do required processing.
The migration to AWS wasn’t a 'lift and shift' process, but what Randich called a “complete redesign of the stack” to work in a cloud-first way. This meant hiring people with cloud skills, retraining staff to work along a devops model and embracing cutting edge open source technologies such as streaming analytics and machine learning using products like Presto, HBase and Hive.
Randich said that when they embarked on this project “we had streams of proprietary database vendors coming in telling us it wouldn’t scale, wasn’t secure, it won’t work and we have proven them all wrong”.
Finra plans to move the remaining 10 percent of its data from relational databases and smaller applications into the cloud by 2018, at which point it will close down its last data centre operations.
The benefits for Finra run along the typical lines when it comes to cloud migration: performance and cost savings as its EC2 usage expands and contracts along with an increasingly volatile market.
Having all the data in one place means analysts get faster access to data and the analytics tools they need to spot fraudulent activity patterns, with the average query taking seconds to complete instead of minutes or even days to cobble together from various sources before.
Responding to security concerns with the public cloud, Finra is bullish on cloud security being more robust than on-premise. “Back in 2015 we determined cyber security is better in the cloud than it is in private data centres, which bucked conventional wisdom,” Randich said. During a deep dive session Tigris Khrimian, VP data platforms at Finra said that cloud offers “superior data protection as everything is encrypted at-rest and in-flight.”
Finra has also commercialised its newfound AWS expertise, offering a consulting service for fellow financial services firms. Randich said: “We have dozens of companies from other regulators to big banks coming in to learn from us and our experience, we have taken this to the point here we have commercial relationships with a number of firms to provide that assistance."
Find your next job with computerworld UK jobs