JP Morgan has reduced the complexity of its derivative trade processing systems after swapping relational databases for a NoSQL system, allowing it to handle a wider variety of data and scale to meet growing demands.
The US banking giant processes hundreds of thousands of transactions within its derivatives business - which involves more complicated financial instruments that are less suited to the standard tables of relational databases - and settles billions or even trillions worth of trades each day.
In order to reduce the strain on the its systems and bring together disparate databases targeting specific functions, the bank partnered with MarkLogic to deploy its NoSQL database technology.
“Previously we had pretty much infrastructure per product type - within derivatives there are lots and lots of products, so we had a very expensive, very fragmented infrastructure. My job was to bring that together into a single infrastructure,” said Keith Pritchard, chief technology officer for the firm’s derivatives and FX business, at Gartner's EI &MDM Summit in London.
“We were very interested in some of the things NoSQL offered us in terms of horizontal scalability, the ability to just add more and not have do a big infrastructure replacement.”
Flexibility and scalability
NoSQL differs from more traditional SQL relational database systems, allowing businesses to analyse large volumes of unstructured data much more easily. One of the reasons JPMorgan deployed the technology was the flexibility of the database, which enabled it to deal with a wide variety of trade data.
“From a data perspective, a lot of the instruments we deal with, the derivatives, are very flexible,” he said.
“Especially in the OTC markets, traders can pretty much put together what they want, they bring together features to a trade, so we have very variable data coming in.
“We also have very variable data needs coming out. Especially at the moment, the regulators are very interested in what the banks are doing, and we get multiple different questions every day almost.
“So we don’t know the data structures coming in and we don’t know the usage patterns, so we needed to get away from the relational [technology], which in my experience of 25 years in financial services tended to lock you in to a particular pattern.”
However, Pritchard warned that the process of moving to non-relational databases created challenges, requiring a shift in mindset from developer staff.
“The biggest lesson I have learnt is to not underestimate the change we need in terms of skillset - or at least thought process and approach to the development community,” he said.
“Going from being a company where most people spent their entire careers using relational databases - and who therefore had a mental model of relational data and how to navigate data - to NoSQL structure, we then ended up creating problems for ourselves.
"We got caught half way and had very inefficient queries because we were trying to do things with a NoSQL database that we shouldn’t be doing. So with hindsight I would have thought more about the organisational preparedness.”
Detecting market manipulation
The bank has now been live with NoSQL technology for several years, but Pritchard said that there is more the company can do with the technology.
“The next big evolution for us, we still have quite a bit to do to get the breadth of product set and locations onto the platform. But my next challenge is to get the organisation to start using the data differently," he said.
“We are really using it for the transactional processing at the moment, we are not using some of the power in, for example, semantic capabilities where a lot of what we have to do is look for trading patterns. So this may mean spotting market manipulation, if a trader is shorting the market to drive the price down..
“Those sorts of things we can do with technology, and as I am bringing all of the products together we have got the capability. That is the next direction we want to take it in.”