RBS IT failure: A result of creaking mainframes, offshoring or poor governance?

RBS IT failure: A result of creaking mainframes, offshoring or poor governance?

One of the most publicised IT failures this year could be a taste of things to come in banking

Article comments

It has been two weeks since Royal Bank of Scotland (RBS), and its subsidiaries’, started reporting catastrophic IT problems that resulted in some 17 million customers not having access to their finances for days on end.

Although the bank is giving away little until it has completed a root casue analysis, the initial consensus pointed to an upgrade to RBS’ batch processing software, CA 7 from CA Technologies, as the cause of the problems, with a junior RBS IT worker based in India make a major mistake by deleting all the scheduling that was in place. This meant that the information had to be re-inputted into the system, causing a significant backlog.

RBS subsequently said in a letter to the House of Commons Treasury Select Committee that the error was made by Edinburgh-based staff.

Where ever the error occurred, RBS’ chief executives while considering the reputational damage and compensation that will inevitably cost the group millions of pounds, should also listen to the IT industry which has begun to debate the broader issues and ask how such a mistake could occur in the first place? Opinion is divided.

Alex Kwiatkowski, research manager for IDC Financial Insights, argues that the banking giant’s creaking mainframe infrastructure has a role to play, which is easily disrupted when coupled with inadequate IT skills.

“The complexity of banks’ mainframes is made up of a combination of in-house developed software, vendor supplied software that has been heavily customised and is running processes in ways and means that it was never originally designed for,” says Kwiatkowski.

“However, this is banking IT infrastructure the world over – it is by no means a problem isolated to RBS. They are now reaching a point where it doesn’t take much to have a disruption to the system, it is starting to creak.”

He adds: “It is effectively a ticking time bomb, which doesn’t take very much for somebody to short circuit the wiring and for it to all to blow up in their face.”

Kwiatkowski’s point about the state of banking IT infrastruture applies to many more organisations than RBS. It also leads to a wider discussion around skills. He describes the upgrade to CA’s software as ‘simply routine’, which suggests that if something was to go wrong, there should have been staff in place, with the correct knowledge, to manage the fallout.

Where is accountability going to lie?

Ovum’s financial services technology practice leader, Daniel Mayo, points to the staff member that was managing the ‘routine upgrade’ as a significant contributing factor. He states that although the upgrade was in fact routine, reprogramming the scheduling order is complex, and he believes required skills ‘well above the pay grade of the guy who was actually working’.

“The actual core system is reliable, the issue is that it’s based on quite old technologies. Because there is pressure to cut costs, there are lots of senior staff retiring or taking redundancies, and the new IT staff coming in don’t have the experience of these old IT systems to manage problems,” he said.

“When something goes wrong like this, there is probably a relatively small pool of people in the organisation that knows what to do. The people working would have understood how to do the upgrade itself, but wouldn’t have known about the scheduling order.”

In the past 12 months RBS has offshored hundreds of jobs to India in a bid to cut costs, which has led to speculation that if it had maintained a highly skilled workforce in the UK, albeit at a higher cost, it would not have suffered such a share price hammering, brand-damaging and costly failure. RBS raised a few critical eyebrows when the move too place, and has now been crucified for the decision, as it is likely that compensation and peripheral financial damage is likely to be far greater than the money saved from offshoring.

Financial services think-tank Balatro’s chief executive, Chris Skinner, however, is adamant that offshoring as a principle cannot be blamed. He argues that if you outsource or offshore functions, these operations need to be considered as if they were a UK based employee of your company’s, where effective management, governance and SLAs should limit any widespread fallout from IT mishaps.

“Where is the accountability going to lie? If you have an outsourced partner, it is the same as having an employee. You should have a vetting process that is effective, so you can’t suddenly turn to the partner and say you screwed up, because even if they did screw up, you should have internal processes to manage that,” says Skinner.

He adds: “You should have disaster recovery and contingency plans in place for these eventualities, and that is what all of us are stunned by. RBS didn’t have the competent processes in place to roll back the migration.”

Skinner questions why RBS was carrying out the upgrade on a Tuesday, rather than, for example, on the Diamond Jubilee weekend, where it would have had four days. He says: “Why did they end up with a corrupted core file without a contingency plan to roll back to a mirror parallel processing system that’s still operational?”

“I don’t think offshoring is a problem. When you outsource to a partner you need to ask yourself – do you have the right disciplines, SLAs and internal due diligence in place to ensure that it will work. If you don’t manage and govern it correctly you will have issues.”

First tremor of a technological earthquake

IDC’s Kwiatkowski is keen to highlight that although inadequate governance and skills played a part, there is also a wider issue within the global industry that these banks are operating on systems that were built decades ago and aren’t sufficient for a modern world. He describes the banks’ systems as their ‘major organs’ that need overhauling, but projects to do so have been side-lined by turmoil in the macro-economic environment.

Kwiatkowski says: “There will be a conversation that has to take place in the next 18 months, where banks and CIOs will be saying – if we need to be modern banks, to be more transparent, we need systems that weren’t built decades ago. It’s like trying to do digital banking on an analogue system.”

“But how do you make this happen? You have to examine the systems that are at the very core of your business. Replacing them will cost you tens, if not hundreds of millions of pounds. Unfortunately, doing nothing with these systems is no longer a strategy – every major UK bank has an enormous obstacle to overcome.”

Kwiatkowski believes that this overhaul will be difficult for the banks and will require outspoken executives to make it happen. However, he argues that if it doesn’t happen, this won’t be the last IT banking failure we see in coming months.

“The blame has to ultimately lie with the CEO, but it’s also how he is being advised. It takes a brave person that puts their head on the line and says – we have got a problem and it we don’t do something about it, it could be a catastrophic failure,” he says.

“RBS’ failure is the first tremor of a technological earthquake that will start to hit some of the major banks. Keeping the lid on it difficult and it doesn’t take very much for there to be a major problem.”



  • Lauraa Boosey Outsourcing to india is a good idea because more people are able to get jobs out there unlike in the UK Outsourcing also allows language barriers to no longer be a problem but this guy is a major problem in outsourcing He shouldnt of touched what he didnt know about and therefore i hope he loses his job
  • Liz Hannaby The mainframe is much more secure better managed and reliable than midrange systems and the expectations are much higher If the mainframe went down there would be a major uproar and if there were no contingencies in place heads would roll but with midrange the expectations are low and processes around it not taken seriously And typically until you get a failure which costs a company money this will be the way forward
  • Korgesquire How do clowns like qwiatkowski get jobs Presumably he thinks Banks are running system 370s instead of Zseries machines And thats completely ignoring the fact that the failing product was a scheduler for Unix and Windows
  • Korgesquire Autosys upgrade not ca7 Backout deleted stuff Rbs running an unsopprted version All the rest about offshoring and this entire article is speculative rentagob crap
  • Fluffy Mergatryod Anyone who imagines that outsourcing to India wasnt the problem is living in a dream world The culture there allows even encourages corner-cutting chalta hai and many of the workers have mediocre to poor qualifications regardless of what it might say on their cv Degrees are often either fake or slipshod The entire selling of India as an IT powerhouse was a PR campaign by Nasscom with little or no substance behind it Wake up and smell the chai Bring IT back home stop offshore outsourcing be very scant and demanding before awarding a guest worker visa or enjoy more debacles like this coming up fast
Send to a friend

Email this article to a friend or colleague:

PLEASE NOTE: Your name is used only to let the recipient know who sent the story, and in case of transmission error. Both your name and the recipient's name and address will not be used for any other purpose.

We use cookies to provide you with a better experience. If you continue to use this site, we'll assume you're happy with this. Alternatively, click here to find out how to manage these cookies

hide cookie message
* *