What really happened when the RBS payment system failed

At the start of this week, we had a
discussion at the Financial Services Club about the issues at the Royal Bank of
Scotland (RBS) over the summer when their payments systems failed.

It was a major failure that garnered
headlines worldwide, with many banks saying: “there, but for the grace of God,
go I”.

Why such a furor occurred is pretty
obvious: when a core payments system goes down, people cannot pay for the
things they need, companies cannot function, commerce grinds to a halt and
economies fail.

Pretty obvious really, and it’s the reason
why banks with critical transaction processing operations in any country are
‘too big to fail’.

Joining the debate were:

• Ralph Silva, Director and Broadcast Analyst with SRN;
• Chris Dunne, Strategy Director with Vocalink;
• Roy Vella, former Director for Mobile Payments with the Royal Bank of Scotland; and
• Anand Vyas, Vice President, Sales for UK and Europe with Thinksoft.

(see end of this blog entry for their profiles)

Ralph opened the debate with an interesting
take on what banking’s all about.  
Asking the audience to tell him what banks do, most responded with
comments like ‘provide credit’, ‘process transactions’, ‘move money’, ‘support
business’ and more.

In typical Ralph style – I should know, I
used to work with him! – he dismissed all of these comments and said, “no, what
banks provide is emotion”.

Very true – many negative.

Actually, not that.  Banks, as Ralph clearly articulates, are
there to help us live our lives.  They
provide a promise that our money is safe and, when we are trying to improve our
lives by buying a house or a car, they help us to fulfill oru dreams.

They don’t provide any physical products,
so their function is to process promises of the fulfillment of emotions.

And that’s why the RBS payments failure was
such a critical issue for all of their customers: because they could no longer fulfill
the basic requirement of processing the promises of payment.

This was an interesting take on things, as
most surveys say that consumers will switch accounts due to poor service,
errors on their account or due to better interest rates (only four percent
would switch due to the banks morals and ethics apparently), so the hygiene
factor for a bank must be processing payments.

But what actually happened?

Chris Dunne answered this question by
pointing to Stephen Hester’s letters to the Treasury Select Committee (TSC).

Stephen Hester, CEO of RBS, wrote two
letters to Andrew Tyrie, Chair of the TSC,
explaining what happened and what the bank was going to do about it.

The first letter on 29^th June - the failure occurred on 19^th June 2012 – explained some of the
background and issues and what the bank was doing about it; the second letter
on 6^th July  gave specific responses to parliamentary
questions.

As Chris put it, the basic issue was that
payment are processed normally in a similar way to traffic on a highway.

The cars travel bumper to bumper (or fender
to fender, if you prefer) and all works fine until one car brakes suddenly and
has a fender bender (or bumper jumper, if you prefer).

Then everything backs up.

That’s fine and easy to sort out.

You send in a fire engine and cutting crew
and sort out the mess and get the traffic moving again.

The problem RBS had is that the accident
occurred in heavy fog, rain, snow, ice … 
you name it.

The weather was massively inclement and
nothing was easy to sort out, so all they could do is get the traffic moving
around the accident whilst the forensic fire engine service and CSI cutting
crew moved in to work what had happened.

[Ed: for the detailed analysis of what
happened, read Hester’s letters]

Roy Vella disagreed, and said that the
issue was more down to bad management of systems, too much legacy
infrastructure, an inability to adapt and adopt modern capabilities like cloud
and internet technologies, and a refusal to break old systems just because they
aren’t broken.

Legacy systems are an issue, as I’ve
blogged before,
and they are becoming more of one.  The
basic mantra of if it ain’t broke, leave
it, is becoming turned on its head as the view today should be that if it’s older than five years, replace it.

Anand Vyas agreed but turned the subject
towards testing and the fact that a bank needs to adequately test and ensure
their systems are risk aware.  The flow
of systems will be broken if the systems are not robustly and rigorously
stressed before change and migration takes place.

In this case, the feeling was that the RBS
change was possibly felt to be a trivial upgrade.  If the upgrade or change is significant, of course
the bank will test and test but, if it’s just a minor alteration, no-one cares.

This may be why the CA-7 upgrade took place
without adequate analysis and stress testing beforehand, and then the corrupt
files caused by the upgrade became the major car crash wreck we saw in the
summer.

Whatever your view, the fact that banks are seeing more and more technology failures just goes to show that the financial system is running on technologies that are
no longer fit for purpose and need more  testing
to be robust.

You can watch the debate through these 15-minute
clips kindly supplied by Ralph Silva.

Part
One

Part
Two

Part
Three

 

Biographies of the Speakers:

Ralph Silva is a director and broadcast
analyst working with SRN out of the company's London office where he covers the
innovation and emerging trends affecting Fortune 1000 companies. He has over 23
years of global experience beginning with 12 years as an investment banker in
San Francisco managing the IPO process for financial and IT companies. He then
moved on to become one of Gartner's most prominent analysts focused on the
financial services industry before he moving to become the first full time
analyst for TowerGroup in Europe. He now runs his own business providing
content for broadcasters andmultinational marketing departments.

Chris
Dunne joined VocaLink in 2004 as part of
the implementation of the £100 million Bacs technology renewal programme. He
then moved to be Director of Customer Services, in charge of customer contact
centres and service management across the entire range of services delivered by
VocaLink, including the Bacs service, Faster Payments, LINK and ATM managed
services. Having completed a successful programme of customer service improvement
and cost efficiencies, Chris is now responsible for business strategy for the
core Global Transaction Services business.

Roy
Vella is a mobile payments expert and
the Non-Executive Director at txt2buy Ltd, a new universal mobile sales channel
enabling consumers to purchase instantly from any offline or online
advertising. Previously, Roy has worked with many respected names in the mobile
and payments industry including Visa, Vodafone, RBS and PayPal with experience
spanning across 20+ years.

Anand
Vyas is Vice President of Sales for
the UK and Europe for Thinksoft, a software testing company. Anand joined
Thinksoft in May 2010 having previously held senior positions with Birlasoft
and HCL Technologies.

 

This Financial Services Club was sponsored by:

 

For details of sponsorship email us.