Although RBC Financial Group finally outlined details of the technology failure that for days kept money out of millions of Canadian bank accounts,
a marketing expert warned the bank has only days to do a better job of smoothing things over with customers.
The bank said it will determine both the cost impact to its business and possibility of employee dismissals once it reviews the cause of what’s now being called a human error and the steps taken to clean up the mess.
IBM has been asked to conduct an independent review to find these answers, but for reasons of contractual confidentiality, a company spokeswoman said she could not provide specific details about how it will help RBC avoid a similar situation or case study examples of how its business continuity and recovery program has assisted other clients. RBC spokespeople refused comment, directing media enquiries to information posted on its Web site late Wednesday.
For the most part, data recovery experts like Ron Austin, vice-president of business development at ActionFront Data Recovery Labs in Markham, Ont., believed the “”situation was handled quite well internally.
“”Typically, what really compounds any IT problem is panicking by system administrators and the database admistrators and the people that are on the front lines to keep things running and address problems,”” he said. “”It may have taken them a couple of days to do it simply because they weren’t panicking. They were taking a step-by-step approach and being quite conservative.””
On its Web site, RBC explained a mistake was made in a software programming change that was corrected within two hours. But correcting customers’ bank account balances took longer because the bank had to stop production until it made sure the error couldn’t harm other systems. Once production re-started, RBC was held up again as it was forced to process two days of banking transactions on the same date.
Adding to the backlog was the requirement that all transaction dates be manually checked. RBC said it decided to give priority to payroll transactions.
The bank said it didn’t resort to its backup system once the problem was discovered because this secondary facility functions only if the primary system isn’t working. Since RBC policy dictates that program changes be made at the same time to both systems, its back-up facility would not have resolved the problem. In the future, Canada’s largest bank said it may make its backup facilities “”more robust.””
RBC’s practice of making simultaneous programming upgrades on its main and secondary systems is “”a little unusual or it may be unusual in my experience, simply because if you start an upgrade on the main system, often you want to have a backup that’s untouched so that you can revert to it,”” said Austin. “”When they say ‘more robust,’ I assume maybe they’re going to add another layer of back-up . . . that’s always left alone.””
Bill Margeson, general manager of CBL Data Recovery in Markham, said it would have been difficult to foresee problems occurring on both systems. Describing putting in a second backup system as “”a little bit of overkill,”” Margeson said RBC will probably be forced to go down this path.
However, it’s “”only a possible two-hour gap that caused a whole week’s worth of problems,”” explained Margeson. “”Even if they had a third backup facility running their old program, and they noticed an issue with this X number of hours into their processes, they’d still have to go back and re-do things as they were.””
Margeson lauded RBC’s decisions to bring in IBM to investigate and to discuss issues around processing financial information with other Canadian banks that perhaps will prompt changes to the policies of other institutions.
As for RBC’s reputation among Canadians, York University marketing professor Alan Middleton said the integrity of the bank’s brand depends on the assurances it provides and actions it takes over the next week to allay people’s fears about the safety of their money.
Middleton said “”nobody’s going to read”” news updates placed on RBC’s Web site explaining the root of the problem and the safeguards put in place to prevent a re-occurrence. He said RBC has to talk to customers via onlike banking, telephone banking and advertising.
“”I’m a customer and I’ve received (no direct communication from RBC),”” said Middleton. “”I shouldn’t have to contact them to find out what I pay them to do is okay. And that’s going to be most customers’ views.
“”So I’m going to watch it for about a week, and then I’m going to get pissed off.””
RBC’s public relations problems continued late Wednesday and Thursday when an e-mail phishing scheme went out to many RBC customers urging them to enter their client card number in order to access their accounts.
Comment: [email protected]
