When Roy Southby tried to sell his colleagues at the Interior Health Authority on the need for a disaster recovery plan for their data systems, they felt the odds of a disaster were low. They changed their minds a year later, as the flames of the British Columbia’s forest fires rapidly approached
their doors.
When the authority, which covers a quarter of the province’s land area, was created in 2001 through the merger of several smaller authorities it was decided to centralize IT services in Kelowna, and use Meditech’s Health Care Information System across the region.
Southby, the IHA’s director of technology services, said he proposed using Kamloops as a backup data disaster site in case a major disaster happened in Kelowna and took the system offline.
The IHA hired Telus Corp. to install a Gigabit Ethernet connection to cover the 200 km between Kelowna and Kamloops, and selected Symmetrix Remote Data Facility (SRDF), a software package from Hopkinton, Mass.-based EMC Corp., to mirror the two sites.
The system is now up and running, with Symmetrix storage-area networking equipment in both locations. They are adding more servers in Kamloops, and Southby said in the event of a disaster or server problem in Kelowna, they would be up and running again quickly.
“”All we need to do in the case of a disaster is call Telus and re-route the IP addresses of the servers so that normal users are totally unaware that anything has happened,”” said Southby.
They don’t have all day to back up to tape drives
Frank Kolb, client solutions manager for EMC Canada, said with SRDF, the IHA has been able to build a solid infrastructure that requires few staff, gives them a known recovery time, and allows them to establish routines and procedures for the recovery of their data.
By contrast, he said, with tape backup, it can take 18 to 20 hours to recover a network, and tape loses five per cent of its data annually. Then there are costs for off-site rental of a storage location, and staffing costs.
The IHA had already purchased a Symmetrix SAN for its Kamloops facility. Officials decided to purchase another Symmetrix for Kelowna, and mirror the data between Kelowna and Kamloops so that if the servers in Kelowna went down, IT services to the region could be quickly restored with little loss of data. The IHA was in the planning and implementation stage when the forest fires broke out.
System failure could close down health system
The system is mirroring automatically every hour, and Southby said in the event of a disaster or server problem in Kelowna they would be up and running again quickly.
“”All we need to do in the case of a disaster is call Telus and re-route the IP addresses of the servers so that normal users are totally unaware that anything has happened,”” said Southby.
From an IT perspective, Southby said EMC’s ControlCenter has allowed them to automate the system entirely, leaving little work for his staff now that the system is up and running. However, they’re still continuing with daily tape backups. The tapes are useful as an incremental backup, in case someone overwrites data and then realizes a few days later they need that information restored.
Southby said he isn’t aware of many health authorities that have put similar systems in place, but backup is becoming more important as health care organizations rely more on IT.
“”Once you get to the size of a health authority like ours, if you lose your data centre when everyone is working off electronic health records and imaging systems you virtually can close down your health system,”” said Southby.
Kolb agrees that the advent of the electronic patient record and events like SARS in Toronto and the forest fires in BC are increasing awareness of the need for disaster recovery applications in the healthcare field.
Among the lessons learned from the terror attacks of Sept. 11, 2001, Kolb said it’s important to locate your secondary data centre at least 100 to 200 km away from your primary data centre, to minimize the effect from the loss of people and communications infrastructure. If your secondary data centre is located within the disaster zone, its utility in a disaster may be limited.