Managing the technology that runs Virgin Entertainment Group’s 10 Virgin Megastores in the U.S. is like staging a rock concert, says Robert Fort, CIO of the media company.
When each store opens its doors, the curtain effectively rises, and the systems-the cash registers, listening station kiosks, digital signage, data warehouse, converged voice and data network-need to perform smoothly and in sync, much like the lighting, pyrotechnics, band and dancers all need to be following the beat of the same drum.
To ensure that store systems don’t skip a beat, Fort says he began looking for tools he could use to proactively police them a few years ago. Fort joined Virgin Entertainment Group five years ago and became vice president of IT and CIO in 2004. Last year, he piloted Microsoft’s Operations Manager 2007, which monitors the performance of all devices on his network, including servers, cash registers, and kiosks.
Fort says Operations Manager has provided him with a tremendous amount of useful information at the device level, but it doesn’t provide him with visibility into individual, customer-facing business transactions. That becomes a problem when he gets an angry phone call from a store manager demanding why credit card transactions are taking so long to process.
So Virgin has supplemented its implementation of Operations Manager 2007 with transaction monitoring software from Inetco. This way, he can track the performance of hardware and software (using Operations Manager) as well as credit card transactions (using Inetco’s Insight software.)
Sound Check
Operations Manager 2007 tells Fort when hardware doesn’t have enough disk space, when communication lines are backed up and when there’s a problem with a particular router, switch or piece of software. He says Operations Manager has provided him with a tremendous amount of useful information at the device level. And as a result, he’s rolling out Operations Manager across the company.
But it doesn’t provide him with any visibility into individual business transactions. For example, he can’t tell how long it takes the cash registers to scan prices and ring up sales. Nor can he tell how long it takes to process credit and debit card transactions, which account for between 60 and 70 percent of in-store sales at Virgin Megastores in the U.S.
Yet he needs those metrics to understand what customers are experiencing in the stores. For instance, if point of sale systems are running slowly, lines could be forming. Customers waiting in line and at the register could be growing impatient. They could decide not to bother to wait-and thus not to bother purchasing those CDs and DVDs they had in their hands.
“Our product is available in other locations,” says Fort. “It’s important to us that customers come into our stores and have a pleasant experience the entire time, even through checkout.”
If they don’t, he adds, revenue and the reputation of the Virgin Megastore brand is at stake.
In a fiercely competitive retail environment where customers can buy music and DVDs from such leading retailers as Amazon.com, Best Buy and iTunes, Fort needs any technology he can get that will improve customers’ experience in the store. Though roughly 10,000 customers walk into the flagship Virgin Megastore in NYC’s Time Square every day, Virgin Megastores have been hammered by the rise of digital media and the subsequent decline of CD and DVD sales.
Since 2004, Virgin Entertainment Group has closed 14 Virgin Megastores in the U.S. Fort says this “rationalization” of the privately-held company’s portfolio of retail stores along with a shift in strategy to become a lifestyle brand has increased the company’s profitability and improved the performance of remaining stores.
At the same time that Virgin Entertainment Group was closing stores, Fort was forced to make cuts to his IT staff in 2005 to keep the company’s IT costs in line with business operations. The size of the IT group went from 16 people when he started in 2003 to its current level of 11 people today.
He doesn’t have the luxury of a team of network and store operations managers to monitor logs and systems. He needs software to do that job automatically and to only report on exceptions when the systems go out of whack.
“Retailers have a big incentive to improve the in-store experience because it costs them a lot more per transaction to enable than the online experience [costs],” says Nikki Baird, an analyst with Retail Systems Research Group. “The American Customer Satisfaction Index for the first time showed that consumers are happier with the online shopping experience than they are with the in-store shopping experience. Store comps have been flat year over year, yet online is growing like gangbusters, even in tough economic times.”
The Show Must Go On
In January 2008, Fort met representatives from Inetco, a provider of transaction monitoring software based in British Columbia, Canada, at the National Retail Federation tradeshow. He learned that Inetco’s technology diagnoses credit card processing problems, which he says the company occasionally experiences.
“When we have a problem [processing a credit card], I get a prompt and loud call from the store,” he says.
Diagnosing the cause of such problems is difficult. Fort and his staff can’t tell if the root cause is on their end (for example, if it’s a problem with a specific cash register or with the network) or on the credit card provider’s end. When a problem processing a credit card transaction occurs, a store manager can override the transaction so that the customer can get on his way with his purchase, but the override increases Virgin Entertainment Group’s financial risk.
“If we go into override mode, we risk not getting the proper authorization for the purchase and we get charged back,” says Fort. In other words, if they override the transaction and the bank later says it’s not going to pay for the purchase for whatever reason (e.g. the customer’s card wouldn’t have been authorized), Virgin Entertainment Group winds up paying for the customer’s purchase, and Fort and his IT systems are effectively responsible for incurring those costs and that risk.
Retail Systems’ Baird says retailers have historically thrown hardware at the problem-they’ve looked at increasing processor speeds when they’ve seen point of sale and credit card transactions slowing down. That’s a waste of time and money when the problem doesn’t have anything to do with the point of sale system but everything to do with the credit card provider.
The Opening Act
Fort deployed Inetco’s Insight product in April to monitor credit card transactions and to make sure customers weren’t being held up by slow ones. He says the implementation was fairly straightforward. Installing the software on a server took a few hours.
Next, Fort and his staff spent another couple of hours verifying that transaction data was flowing into the Inetco Insight software as the transaction data was moving from the point of sale servers to the credit card processor.
They also had to create standard data records for each transaction. These data records indicate the path the transaction is travelling-for instance, whether they’re inbound or outbound, the IP addresses where they’re going and coming from, and the return code from the credit card authorizer.
Fort and his IT staff worked with Inetco to create these records based on templates Inetco had created for standard credit card processing transactions. The templates essentially tell the software what to do, what exceptions to look for and when to send an alert. They interpret every transaction and break down each transaction into the appropriate data field on the record. The Insight software identifies failed authorizations based on the return codes.
Within days of deploying the software, Inetco Insight began identifying slow transactions.
“As we were first running this, we noticed a serious, continual blip of transactions running close to three seconds,” says Fort. Most credit card transactions take anywhere from four-tenths of a second to one-and-a-half seconds to process, he says, adding that the optimum time is between one half and one second.
The software pinpointed that these long processing times were coming from one particular cash register-information they never would have had before deploying Insight. Knowing a single cash register was the source of these slow transactions, Fort’s team began trouble-shooting.
They made sure the point-of-sale system software was loaded properly on the machine. They double checked communication lines. Though they haven’t yet determined why certain transactions on this particular cash register take as long as three seconds, they continue to monitor the situation, which fortunately, Fort says, is not a problem at the moment.
Since discovering this first anomaly, Fort and his team have set up another alert within the Insight software so that if transactions on any register exceed 1.5 seconds, the network administrator gets an e-mail notification and the alert shows up on the Inetco Insight dashboard on the network admin’s computer.
Fort says he’s also going to use the Insight software to do a trend analysis by credit card type and by store. “We want to make sure we’re not having performance problems with any one vendor,” he says. “We also want to understand if one store has better performance than another and why that is.”
Encore
Even though the software has only been running for about a month and the alerts for just three weeks, Fort is seeing several other ways he can use this transaction monitoring technology.
He’s currently working with the loss prevention department to set up notifications when credit card charges are reversed, when transactions exceed a certain amount of money and when the same credit card is used multiple times a day to identify fraudulent activity and to make sure that store employees follow proper procedures for returns and charge reversals.
Fort also anticipates that this software will help him and his staff be more attuned to activity that could indicate a potential data breach. Because the Insight software gives him benchmarks of standard performance for transactions, if he starts seeing a change in those benchmarks, he knows he needs to take a closer look.
“I have this aggregated information and I have all these details I can dive into,” he says. “If I start spotting an extra hop in communication or a new IP address, I know we have an anomaly in our system.”
In short, he says, the software can tell him if the lug nuts are coming off the wheels before he has a blowout crash. (Fort, who attended the Indianapolis 500 over Memorial Day weekend, is a drag racing fan.)
In addition, Fort thinks Insight can help monitor the performance of the enterprise service bus, a critical component of service oriented architecture.
“As more organizations move toward SOA and put more emphasis on the enterprise bus, there are going to be performance problems and a lot of finger-pointing,” says Fort. “We’re going to need diagnostic tools to monitor specific transactions end to end and at a high level of detail.”
And he’s looking for more opportunities to use Insight, including monitoring purchases of in-store kiosks. “We can look at other transactions in systems we suspect we have issues on. They don’t have to be credit transactions,” he says.
Finale
It’s too early for Fort to quantify a hard ROI on the software, but it has already delivered on its promise to identify problems with credit card transactions before customers notice them and before Fort gets an angry phone call from the store.
“We will be avoiding outages, disgruntled customers and negative experiences,” he says. “My ultimate job, my first responsibility is to keep existing systems up and running, and I need to do that quietly and effectively.”
What’s more, Fort has been able to enhance his monitoring capabilities without having to add staff.
“With the introduction of tools like Microsoft Operations Manager and Inetco Insight, we’re achieving more in the monitoring and administration of systems now than we did before when we had more headcount,” he says.
Fort adds the Inetco software does what his staff could never do on their own or even collectively.
“With all the activities we have, we couldn’t look at all the logs. Even if we could, we’d be looking at them historically and not seeing things as they’re happening. This tool persistently and consistently monitors the conditions we feel are worthy of alert.”
