A brand intelligence company has rolled out a database with 8TB of data on common hardware, which traditionally would have required a mainframe.
Brandimensions is a Canadian company that uses a massive Web spider to scan the Internet – to the tune of about one billion Web pages per month. It offers brand protection services to large enterprises by searching for Internet content of interest – if the enterprise’s logo or intellectual property is mentioned, for example.
“The whole idea is being able to protect the reputation of a company,” said Hugh Hyndman, chief technology officer with Toronto-based Brandimensions, adding that it’s looking for sites that are fraudulent or misrepresent a company.
Brandimensions also has a brand intelligence business that shares a similar technology platform. In that case, it’s more interested in what consumers are saying about products, services and companies, so marketing teams can “hear the voice of the people.”
The company’s database has grown in size to 8TB of data, and it continues to grow on a daily basis. So Brandimensions worked with Toronto-based Affinity Systems, a Microsoft partner, to come up with a solution. It made the decision to upgrade to Microsoft SQL Server 2005 and is now able to store more than 70 billion links mapped for nine billion URLs and 240 million Web sites in its database.
“When we first started working with Brandimensions, we looked at the possibility of using Oracle,” said Leslie Goldsmith, director of Affinity Systems. Oracle and Microsoft were obvious contenders for storing the massive amount of data that Brandimensions wanted to accumulate, he said. Affinity was familiar with SQL Server and had success with it in previous large rollouts – but it had never worked on anything as large as this before.
“It’s absolutely massive with the largest table holding 70 billion odd rows,” he said. “Oracle could provide absolutely no guarantee that it could handle the size of the database. We did some additional load testing with SQL Server and we decided to go down that path.”
Thanks to the upgrade, Affinity has been able to improve the performance of the database and provide more flexibility in configurations, which saves development time.
“Not only were we upgrading the database engine to the latest engine, we were getting a whole new set of hardware end to end to deal with higher volumes,” said Hyndman. “Affinity made some significant software changes to cope with volumes and be able to exploit some of the performance features available in 2005.”
Brandimensions is doing somewhere between 7,000 and 10,000 transactions a second, which is extremely high, said David McJannet, senior product manager for SQL Server with Microsoft Canada. “That’s similar in scale to the market data dissemination system at the NASDAQ, which is also built on SQL.”
When it comes to systems of that scale, Brandimensions surpasses – by a long shot – the needs of a typical enterprise, he added. However, we’re seeing the scale of data to be managed growing exponentially. At the same time, we’re seeing processing requirements increase.
“Hardware has evolved to the point where companies that were previously running this high-volume transaction system on a mainframe system are now able to address those similar high-volume requirements on much more common hardware,” said McJannet.
For Brandimensions, this is just the beginning. There are new types of applications the company wants to build as it spiders the Internet, such as analyzing images.
“The whole idea is to deal with counterfeit and fraud on the Internet,” said Hyndman. “Since we’re visiting all these Web sites, we figure there’s a lot of other services that we can offer on top of that.”
Comment: [email protected]
