Share, one of the USA's largest online property search engines for buyers and renters, has deployed Splunk to not only improve operations but also create products that drive revenue for the business.

One of these key products is an engine that delivers personalised recommendations to anyone searching for a property in a specific area within a certain price range.

ComputerworldUK spoke to Robert Osborne,'s director of systems and application development, at Splunk's annual user conference in Las Vegas this week. Splunk's technology helps companies improve their operational efficiency by reading the data logs created by machines. began using Splunk just over a year ago, initially as a cloud service to help the company cope with rapid growth in machine data. uses NetApp in the back end for storage, is fully virtualised on VMware, operates on Linux, and its web servers are load balanced behind an F5. However, 40 percent year-on-year growth in page views resulted in a creaking infrastructure.

Bursting to the cloud

“We were saturating parts of our infrastructure and we needed to figure out what was starting to saturate so that we could figure out a plan to fix that. When you are dealing with so many moving parts in your architecture it's hard to pinpoint what the problem is,” explains Osborne.

“Is it the virtual layer? Is it the operating system layer? Splunk helps us pinpoint those saturation points.”

Luckily,'s infrastructure is hosted in a data centre in Virginia where Amazon has an availability zone. This allows to run a low latency fibre link directly into Amazon's cage.

“From there we can easily burst traffic to the cloud, so we can use Splunk to say we are hitting some saturation points and let's direct 30 percent of our traffic to Amazon's cloud,” says Osborne.

“We are doing this sort of thing with Splunk. The other part is that when we are bursting to the cloud, we are also using Splunk to monitor those virtual machines.”

Getting buy-in from the CIO

However, when Osborne asked the CIO to purchase Splunk for enterprise-wide use he faced some resistance. The company's CIO agreed that Splunk is great for operational efficiency, but he needed examples of how it could also be used to drive revenues.

As a result, Osborne and his team went away and drew up some ideas about how the logs can be used to create new products for

“It allows us to spin up some applications that are very feature rich, but also do it in a very fast timeframe. The first project that we did in Splunk, which will be released in late Q4, is that we are going to tie in a recommendation engine into our map search,” Osborne explains.

“We have all this access log data that we use for operational purposes, which we are going to repurpose for product value. A recommendation engine makes sense because we know exactly how engaged a user is with a property.” can measure how attractive a property is by the looking at certain factors, such as the number of photos viewed, whether the property has been printed and whether the buyer has contacted the estate agent.

“What we can do is aggregate all that data and find out which of the properties are most engaging. Once you have the most engaging properties, from there you can recommend those properties when people are searching in a particular area within a certain price range,” says Osborne.

A quick build time

In order to get the recommendation engine built, hired a programmer. Osborne was keen to highlight that building products with Splunk that create additional revenue streams for the business don't have to be long, complex projects.

“Our new programmer was not very familiar with MongoDB, was not familiar at all with Splunk, was only vaguely familiar with node.js – but he still managed to build the recommendation engine in three weeks. If it had been a senior developer it could have been done in three days,” he says.

“What Splunk allows you to do is quickly extract the data that you want. It was easy to write a query to get the results set back, have node massage the data and figure out some mathematical values behind the engagement, and then just populate that into MongoDB. You then just keep updating that Mongo record with the fresh Splunk data you are receiving.”

He adds: “We have a job that runs on Splunk once a minute, ships that data off to Mongo via node.js, which is just constantly updated.”

Future projects

Because the recommendation engine was such a success, is now using Splunk data to build other revenue-driving products. For example, it is using the data to identify migration patterns, where buyers are shifting their buying behaviour to new areas. It is then using this information to sell products to estate agents that can use it for marketing activity.

It is also using the Splunk data to improve its search engine. Buyers typing in the name of a city had previously been delivered results in alphabetical order, but is now using Splunk to identify the most popular city searches and drive these to the top of the results for buyers.

While Osborne says that it is “tricky” to put a return on investment figure on purchasing a tool like Splunk, he has no doubt about its value to the business.

“We haven't tied a dollar value to it, I'm not sure that we will. However, the value is clear - we went from taking a day to do cause analysis to it taking 80 to 90 percent less time,” he says.

“We haven't written it all down or put a sum of money to it, we haven't gone that far, but engagement is going to increase with things like the recommendation engine. We know the value is there.”

Join ComputerworldUK at the Open Group London 2013 Conference

London 21 -24 October

Get to grips with Enterprise & Business Architecture, Global Supply Chain, Risk Management and Information Security
Professional Development as well as  Open Platform 3.0™ – the convergence of Cloud, Social Media, Mobile Computing and Big Data

Book here using promotion code LON13-CWORLD for a substantial readers’ discount.