Businesses have long been in search of ways to improve the management and use of the information they hold. Celona Technologies’ Tony Sceales considers the consequences of increased integration and the application of master data management solutions, and asks whether enterprises still need to migrate data?

For a very long time the received wisdom has been that if only enterprises could gain a 360° view of the information they hold – provided that information is also accurate and up-to-date – they could revolutionise the way they do business.

Their customers will be happier, their businesses will run more efficiently, new revenue opportunities will be discovered, new product designs will be improved and campaigns will be better targeted and managed. The problem is finding the road that leads to this converged simplicity.

Today’s IT landscape is complex and it seems as if whatever is done to simplify it, it just keeps getting more complicated and more integrated. And, while we know that IT proliferation and complexity means that operational costs rise, there is always a reason to buy more, better or faster technology.

According to Bloor Research, the average large enterprise implements 4.5 major applications each year. And that’s just the big ones. Add to that all the applications we’ve inherited plus the smaller applications we’re implementing, plus the applications that are self-developed and could even be outside IT’s control, and the total rockets. Now add in the effects of merger and acquisition (M&A) which creates a moving pavement for many enterprises so far as IT rationalisation is concerned.

For all these reasons the proliferation of solutions and the data silos they use shows little sign of abating. Yet the requirement to tie these together so that the enterprise can function effectively is also strong.

In fact, ‘integration’ is arguably one of the most over-used words in IT: we integrate solutions, we integrate channels, we integrate data, we integrate within the enterprise and we integrate with the outside world. We are often found discussing the merits of new ways of integrating – like Web services – or finding new things that need integrating (as with master data management). Our appetite for it is undiminished.

Yet the fallout from increased use of integration is complexity, interdependence and higher operational costs. This has resulted in a situation where most of our IT budget is spent simply maintaining legacy, with less and less money available for innovation (See The Burden of Legacy by Toby Sucharov and Philip Rice). IT may wish to reduce the number of redundant or duplicated applications and databases, but often the rate at which applications are being switched on exceeds the speed at which IT can turn them off.

Quite often IT resorts to making a tactical choice to integrate, rather than to migrate or consolidate applications and data. The reason for such a choice is that migrating applications has historically been difficult, time-consuming and risky. Integration has offered an easier, cheaper, faster and less intrusive method of achieving corporate goals – at least in the short term.

The drivers to achieve high-quality, complete, consistent and consolidated datasets are frequently the desire to improve processes and performance (stimulated for example by a CRM, BI or ERP initiative), the need to comply with new regulation (eg SOX) or the wish to implement service-oriented architectures (SOA) or software as a service (SaaS).

Often it’s only part-way through one of these initiatives that enterprises realise just how challenging their data issues are.

Recently, attention has focused particularly on the improvement of so-called master data, which comprises some of the most valuable data enterprises hold: information about people (customers, employees etc), assets, products and places (office locations, geographic divisions and so on).

There is little new under heaven and earth, and so master data management (MDM) is an extension of previous concepts such as customer data integration (CDI) and product information management (PIM).

Managing master data is challenging for many large enterprises, and it is questionable whether simply integrating datasets is the best way forward. Take, for example, the situation that arises after a round of M&A activity. The enterprise acquires a complete set of master databases from the acquired company, each of which has dependent applications.

The complexity combined with the requirement for day-to-day operations to continue undisrupted means that IT staff decide that they will leave the master databases physically separate but tie them together using a reconciliation process. Over time, however, and as the number of master databases and their dependent systems increase, the reconciliation process becomes more and more complex, unmanageable and unreliable.

Marketeers would have us believe that master data management technology can solve our data problems through the use of sophisticated technology. By plugging an MDM solution into existing applications, such as CRM, ERP, billing, inventory and logistics, vendors argue that you can build a true, consistent view of data which can them be fed back into these systems.

However, the truth is that master data management will not solve our data problems overnight, involves far more than just technology alone, and is not simply a matter of integrating data sets.

Integration is part of the answer, but it does not obviate the requirement to consolidate and migrate data. For example, it’s not uncommon for enterprises to have many copies of customer data – this could be 20, 30, 40 or even a 100 datasets.

Customers are highly likely to be duplicated between these datasets leading to sub-optimal outcomes such as a customer being mailed multiple times by the same company.

To deliver the single, unified view of a company’s data that is desired, enterprises therefore need a data management strategy that fixes poor data management processes, improves data quality and consolidates datasets (which will decrease both the risk of variant data as well as operational cost).

The project should start small and grow over time, and the enterprise should begin by deciding its key priority – which might, for example, be to fix its customer data first. Each enterprise is different, but some of the factors that should be considered before embarking on an MDM initiative include:

  1. Ensuring the business takes priority.
    Like any application migration project, it’s essential that business priorities take precedence and the business users drive the project. Only business users can decide which datasets should be consolidated or migrated first, which datasets do not need to be migrated, how to resolve data conflicts or variants, and so on. In addition, getting buy-in from business users means that new data management processes are more likely to be adhered to going forward.
  2. Taking small steps.
    This type of project is complex and you need to approach it as a series of inter-related projects each with its own goals and deadlines, but each of which fits within the overall goals of an MDM strategy.
  3. Developing the master data model.
    What do you want your master data records to include? This step should include understanding the producers and consumers of master data, and mapping between current data sources and the master data model. At this step you will need to recognise inconsistencies and resolve how you are going to handle them. For example, which naming convention will you use consistently for your customers? (eg Mr John Smith, Mr J Smith, Mr J. Smith, John Smith, Smith J.)
  4. Choosing an appropriate tool.
    There are various tasks that you will need to perform in order to create a master data list. You will need to clean, normalise and standardise your data, as well as de-duplicate it. It is helpful if you have a tool that gives you a high degree of dynamic control and visibility along with bi-directional synchronisation. These features will certainly make the process of creating master data lists easier.
  5. Implementing data management strategies that will keep your master data intact.
    Over time you risk duplicating master data lists (eg due to M&A), divergence between copies of your master data, and errors creeping in. Overhauling your data management strategy to ensure your goals are met long-term is essential.

Tony Sceales is CTO of data migration specialists Celona Technologies