Root Cause for Data Quality Issues

It is difficult to prioritize this list, although philosophically I can say that lack of governance will most definitely lead to bad data. At the same time, the list is not finite or complete. Organizational events like mergers & consolidations can also lead to bad data quality. The fins on the upper side are processes that bring data into your system, the inflow. The lower fins are internal processes that cause bad data to persist. Either of the fins can cause data corruptions. I have summarized each of the fin below, without bloating this post.

1. Legacy Migration: Refers to data that is often migrated from a legacy system. In most cases the data structures and data models are inconsistent between the legacy architecture and the new architecture.
2. System Migration: This is almost similar to the above, except that these are due system upgrades. As applications evolve, designs change. New fields get added, when no historical data exists for this field. Or god forbid, some fields are deprecated/removed which may lead to serious problems.
3. Workarounds: This is typical of the business community and packaged applications (ERP/CRM et al). Custom fields are heavily used (often with no documentation), which later lead to some misinterpretations.
4. Manual Data Entry: Mostly happens when systems collect data from users via a “free text” field. Common examples include Addresses, Phone Numbers etc. In the absence of standards/conventions, or lack of policies, data entry users would want to finish a transaction as quickly as possible rather than worry about the accuracy of the data. If the system is not self correcting, users will never understand that they are introducing bad data.
5. Interfaces: These are the connectors between one system to another. For large enterprises, this is how data typically flows – Campaigns to Opportunity to Quote to Order to Manufacturing to Service. To compound the matters, each system is sold/supported by a different vendor with no accountability for data correction.
6. Process Automation: This is different from the Interface issue discussed above. This is more about how a system process (within that system) is automated. As existing business processes are re-engineered (due to dynamic nature of the business), applications get out of sync or new data assumptions are made. If this is not relayed to the IT team that supports the system, there will be some data corruption.
7. Time Decay: This is especially true for Master data (like customer), where the data was good at some point in time but has since been not updated. Consider your email address (specifically work emails), as customer contacts move from one organization to another their email changes with the move. The data you once had for this customer contact is no longer accurate.
8. Data Quality Programs: The irony. Yes, sometimes the data quality programs/initiatives are themselves a cause for bad data. This is mostly because of wrong assumptions on business data and rules around the data. So data may be cleansed incorrectly, aggressive merges (in the case of Master Data Management), data purges etc.
9. Lack of Ownership:  Very few organizations have complete ownership of a system (a CRM is often shared by Sales and Marketing), they often share sections of the data. With shared ownership, comes conflicting business rules and priorities. Concepts like Data Quality Organizations or Data Stewards are new to most organization, which bring accountability to an enterprise.
10. Lack of Governance: Data Governance is a vast discipline that is beyond the scope of this post. It is about arriving at a standard definitions the the common data, via meta data management. It is about analyzing, defining and base lining the current quality of the data; so some of the quality metrics can be monitored. Its about MDM and a lot more. Lack of governance, means the information management strategy is poorly executed leading to more data issues.

In summary, the reasons for bad data quality are many. Before we start looking at cleaning the data, it is prudent that we understand the root causes for bad data. Prioritize and strategize the cleanup activities; devise ongoing monitors to gauge the data and control the inflow of bad data.

Organizations today use data to generate a multiplicity of information assets (campaigns, operational systems, reports, dashboard etc). These assets form the basis for any strategic action the organization may take.  So when the incoming data is bad, all the downstream systems and assets are contaminated thereby jeopardizing the success of the organization. The impact of bad data has been quantified by several vendors and consulting organizations. Below are some facts and figures around the impact of bad quality (anecdotal & quantified).

• If customer preferences (Opt outs) is not maintained accurately, enterprises have to fork some serious penalties that increase with each incident.
• The ratio of the cost to process a transaction when data is clean and what it is accurate is 1:10. Organizations make millions of transactions a year.
• Data Integration and BI projects either fail or delayed because of bad data.
• Inaccurate medical diagnosis can sometimes be fatal.
• Lack of single version of truth (for master data) results in additional spend and/or bad customer service.

To summarize, Data Quality impacts range from a pure transaction level loss up to catastrophic impact for an enterprise. In the words of Larry English,  the cost of bad data may be 10-25 percent of the company’s total revenues. Marketing folks know the cost of customer acquisition, and the renewal potential of each customer on file. Once an organization loses its loyal customer, all the associated revenue potential goes down the drain. So what exactly are the reasons for poor Customer Data Quality? I shall cover them in a later post, for now here is the result from the TDWI Data Quality Survey on this question.

Sources of Data Quality Problems

Each system has its own constituents of users, and thus each have their own interpretation  of what “useful” data is. For the IT organization implementing a BI system, data is useful if it is “Complete”. Yet the same data for a Sales organization is useful if it is “Accurate”. For a credit card collection agency, this data is useful if its “Timely”. (It is easily conceivable that a department is interested in Complete, Accurate and Timely data). Thus we have many dimensions where data can be assessed being good (or bad). The science of assessing how good is data, for use by a department/enterprise, is usually referred to as Data Discovery.

Before we start on a trip to discovering data, it is important to understand why such a process is needed. Why is it important to know how bad is the customer data quality? In other words, there has to be a business case for Data Quality Initiatives. A business case usually takes the form of “Lost revenues  due to missed shipments” (incomplete/inaccurate address information), “Inability to up-sell/cross-sell into existing base” (no contact data) et al. If you have ever run a marketing campaign (say email), you would know the failure rates of these campaigns and how much of it is attributed to bad data. This is anecdotal evidence. So the objective of Data Discovery is to profile the data, so that the anecdotal evidence can be quantified in a scientific manner.

Like most things, too much of Data Quality comes at a price. And it is so because, Data Quality works on the law of diminishing returns. Once the most offending processes have been fixed/cured, it may cost more to fix other processes which may not yield much value. This is where Data Governance plays in nicely. Without getting deep into the details, Data governance is a set of processes that ensures that important data assets are formally managed throughout the enterprise. So what to measure and how much to measure is dictated by the Data governance board.

To summarize, Data Quality is about “fitness of use” that needs to be measured across many “dimensions”. Data discovery or profiling needs to be applied to understand how bad the data symptoms are. Data governance (besides a lot of things) defines the boundaries for the data quality initiative.

Mistake 1. Failing to Use a Methodology
Mistake 2. Ineffective Project Team Structure
Mistake 3. Failing to Involve the Business People
Mistake 4. Failing to Have Application Releases
Mistake 5. Failing to Have an Active Project Charter
Mistake 6. Lack of a Readiness Assessment
Mistake 7. Inadequate Testing
Mistake 8. Underestimating Data Cleansing Efforts
Mistake 9. Ignoring Metadata
Mistake 10. Being a Slave to Project Management Tools

Technorati Tags: Data Warehouse, Project Managers, Data Marts, Business Intelligence, Mistakes

Lets take a subset of some of the BI offerings, and see how Open Source solutions match up against them.

1. Databases: PostGres (or Bizgres) and MySQL.
2. ETL: Enhydra Octopus and Clover
3. OLAP: Mondrian and GreenPlum
4. Dashboards (Portal approach): JBoss Portal, JetSpeed, LifeRay and Gluecode (now owned by IBM.
5. Reporting: Jasper Reports, BIRT

If this doesn’t sound promising, Pentaho is planning a Open Source BI distribution that brings all of the above together and some more. This sounds very exciting for myself and for my company, which has been providing Open Source solutions for almost 5 years.

Technorati Tags: Pentaho, JasperReports, MySQL, JBoss, Portal, Business Intelligence, BI