What is Knowledge Integrity? - Published in DM Review October 2002

What is the difference between "data," "information," "knowledge" and "intelligence"? Every time I have posed this question to an industry expert, I am either treated to a long-winded exposition full of hot air, or I am chided that the question is of limited value. Yet, there is a significant difference between terabytes of data and true business intelligence.

Companies have been building (and rebuilding) large-scale data systems, driven by the information technology group, hoping to extract some knowledge out of that system in order to increase their business intelligence. As the writer of this new column called "Knowledge Integrity," I believe that this question is a good way to introduce my motivation behind the title and the concepts that I hope to discuss in this column.

Over the past 20 to 30 years, the decrease in both cost and footprint size of hard-drive systems has encouraged many organizations to increase their reliance on mass storage solutions. Organizations can afford to build larger transaction systems and data warehouses, but how frequently is any of this amassed data used? More importantly, how well has this data been used? Even more acutely, how has the low cost of mass storage actually created an environment that drains additional resources (both capital and intellectual) due to increased data management issues?

As data creation, production, storage and availability have increased, one's ability to make sense out of that data has decreased. Representatives of companies in the data warehouse (DW), business intelligence (BI) and customer relationship management (CRM) as well as other "knowledge-intensive" markets are constantly trying to grab our attention by claiming that their componentry is critical to the success of your project. However, it seems that although we are approaching the maturation stages of these technologies, the promise of BI and CRM still largely outpaces the industry's ability to deliver. This failure is not just a technical failure; it is tied to the political environment of data: traditional views of data, vertical management hierarchies, outmoded compensation structures, ill-defined ownership roles, etc.

We are in the information age, but our information is processed and managed in a manner that recalls the industrial age, especially in terms of an assembly-line sequence of processing stages. Although significant advances have been made in the areas of distributed computation and parallel processing, most legacy information applications still operate in a linear fashion, creating "processed information" in isolated stages, the same way that cars are manufactured. This factory-style processing creates artificial barriers to the effective use of information, leading to lost opportunities and decreased competitiveness.

Knowledge integrity begins where the infrastructure ends. An organization can have the best-of-breed data warehouse or CRM software, but the application can only succeed if the information in that system is of high value. We are slowly migrating from a universe where system architecture and infrastructure are the commanding principles to a universe where application success is directly tied to content, context and value. Knowledge integrity focuses on the following concepts:

Understanding the intrinsic value of information.
Characterizing how information is used.
Formally defining expectations of the state of a collection of one or more data sets and being able to measure conformance to these expectations.
Looking at ways of enhancing the value of information.
Methods to benchmark information value and to measure improvement against that benchmark.
Exploring the tangled concepts of data ownership.
Discovering and exploiting knowledge embedded in data.
In this column, we will explore how organizations can move toward a knowledge-centric view where information is a critical corporate asset used for competitive and strategic advantage. We'll discuss migration from the industrial age to the knowledge age where multiple data sets are qualified, aggregated, fused, enhanced, shared and broadcast as part of an enterprise-wide knowledge system. Together, we will challenge the conventional thought processes revolving around the creation, use and management of information and trigger novel ways to think about and manipulate data such as:

• Changing the conventional wisdom about data.
• The politicization of information within an organization.
• Data ownership paradigms, roles and responsibilities.
• Data valuation and building the information ROI.
• The positive value of managed information quality.
• Business rules as content.
• Discovering knowledge in data.
• Changing models in a changing world.
• Static data modeling versus associative model representation.
• Information control.
• Data aggregation as an intelligence tool
• Knowledge discovery and privacy ­ can they coexist?
• Exploiting high-performance technology in the data world.
• Abstracted entity representation.
• Behavior modeling as a byproduct of activity logging.

This experiment will be successful as long as it triggers some reaction on behalf of the readers, either positive or negative. My hope is that if you read this column and either want to agree or disagree with what I have presented, that you won't hesitate to e-mail me with your comments – let's turn this into a real dialogue! I look forward to presenting my ideas and especially to hearing yours!