Moss L.T., Atre S. Business intelligence roadmap: The complete project lifecycle for decision-support applications
Подождите немного. Документ загружается.
- What generation is the data (month-end, weekly, daily)?
- Was the data obtained from direct sources or from downloads?
- Is the source of the data known?
- Is the data a duplicate of data in another data store? If so, is the data
current?
Data format: The closer the data is to the destination data format, the fewer
the conversion requirements will be. From highest to lowest, the format
priorities are:
- Data from a relational database (e.g., DB2, Oracle)
- Data from a nonrelational database (e.g., IMS, CA-IDMS)
- Flat files (e.g., VSAM, ISAM)
Figure 5.7. Source Data Selection Criteria
Source data quality will be only as good as the
enforcement of quality processes in the operational
systems. Mandatory quality processes should include data
entry rules and edit checks in programs. If those
processes are not enforced or do not exist, data usually
gets corrupted, regardless of whether the data is in a
relational database or in an old VSAM file.
To Cleanse or Not to Cleanse
Many organizations struggle with this question. Data-cleansing research indicates
that some organizations downplay data cleansing to achieve short-term goals. The
consequences of not addressing poor-quality data usually hit home when their
北斗成功社区 BeiDouWeb.com 教育音视频/电子书/实用资料文档/励志音乐影视 仅供免费试用/版权原著所有
191/631
business ventures fail or encounter adverse effects because of inaccurate data.
It is important to recognize that data cleansing is a labor-intensive, time-consuming,
and expensive process. Cleansing all the data is usually neither cost-justified nor
practical, but cleansing none of the data is equally unacceptable. It is therefore
important to analyze the source data carefully and to classify the data elements as
critical, important, or insignificant to the business. Concentrate on cleansing all the
critical data elements, keeping in mind that not all data is equally critical to all
business people. Then, cleanse as many of the important data elements as time
allows, and move the insignificant data elements into the BI target databases
without cleansing them. In other words, you do not need to cleanse all the data, and
you do not need to do it all at once.
Cleansing Operational Systems
When the selected data is cleansed, standardized, and moved into the BI target
databases, a question to consider is whether the source files and source databases
should also be cleansed. Management may ask, why not spend a little extra money
and time to cleanse the source files and databases so that the data is consistent in
the source as well as in the target? This is a valid question, and this option should
definitely be pursued if the corrective action on the source system is as simple as
adding an edit check to the data entry program.
If the corrective action requires changing the file structure, which means modifying
(if not rewriting) most of the programs that access that file, the cost for such an
invasive corrective action on the operational system is probably not
justifiable—especially if the bad data is not interfering with the operational needs of
that system. Remember that many companies did not even want to make such
drastic changes for the now infamous Y2K problem; they made those changes only
when it was clear that their survival was at stake. Certainly, a misused code field
does not put an organization's survival at stake. Hence, the chances that operational
systems will be fixed are bleak.
[ Team LiB ]
北斗成功社区 BeiDouWeb.com 教育音视频/电子书/实用资料文档/励志音乐影视 仅供免费试用/版权原著所有
192/631
[ Team LiB ]
Data Analysis Activities
The activities for data analysis do not need to be performed linearly. Figure 5.8
indicates which activities can be performed concurrently. The list below briefly
describes the activities associated with Step 5, Data Analysis.
Analyze the external data sources.
In addition to requiring internal operational source data, many BI applications
need data from external sources. Merging external data with internal data
presents its own set of challenges. External data is often dirty and incomplete,
and it usually does not follow the same format or key structure as internal data.
Identify and resolve these differences during this step.
1.
Refine the logical data model.
A high-level, project-specific logical data model should have been created
during one of the previous steps. In addition, some or all of the internal and
external data may have been modeled on other projects and may already be
part of the enterprise logical data model. In that case, extract the
representative portion of the enterprise logical data model and expand it with
the new data objects, new data relationships, and new data elements. If the
required data has not been previously modeled, create a new logical data
model for the scope of this BI project. It should include all internal as well as
external data elements.
2.
Analyze the source data quality.
At the same time that the logical data model is created or expanded, the quality
of the internal and external source files and source databases must be analyzed
in detail. It is quite common that existing operational data does not conform to
the stated business rules and business policies. Many data elements are used
for multiple purposes or are simply left blank. Identify all these discrepancies
and incorporate them into the logical data model.
3.
Expand the enterprise logical data model.
Once the project-specific logical data model is relatively stable, merge it back
into the enterprise logical data model. During this merge process additional
data discrepancies or inconsistencies may be identified. Those will be sent back
to the BI project for resolution.
4.
Resolve data discrepancies.
Occasionally data discrepancies discovered during data analysis involve other
business representatives from other projects. In that case, summon the other
business representatives as well as the data owners to work out their
differences. Either they will discover a new legitimate subtype of a data object
or a new data element, which must be modeled as such, or they will have to
resolve and standardize the inconsistencies.
5.
Write the data-cleansing specifications.6.
北斗成功社区 BeiDouWeb.com 教育音视频/电子书/实用资料文档/励志音乐影视 仅供免费试用/版权原著所有
193/631
Once all data problems are identified and modeled, write the specifications for
how to cleanse the data. These specifications should be in plain English so they
can be validated by the data owner and by business people who will use the
data.
6.
Figure 5.8. Data Analysis Activities
[ Team LiB ]
北斗成功社区 BeiDouWeb.com 教育音视频/电子书/实用资料文档/励志音乐影视 仅供免费试用/版权原著所有
194/631
[ Team LiB ]
Deliverables Resulting from These Activities
Normalized and fully attributed logical data model
This project-specific logical data model is a fully normalized entity-relationship
diagram showing kernel entities, associative entities, characteristic entities,
cardinality, optionality, unique identifiers, and all attributes.
1.
Business meta data
The business entities and attributes from the logical data model must be
described with meta data. Data-specific business meta data components include
data names, data definitions, data relationships, unique identifiers, data types,
data lengths, domains, business rules, policies, and data ownership. These are
usually captured in the tool repository of the computer-aided software
engineering (CASE) tool.
2.
Data-cleansing specifications
This document describes the cleansing logic that must be applied to the source
data in order to bring it into compliance with the technical data conversion
rules, the business data domain rules, and the business data integrity rules.
This document will be used to create the transformation specifications on the
source-to-target mapping document in Step 9, ETL Design.
3.
Expanded enterprise logical data model
This deliverable is produced behind the scenes by data administration or the
enterprise architecture group when it merges the project-specific logical data
model into the enterprise logical data model. Any rejected entities or attributes
and any discrepancies between the models will be presented to the BI project
team for resolution.
4.
[ Team LiB ]
北斗成功社区 BeiDouWeb.com 教育音视频/电子书/实用资料文档/励志音乐影视 仅供免费试用/版权原著所有
195/631
[ Team LiB ]
Roles Involved in These Activities
Business representative
The business representative assigned to the BI project is a major contributor
during the top-down logical data modeling activities as well as the bottom-up
source data analysis activities. He or she provides the business meta data to
the data administrator and assists the data quality analyst in analyzing the
source files.
Data administrator
The data administrator is trained in logical data modeling, business meta data,
normalization techniques, business data domain rules, business data integrity
rules, and standardization methods. The job description of a data administrator
matches the activities of the Data Analysis step. The data administrator will be
the lead person during this step and will facilitate all of the data modeling
sessions. He or she also has the responsibility of documenting the logical data
model and the supporting business meta data in the CASE tool.
Data quality analyst
The data quality analyst is a systems analyst, trained in using the technical
data conversion rules, in reading as well as writing programs, and in extracting
data from all types of source files and source databases. Finding the data
violations in the source files and source databases is the prime responsibility of
the data quality analyst. He or she works closely with the data administrator to
model data anomalies and to correct the data violations with help from the
business representative and the data owners.
ETL lead developer
The ETL lead developer must be involved in the modeling reviews and must be
aware of the magnitude of data quality problems found in the source files and
source databases. He or she needs to understand the complexity of cleansing
the data because the cleansing algorithms must be incorporated into the ETL
process. In some cases, the ETL tool will not be able to support some cleansing
algorithms, and custom code may have to be written.
Meta data administrator
As the custodian of meta data and the administrator of the meta data
repository, the meta data administrator needs to know what business meta
data components are being collected and how. Some meta data components
may be entered into a CASE tool, while other components may be captured in
word processing documents or in spreadsheets. The meta data administrator
will have to extract the meta data components from the various files and tools
and merge them into the meta data repository.
Stakeholders (including data owners)
The business people using the BI applications are usually downstream
北斗成功社区 BeiDouWeb.com 教育音视频/电子书/实用资料文档/励志音乐影视 仅供免费试用/版权原著所有
196/631
information consumers and not the data owners. During this step, both
information consumers and data owners have the responsibility to standardize
the business data and to set rules and policies for the data. Continuing
disagreements over data between data owners and information consumers
must be pushed up to business executives for resolution.
Subject matter expert
The subject matter expert assists the data administrator and data quality
analyst by interpreting the business data, explaining the business rules and
policies for the data, and determining the domain (valid values) of the data. In
addition, the subject matter expert is responsible for finding data problems in
the source data files and source databases and for suggesting how to correct
them.
[ Team LiB ]
北斗成功社区 BeiDouWeb.com 教育音视频/电子书/实用资料文档/励志音乐影视 仅供免费试用/版权原著所有
197/631
[ Team LiB ]
Risks of Not Performing Step 5
Business managers, IT managers, and IT technicians often do not want to take the
time to perform rigorous data analysis, which involves logical data modeling, source
data archeology, and data cleansing. They see those activities as a waste of time.
They judge the success of a BI project by the speed with which it gets delivered,
rather than by the quality of its deliverable. As a result, organizations often create
stovepipe data marts and populate them "suck and plunk" style with the same data
they have on the source files and source databases, thereby copying all the existing
data impairments to the new BI decision-support environment. Instead of
eliminating their existing data problems, they just compounded them—now there are
additional redundant and inconsistent BI target databases and applications to
maintain.
Of all the 16 steps presented in Business Intelligence Roadmap, Step 5, Data
Analysis, is the most critical cross-organizational step. This step is a major
differentiator between a traditional systems development approach and a cross-
organizational development approach. The activities of business-focused data
analysis force the information consumers and the data owners to reconstruct a cross-
organizational view and to clean up their expensive data chaos, not only in the BI
decision-support environment but in their operational systems as well. These are all
prerequisites for improving the business executives' abilities to make decisions.
Without this step, you are just building another traditional stovepipe decision-
support system, not a BI solution.
[ Team LiB ]
北斗成功社区 BeiDouWeb.com 教育音视频/电子书/实用资料文档/励志音乐影视 仅供免费试用/版权原著所有
198/631
[ Team LiB ]
Bibliography and Additional Reading
Adelman, Sid, and Larissa Terpeluk Moss. Data Warehouse Project Management.
Boston, MA: Addison-Wesley, 2000.
Aiken, Peter H. Data Reverse Engineering: Slaying the Legacy Dragon. New York:
McGraw-Hill, 1995.
Atre, Shaku. Data Base: Structured Techniques for Design, Performance, and
Management, Second Edition. New York: John Wiley & Sons, 1988.
Bischoff, Joyce, and Ted Alexander. Data Warehouse: Practical Advice from the
Experts. Upper Saddle River, NJ: Prentice Hall, 1997.
Brackett, Michael H. Data Resource Quality: Turning Bad Habits into Good Practices.
Boston, MA: Addison-Wesley, 2000.
———. The Data Warehouse Challenge: Taming Data Chaos. New York: John Wiley &
Sons, 1996.
Downes, P. M. Practical Data Analysis. Pinner, Middlesex, UK: Blenheim Online
Publications, 1989.
English, Larry P. Improving Data Warehouse and Business Information Quality:
Methods for Reducing Costs and Increasing Profits. New York: John Wiley & Sons,
1999.
Hoberman, Steve. Data Modeler's Workbench: Tools and Techniques for Analysis and
Design. New York: John Wiley & Sons, 2001.
Inmon, William H., John A. Zachman, and Jonathon G. Geiger. Data Stores, Data
Warehousing and the Zachman Framework: Managing Enterprise Knowledge. New
York: McGraw-Hill, 1997.
Jarke, Matthias, Maurizio Lenzerini, Yannis Vassiliou, and Panos Vassiliadis.
Fundamentals of Data Warehouses. New York: Springer, 2000.
Kuan-Tsae, Huang, Yang W. Lee, and Richard Y. Wang. Quality Information and
Knowledge Management. Upper Saddle River, NJ: Prentice Hall, 1998.
Reingruber, Michael C., and William W. Gregory. The Data Modeling Handbook: A
Best-Practice Approach to Building Quality Data Models. New York: John Wiley &
Sons, 1994.
Ross, Ronald G. The Business Rule Concepts. Houston, TX: Business Rule Solutions,
Inc., 1998.
Simsion, Graeme. Data Modeling Essentials: Analysis, Design, and Innovation.
Boston, MA: International Thomson Computer Press, 1994.
Von Halle, Barbara. Business Rules Applied: Building Better Systems Using the
Business Rules Approach. New York: John Wiley & Sons, 2001.
北斗成功社区 BeiDouWeb.com 教育音视频/电子书/实用资料文档/励志音乐影视 仅供免费试用/版权原著所有
199/631
Zachman, John. The Zachman Framework: A Primer for Enterprise Engineering and
Manufacturing. La Canada, CA: Zachman International, 2002.
Information Impact International, Inc.: http://www.infoimpact.com
Zachman Institute for Framework Advancement: http://www.zifa.com
[ Team LiB ]
北斗成功社区 BeiDouWeb.com 教育音视频/电子书/实用资料文档/励志音乐影视 仅供免费试用/版权原著所有
200/631