Moss L.T., Atre S. Business intelligence roadmap: The complete project lifecycle for decision-support applications
Подождите немного. Документ загружается.
Do we need to build a meta data repository directory as part of the
access interface for the business people and for the technicians?
Do we need to build a context-sensitive online help function for the
meta data repository?
Do we need to write meta data reports? How will we distribute
them? On paper? On the intranet?
Will meta data be integrated with the queries and reports of the BI
application? What type of process do we have to build to support that?
How can we ensure that the meta data in the meta data repository
does not get out of synch with the meta data in the other tools and
the database management system (DBMS)?
Staffing
How much can we modularize the coding and testing? Do we have
enough developers and testers to speed up the development effort?
Will the same developers work on the meta data repository online
help function? Or can we develop the help function in parallel?
Will we have full-time or part-time support from the database
administrator? Will the same database administrator support the BI
target databases?
Who will continue to maintain the meta data repository? Do we
have a full-time meta data administrator? Is one person enough?
Are there any special training needs we have to consider? For the
developers? For the business people?
Preparation for Production
Will the production meta data repository be installed on a
dedicated production server? Who will install and test the server?
Do we need to set up any regularly scheduled meta data repository
programs on the job scheduler? Will the operations staff run and
monitor them?
Do we need to write operating procedures for the operations staff?
Do we need to write a reference guide for the help desk staff and
for the business people?
To navigate through the BI decision-support environment more efficiently, business
people must have access to a meta data repository. There are only two options:
license (buy) a meta data repository or build one. Once a meta data repository is
implemented, it has to be maintained and expanded over time. It also has to be
populated and updated during each ETL process cycle with load statistics,
reconciliation totals, data reliability metrics, data rejection counts, and the reasons
for the data rejections.
北斗成功社区 BeiDouWeb.com 教育音视频/电子书/实用资料文档/励志音乐影视 仅供免费试用/版权原著所有
401/631
[ Team LiB ]
北斗成功社区 BeiDouWeb.com 教育音视频/电子书/实用资料文档/励志音乐影视 仅供免费试用/版权原著所有
402/631
[ Team LiB ]
Populating the Meta Data Repository
Populating a meta data repository is usually not a manual effort. A meta data
repository receives most of its meta data from many different meta data sources.
These meta data sources are controlled by people other than the meta data
administrator, as illustrated in Figure 14.1. A meta data migration process has to be
developed to extract the meta data from these sources, associate (link) related meta
data components, and populate the meta data repository. The different sources for
the meta data are briefly discussed below.
Word processing files can be manuals, procedures, and other less formal
documents that contain data definitions and business rules. Embedded in these
business rules could be policies about the data, processing rules, data domains
(in code translation manuals), and sundry notes about the history and
ownership of the data or the processes performed on the data. Some word
processing files also contain valuable technical documentation describing data
and process rules enforced by programs.
Be cautious with word processing files. These files are
rarely maintained, and the information contained in
them could be out of date and no longer applicable.
Spreadsheets contain calculations and macros, which are executed on the data
in the private spreadsheets of business analysts after they have downloaded the
data from the various operational systems. These calculations and macros could
be the source for transformation rules, cleansing rules, derivations,
aggregations, and summarizations.
CASE tools contain the names, definitions, sizes, lengths, relationships,
cardinality information, referential integrity rules, and notes about data that
has been modeled either for an operational system or for a BI application. In
addition, CASE tools usually can store the technical names of tables and
columns, as well as primary keys and foreign keys. Some of the more
sophisticated CASE tools have modules to include meta data for process
components, such as programs, screen displays, and report layouts.
Internal DBMS dictionaries are an integral part of all DBMSs since the
dictionaries control the database structures. In relational databases, these are
usually called SYSTABLES, and they store the names, definitions, sizes, lengths,
relationships, and volumes of database structures, such as storage groups,
tablespaces, tables, columns, primary keys, foreign keys, and indices.
ETL tools would not function without instructions (technical meta data) for the
required transformations. The internal dictionaries of ETL tools store the
source-to-target mapping as well as all the transformation algorithms, which
are applied to the source data during the ETL process.
OLAP tools store specifications about data derivations (calculations),
aggregations, and summarizations in their internal directories. These
北斗成功社区 BeiDouWeb.com 教育音视频/电子书/实用资料文档/励志音乐影视 仅供免费试用/版权原著所有
403/631
specifications allow the OLAP tool to perform its drill-down and roll-up
functions. Some OLAP products have the capability to drill across into another
database under the same DBMS or even into another database under a different
DBMS to extract detailed data for a query.
Data mining tools store the descriptions of the analytical data models against
which the data mining operations are executed.
Figure 14.1. Sources for the Meta Data Repository
If any meta data contained in these meta data sources is about to change, the meta
data administrator must be notified before the change occurs. He or she will then
have to determine whether the meta data repository can accommodate that change
or whether the meta data repository has to be modified or enhanced. Therefore, in
order to maintain a healthy and useful meta data repository, the meta data
administrator must collaborate with the ETL team, the OLAP team, the data mining
expert, the data administrator, the database administrator, and the business people
on the BI projects. In addition, and more importantly, the meta data administrator
must have full cooperation from the operational systems people who maintain the
operational source files and source databases.
Changes made to operational source systems are
frequently not communicated to the meta data
administrator in time to make the necessary changes to
the meta data repository. Because these changes also
affect the ETL process, and in some cases the structures
of the BI target databases, the ETL team is also
impacted. This breakdown in communication between the
operational systems people and the BI decision-support
staff can cause severe delays.
Ideally, meta data repositories should be active repositories, similar to the DBMS
dictionaries. In an active meta data repository, changes would be made only to the
meta data repository, and the meta data repository would propagate the changes
into the appropriate target tool or DBMS. However, currently the meta data
repository products on the market are still passive repositories. That means changes
must be made both to the meta data repository and to the appropriate target tool or
DBMS, and these changes must be kept synchronized, either manually or with
北斗成功社区 BeiDouWeb.com 教育音视频/电子书/实用资料文档/励志音乐影视 仅供免费试用/版权原著所有
404/631
programs.
[ Team LiB ]
北斗成功社区 BeiDouWeb.com 教育音视频/电子书/实用资料文档/励志音乐影视 仅供免费试用/版权原著所有
405/631
[ Team LiB ]
Meta Data Repository Interface Processes
Every meta data repository must have two interfaces in order to function: a tool
interface to accept meta data from other tools and an access interface to interact
with business people and technicians. If the meta data repository is built in-house,
the processes to provide both types of interfaces must be developed (Figure 14.2). If
a meta data repository product is licensed, the access interface will be provided as
part of the product, but a tool interface process must still be developed.
Figure 14.2. Meta Data Repository Interface Processes
The Tool Interface Process
Most of the business meta data and the technical meta data is not entered directly
into the meta data repository but is extracted from the tools that capture it and use
it to support their own functionality. In order to exchange this meta data, the meta
data repository and the tools where the meta data originates need to communicate
with each other using common meta data standards. Unfortunately, these tools do
not interact with each other very well outside of rudimentary import and export
capabilities, and common meta data standards have not yet been agreed upon and
ratified by the meta data repository vendors. Therefore, it is necessary to develop a
tool interface process with several interface programs to extract the meta data from
the various tools.
Meta data repository vendors are recognizing the lack of tool integration as a
problem, and meta data standards are being debated by two authoritative groups:
the Meta Data Coalition (MDC, influenced by Microsoft) and the Object Management
Group (OMG, influenced by Oracle). At this time it is unclear which meta data
standards will be adopted as the industry standards.
The Access Interface Process
北斗成功社区 BeiDouWeb.com 教育音视频/电子书/实用资料文档/励志音乐影视 仅供免费试用/版权原著所有
406/631
A meta data repository must also have a common access interface for the people
who need to access it. Business people and technicians will want to retrieve
information (meta data) about the business data in the BI target databases, the load
statistics from the ETL processes, and other process-related meta data. Because
business people and technicians may have different requirements, some access
interface screens and programs may have to be designed slightly differently for these
two constituencies. Technicians and "power users" may also get authorization to
access the meta data repository directly without going through the access interface.
If you have only one centralized meta data repository, building a common access
interface process for it seems like an obvious requirement and relatively easy to do.
But if you are implementing a decentralized meta data repository solution, or even a
distributed one, building a common access interface process to all meta data
repositories or tools may not be such an obvious requirement. It is definitely a lot
harder to build a common access interface process for a decentralized or distributed
meta data repository—with or without the help of Extensible Markup Language
(XML). However, without that common access interface, everybody who wants to
access meta data from multiple meta data repositories or tools would have to learn
to use a different access interface for every meta data repository and tool. To avoid
having to use multiple access interfaces, business people may well end up insisting
on duplicating common meta data into their own "preferred" repositories. This can
rapidly lead to redundant meta data repository silos, which is precisely what must be
avoided.
[ Team LiB ]
北斗成功社区 BeiDouWeb.com 教育音视频/电子书/实用资料文档/励志音乐影视 仅供免费试用/版权原著所有
407/631
[ Team LiB ]
Meta Data Repository Testing
Developing a meta data repository can be as complicated as developing any other
application. Therefore, the same structured development guidelines should be
followed, particularly the testing guidelines.
While most business managers and information technology (IT) managers demand
quality in their applications, they often do not want to spend much time on testing.
That does not work! In addition, many project managers constantly underestimate
how long testing will take, and they do not allocate sufficient time for testing in the
project plan. What is worse, when milestones are missed, the short time originally
planned for testing is reduced further.
The point is that you must schedule sufficient time for meta data repository testing
and reflect it in the project plan. In addition, plan to have a sufficient number of
testers available at the appropriate time. Fortunately, testing is one of the few
activities that can easily be modularized, and adding more testers will speed up the
testing effort.
Experience has shown that for every day of coding, you
should plan three days of testing.
Of the various types of testing discussed in Step 11, ETL Development (unit testing,
integration testing, regression testing, performance testing, quality assurance [QA]
testing, and acceptance testing), only four types of testing are usually required for a
meta data repository, as shown in Table 14.1.
Unit testing for meta data repository programs is no different from unit testing
for other types of application programs. The programs must compile without
error; they must perform their functions according to the program
specifications; and they must include the appropriate edit checks.
Table 14.1. Required Testing for a Meta Data
Repository
Unit testing
Does the code compile without errors?
Integration
testing
Do the deliverables meet the requirements?
Regression
testing
Did the changes to the program cause damage to
this or any other program?
Acceptance
testing
Are all aspects of the deliverable acceptable?
北斗成功社区 BeiDouWeb.com 教育音视频/电子书/实用资料文档/励志音乐影视 仅供免费试用/版权原著所有
408/631
Integration testing, sometimes referred to as system testing, verifies that the
original objectives of the meta data application are being met and that all meta
data repository functions perform as expected. Integration testing should
include the initial population of the meta data repository, periodic or
intermittent updates to it, the access interface process, the tool interface
process, scheduled reports, canned queries, and the online help function. This
test is carried out according to a test plan (described in Step 11, ETL
Development), and the test cases are run by testers rather than by the
developers of the code. (Developers can test other developers' code, just not
their own.)
Regression testing is performed on different versions of the same software,
which can be produced during the development cycle within one project or
when the same software is revised during different subsequent projects. Each
time changes are made anywhere in the existing programs, a full regression
test cycle should be run to ensure that the changes did not inadvertently break
some programming logic downstream in the program or somewhere in the
entire stream of programs. Regression testing is a formal test activity and must
be performed with a test plan. Since a complete meta data repository will
probably not be built all at once but over time, regression testing is an
important part of all projects that enhance the meta data repository.
Acceptance testing is even more important than the other forms of testing. This
test determines whether the meta data repository (new or enhanced) is ready
to be rolled out. This final testing process for the meta data repository usually
combines QA testing and acceptance testing. The business people and the
technicians who will be using and supporting the meta data repository are
involved in this testing. Similar to integration testing and regression testing,
this test is executed with a test plan, predefined test cases, and expected test
results; all test runs are documented in the test log.
[ Team LiB ]
北斗成功社区 BeiDouWeb.com 教育音视频/电子书/实用资料文档/励志音乐影视 仅供免费试用/版权原著所有
409/631
[ Team LiB ]
Preparing for the Meta Data Repository Rollout
Moving a meta data repository into production requires preparation, as shown in
Figure 14.3. This preparation should start early, not when all the coding and testing
have been completed. Novice meta data repository teams often underestimate the
time it takes to prepare the basic aspects of the production environment outlined
below.
Server platform: The meta data repository should reside on a production server
and not on the development server. Therefore, the production server platform
has to be installed and tested, which includes the hardware components,
operating system, monitoring utilities, and network connectivity.
Production DBMS: If the meta data repository is installed on a new production
server, an instance of the DBMS has to be created, and parameters for it have
to be set and tested under the operating system. If a meta data repository
product is being licensed, all product components including the meta data
repository database have to be installed and tested on the production server.
Program and query libraries: All meta data migration programs (including the
tool interface programs) and all meta data application programs (including the
access interface programs, the online help function, the reports, and the
queries) will reside in a version-controlled library. A library management
product has to be installed and tested before the meta data repository
programs can be moved into production.
Security: The production server, the DBMS product, the meta data repository
database, and all programs need to have the proper levels of security
implemented. Security levels in a production environment are much stricter
than in the development environment. Developers who could change the meta
data repository database structures and its content at will in the development
environment should not be granted the same authority in the production
environment.
Instruction guides and manuals: Once the meta data repository is in production,
some of the meta data programs will be scheduled to run automatically with
every ETL process cycle to capture the load statistics and data quality metrics
and to load them into the meta data repository. Certain reports or queries may
also be put on a schedule.
Operations staff will monitor the scheduled runs and notify the meta data
administrator if something goes wrong. Operating procedures should be
prepared for them, listing the scheduled jobs, the order in which the jobs
should be run, and what to do if a job fails. The help desk staff will mentor and
support the business people with their meta data questions. Therefore, the help
desk staff will need a reference guide to fall back on in case they get inquiries
they need to investigate. This reference guide could also be given to business
people since it contains helpful hints about where to locate specific meta data
components and where to get additional help.
Meta data repository training: Business people and technicians need to be
北斗成功社区 BeiDouWeb.com 教育音视频/电子书/实用资料文档/励志音乐影视 仅供免费试用/版权原著所有
410/631