Hands-On Microsoft SQL Server 2008 Integration Services part 57 ppt

538 Hands-On Microsoft SQL Server 2008 Integration Services
of an ODS cannot provide answers to such complex questions because either the data
is not available in the ODS systems or the data model doesn’t support such analysis.
This results into creation of a data warehouse or a decision support system. A data
warehouse or a DSS collects data from OLTP or ODS systems and might keep it
in multiple forms that is, in its most granular form and in aggregated form. A data
warehouse keeps data for longer periods of time (generally spread across several years)
even after it has been deleted from the source systems.
Data Warehouse Design Approaches
Now we know a bit about a data warehouse: that it keeps years of history, that it keeps
data in granular as well as aggregated format, and that the primary function of a data
warehouse is to do business intelligence or analytical reporting. So the next question is
how we design a data warehouse. There are two primary schools of thought—top-down
and bottom-up approaches—and both are good and relevant to their own applications
but also have associated risks involved.
Top-Down Design Approach
Bill Inmon, who is best known as the father of data warehousing, has defined
this approach in which he suggests a data warehouse to be at the center of the
Corporate Information Factory (CIF) designed using a normalized data model. The
Corporate Information Factory approach takes the holistic view of the enterprise
and its informational needs. In such a data warehouse, data is collected from most
of the organization’s operational systems and is held at the atomic level, that is,
at the lowest level of detail. Further, the subject-oriented dimensional data marts
containing aggregated data are built from the central atomic data warehouse to serve
the departmental needs. As the data warehouse becomes a source system for all the
analytical data marts and organizational reporting, this creates a consistent view—a
single version of truth across the enterprise. It is easier to realign the data warehouse
built with this approach to support business changes, and the data marts can be
recreated easily from the central data warehouse. However, the downside of this
approach is that the difficulty involved in building such a data warehouse, collating
the entire enterprise data from almost all the operational data sources, makes this a

very large project. So, implementing this project requires a high upfront investment, a
large effort, and a long time to deliver. This long delivery time results in users losing
patience and developing their own solutions, thus deviating from the original point
of consolidation. The departments or business units generally don’t buy into such
projects with large delivery timescales.
Chapter 12: Data Warehousing and SQL Server 2008 Enhancements 539
Bottom-Up Design Approach
In the absence of a central data warehouse, when departments or business units
need a data mart for a specific business process, they can’t wait for the central data
warehouse to be built but create their own departmental data mart. Ralph Kimball,
who is a well-known author on data warehousing, is a proponent of this approach, for
which he suggests creating business-specific dimensional-modeled data marts first.
These data marts can contain atomic as well as aggregated data in a denormalized data
model. These data marts are created in phases and are linked together via conformed
dimensions. A strict discipline is adopted to implement conformed dimensions that
are consistent with each other and use the same structure, attributes, and attribute
values. Either these dimensions are exactly the same, including the keys, or one
is a perfect subset of the other. The backbone of this approach is the conformed
dimension’s principle, and management of the conformed dimensions is fundamental
to maintaining integrity of the data warehouse. Such a structure in which a data mart
is linked to another data mart via a conformed dimension is called data warehouse bus
architecture. Later these data marts are joined together to create an enterprise data
warehouse. Because dimensional data marts are at the center of the bus architecture
approach, the data model is simple to understand and use. The dimensional or star
schema data model enables queries to retrieve data very efficiently. Implementing a data
warehouse using this approach realizes business investment quite quickly—as soon as
the data marts start functioning. The data warehouse is developed in phases and hence
is more feasible than the central data warehouse approach. The downsides of choosing
this approach are complicated data loading routines to maintain integrity within the
data marts and the difficulty in modifying data warehouse structures when business

changes occur.
Data Warehouse Architecture
Based on the previous two design approaches, you can have two different architecture
layouts.
Centralized EDW with Dependent Data Marts
This architecture is based on Bill Inmon’s Corporate Information Factory model.
As the name implies, in this approach the data and applications reside on a central
mainframe system. In this model, the data marts are fed exclusively from the centralized
EDW in order to assure the single version of truth, thus creating a structure in which
the data marts depend on the centralized EDW. Large enterprises use this approach, as
it provides some key benefits that are vital to their success. However, there are instances
540 Hands-On Microsoft SQL Server 2008 Integration Services
when this approach has failed to achieve desired results due to long deployment periods
and huge upfront investments. The benefits and the issues involved are listed here:
A single version of truth is the biggest reason why enterprises adopt this approach.
c
Centralized data management makes it easy to apply corporate standards and c
practices and allow businesses to comply with the legal requirements; a recent
example of compliance is the application of PCI DSS standards issued by the PCI
Security Standards Council.
With a holistic approach, the development of such a warehouse does not follow c
the general principle for phased-iterative development.
Due to their implementation plans, centralized EDWs can become quite inflexible.
c
When business units want to create analytical data marts, they get frustrated by
this inflexibility and start moving away from this approach by creating federated
data marts.
Historically, the systems deployed in this architecture have been proprietary to a c
vendor that is slow to respond to fast improvements in technology and hence, they
do not benefit from the advances in general-purpose computer systems.

is is a high-risk approach, as implementing a centralized EDW is very expensive
c
with a huge upfront investment that is realized only after the data warehouse
project is completed and data marts are beginning to emerge. e maintenance and
scalability costs have been found to be high in such an approach due to inflexibility.
All these costs add up to a very high total cost of ownership.
Distributed Independent Data Marts
This architecture is based on the Ralph Kimball’s Bus Architecture approach and
is opposite to the monolithic design of a centralized EDW. In this architecture,
distributed departmental data marts are created as and when the need arises. These
data marts are designed independent of other data marts, and they get their data from
operational source systems rather than centralized data warehouse. These data marts
are highly relevant to their departments or the business units and are quick to build,
as the scope is generally quite clear. However, due to the lack of control over their
implementation, they often result in many versions of truth and are very difficult to
keep consistent across the enterprise. The benefits and issues affecting this architecture
are listed here:
e data marts are easy and relatively quick to build. e cost to build one data
c
mart seems low, as the return on investment starts pretty soon.
Chapter 12: Data Warehousing and SQL Server 2008 Enhancements 541
ough the cost of developing a data mart seems low, the overall cost to develop a c
distributed data warehouse is not low. e cost is spread across several data marts
and remains unnoticed in the overall expenditure.
e data management across independent data marts is not easy. A lot of redundant
c
data exists across many data marts.
Many versions of truth exist across an enterprise, and a data mart does not align
c
with another department’s data mart, though the dimensions may align.

Data Warehouse Data Models
You can choose either of the two data modeling techniques–Entity-Relationship (ER)
modeling and dimensional modeling—depending on your application scenario. The entity
relationship model has been in use much longer than the relatively new dimensional
model.
Entity-Relationship Model
The ER modeling is more popular in operational applications or OLTP systems,
though it is also used in data warehousing as well. This is a requirement in the top-
down approach, where a central data warehouse is built using the ER model. There are
several graphical tools that help you to create an entity-relationship diagram (ERD) to
conceptualize data. These tools primarily use three basic constructs: entity, relationship,
and attribute.
Entity
c An entity is a concept, real or abstract, about which information is
collected. It represents a class of objects such as products, an object such as
a car, or an event such as a sale that can be classified by their properties and
characteristics. It will usually have a business definition and is defined uniquely
using primary keys in the data model.
Relationship
c A relationship is an association among entities in a model and
indicates how two or more entities are related to one another. For example, a
customer owns a car—this indicates the relationship, how the customer is related
to the car. It is represented by a line drawn between entities. e entities are related
to each other in differing cardinality, which defines the maximum number of
instances of one entity related to a single instance of another entity. e relationship
has one-to-one, one-to-many, and many-to-many cardinalities.
Attributes
c Both entities and relationships have attributes that describe their
characteristics or properties. For example, a car has a VIN number attribute.
542 Hands-On Microsoft SQL Server 2008 Integration Services

The relational databases are designed using the normalized ER model to remove
redundant data. Six normal forms have been defined to date, but a database is
considered adequately normalized if it is in third normal form (3NF). Normalization
is a systematic process for assigning attributes to entities to ensure that a database is
integral by breaking down information to its smallest divisible parts and removing data
duplication. It is an incremental process, which means that to be transformed into
3NF, the entities must first qualify for 2NF. The 1NF removes repetition in data by
creating one-to-many relationships between master and detail entities. For example,
you will remove repeating similar columns from a table into another table. 2NF takes
removal of repeating data a further step by removing duplicate rows of data from a table
into a separate table. 3NF removes the columns that are not dependent on the primary
key and resolves many-to-many relationships into unique values.
As the normalization level increases, the data is further broken down and granularity
of data is increased. Such highly granular data models are very efficient in returning
small amounts of information or updating small amounts of data. This is required by
OLTP systems that have many users working on small pieces of information. That’s the
reason the ER Model is highly successful in relational database applications. However,
the requirements of a data warehouse are different. The queries are usually small in
number, but they can perform huge I/O activity on the server. These requirements are
met with the dimensional model.
Dimensional Model
Dimensional modeling is a relatively newer modeling technique than ER modeling.
Recent trends in modeling preference favor dimensional data modeling, as it is simple
to build, is easily to understand even by business folks, and aligns with the questions
usually asked of a data warehouse. This model is very efficient at summarizing values
and presenting data to analytical tools. This model keeps the numerical values called
facts in one table, while the attributes that measure these values are grouped together in
tables called dimensions. This structure makes it particularly suitable to answer business
questions such as sales this quarter or the sales this quarter compared to sales the
previous quarter.

Fact
The numeric values along with some contextual data are stored in fact tables. A data
warehouse contains one or more subject-oriented fact tables. A fact typically represents
an item, an event, or a business transaction such as sale of an item that can be used to
perform business analysis. Further, a fact consists of some columns containing values
and some foreign keys linking to dimensions. As the transactions are added to the fact
Chapter 12: Data Warehousing and SQL Server 2008 Enhancements 543
table, it can soon become quite large; consider that each transaction could represent a
row in the fact table.
It is the granularity that can really affect the size of your fact data. You have to
understand the business requirements completely—what they call for now and in
the future—to decide the level of detail you want to keep in the fact data. You might
choose to keep every transaction, or you might choose to keep aggregated data at the
day level if business is never going to drill down to the transaction-level detail. Thus
the lower the granularity level, the more the data and hence, the more disk space will
be required. However, disk space should not be an issue, because data warehouses are
meant to be large in size. Always be careful while choosing granularity, as a change in
granularity means the whole data warehouse has to be reseeded.
Measure
A measure is closely related to a fact, and sometimes the terms are used interchangeably.
A measure is what you want to measure, and the fact is a measure with context. So, a
measure is a numeric value used to indicate the performance of the business, and the fact
equates to a row in the Fact table. A measure is used in combination with dimension
members, while the value is taken from facts. For example, TotalSales-by-year and
SupplyCost-by-month are measures.
Dimension
A dimension contains the same type of information broken down in levels of interest. For
example, a time dimension can contain year, quarter, month, and day levels. A dimension
contains the information that a business wants to analyze the facts with. This information
does not change very often. Typically, a dimension table is relatively quite small compared

to a fact table. A data warehouse can have many dimensions attached to each fact table.
Some of the common dimensions could be Time, Product, Employee, Location, and
Customer. A dimension is made up of members and hierarchies.
Dimension Members
Member of a dimension are arranged in levels, for instance, members in a time dimension
have different levels: day, week, month, quarter, and year. Similarly all cities, states, and
countries are members of a geography dimension. While analyzing the data, you can choose
any dimension member level and associated facts will be used in analysis automatically.
Dimension Hierarchies
The members of a dimension can be arranged in a hierarchical order with multiple
levels to create dimension hierarchies. You can create more than one hierarchy and a
dimension member can participate in more than one hierarchy. For example, you can
544 Hands-On Microsoft SQL Server 2008 Integration Services
create two hierarchies for time dimension—one with Day, Month, Quarter, and Year
as levels and the other one with Day and Week as levels.
Dimension Types
Dimensions can be designed in different ways to meet specific business functions and
to enhance performance. Four types of configurations are covered here.
Conformed Dimension
At times you will use a dimension in more than one subject-oriented data marts. If you
are keeping the dimensions exactly same or are sourcing them from the central data
warehouse, obviously making them the same, then these dimensions are called conformed
dimensions. A conformed dimension does not need to be exactly the same as the
main dimension; it is still conformed to the main dimension if it is a subset of the detailed
dimension. In this case, the attributes in a conformed dimension need to be labeled exactly
in the same way as in a detailed dimension. The most common example could be a date
dimension used across many data marts having same attributes such as date, month,
quarter, and year.
Junk Dimension
The business data generally contains some attributes that are not related to any

dimension but are associated more with the fact. These can be easily identified, as
generally these attributes represent themselves in the form of indicators or flags. For
example, in a car rental business flags such as IsDamaged, IsStolen, and IsExchanged
are common in the databases. These flags are not exactly part of any dimension, but
businesses do want to analyze data using these flags. If you leave them with the fact
data, the performance of the queries will be very poor due to the large size of the
fact table, and indexes won’t help due to yes/no nature of such flags. You could put
them in their own dimension, in which case you would have as many dimensions as
there are such attributes. Very frequently you will see that the number of indicators or
flags that exist in the data reaches 20 plus. In this case, your data mart design will be
cluttered with lots of dimensions that have only one member, enough to confuse users.
A recommended approach is to club all these nonrelated attributes and put them in a
table, thus creating a junk dimension. For instance, you can select distinct combinations
of the attributes and add them to the junk dimension where each distinct row is
assigned a surrogate key that is referenced in the Fact table. Keeping the flags and
indicators in one dimension makes it easier for users to find out these attributes and is
useful in that the queries perform much better.
Chapter 12: Data Warehousing and SQL Server 2008 Enhancements 545
Degenerate Dimension
Another type of attribute in the data is the number associated with each business
transaction, such as an invoice number or a ticket number. Though these attributes are
not actively used in analysis, every now and then businesses do have a requirement to
look for such attributes, as they link back with operational systems. These attributes
are also very useful in reconciling the data warehouse with ODS systems. When the
grain of the fact table is the same as that of a transaction, the likelihood of a degenerate
dimension increases. As the fact grain is the same as that of the degenerate dimension,
so sometimes it forms an integral part of the primary key of the fact table. A degenerate
dimension should exist along with the facts in the fact table. There is no point in
creating a separate dimension table for a degenerate dimension, as it will grow with the
fact and will become quite large.

Role-Playing Dimensions
These types of dimensions exist when the same attribute is used multiple times. For
instance, a fact row could contain SaleDate and DeliveryDate columns, both pointing
to the date attribute of the date dimension. In this case, the key of the date attribute
of the date dimension is used multiple times in the fact row and is often referred to as
a role-playing dimension. You create a table alias or a view to use the date dimension
foreign key in the fact table.
Loading a Dimension Using
a Slowly Changing Dimension
Loading a static dimension is a simple one-off task, but you will come across dimensions
that change with time and will find loading such a dimension a challenging task. Some of
the data warehouse dimensions do not change that often. For instance, a date dimension’s
members stay as is and never change. By contrast, other business dimensions do change
with time as a business evolves; for example, a product could change in size and volume.
Typically, a row in the dimension table will have different attribute requirements:
Some attributes never change, such as IDs, or in the case of a car, a VIN number.
c
Some attributes will change but a business always want to see the current value; c
for instance, a business may not bother to see the old description of the product
if the description of a product is changed. is type of change is called a Type 1
change in which the attribute value is overwritten and a business cannot go back
and see the old value or learn when the change in value has been applied.
546 Hands-On Microsoft SQL Server 2008 Integration Services
Some attributes will change and the business will want to see the current value as c
well as the historical value. is happens when a business is interested in analyzing
the facts before and after a particular change has been applied. For instance, a
business might want to see the effect on sales when the size of a can of beans is
changed from 400 g to 350 g. is type of change is called a Type 2 change.
Most of the business requirements can be covered with use of the preceding types.
c

Other types of changes have been defined such as Type 3 and so on; however, their
primary function is to improve storage efficiency and query performance. ese
change types are not covered here. Refer to data warehousing books if you want to
know more about them. Also, the SCD transformation in SSIS supports only up
to Type 2 changes.
You have studied the Slowly Changing Dimension (SCD) transformation in
Chapter 10 and have done a Hands-On exercise as well. Here just to recap: the SCD
transformation is designed to help you load a dimension that changes in time, which
is generally a challenge with an ETL tool. The SCD transformation supports the
following attribute change types to support the previously mentioned requirements.
Fixed Attribute Change Type
c is change type supports the attributes that do
not change (Type 0) and align with the first scenario mentioned previously.
Changing Attribute Change Type
c Using this change type, you can load the
attribute values that are Type 1 changes, and it overwrites the existing values with
the new values. is is an in-place modification.
Historic Attribute Change Type
c In this change type, a new row is added that
will be valid for the current or future transactions. Typically, three columns are
used to handle this type of change—StartDate, EndDate, and IsActive. When
a row is getting a Type 2 update, it will update the IsActive flag to ‘N’ and will
timestamp the EndDate with the current date and time to indicate that this
row is no longer active, while the activity period of this row can be found using
StartDate and EndDate. Also, at the same time, a new row is added with same
values in all the columns except in the Type 2 column that is getting the update.
In this Type 2 column, the updated new value is inserted, StartDate gets the
current date and time stamp, EndDate is kept as null, and the IsActive column
gets a ‘Y’ value to indicate that this row is active for the particular member.
Among other methods, SCD transformation can be tested to load a dimension. If

you think that your dimension is too huge and the SCD transformation is not a fit for
the purpose, you can always create a script in your package to load a slowly changing
dimension.
Chapter 12: Data Warehousing and SQL Server 2008 Enhancements 547
Data Model Schema
You can model your data model with two different types of schema.
Star Schema
As mentioned earlier, a data mart is a subject-oriented mini–data warehouse and will
have one or few fact tables surrounded by a relatively large number of dimensions.
When these are drawn on a piece of paper, the structure looks like a star, hence the
term star schema. The star schema is a simple model in which the denormalized
dimensions are connected to facts using foreign key relationships. This model has
become a building block in dimensional modeling. In a large data warehouse where
multiple fact entities can exist, you can imagine a multiple star schema model with each
dimension connected with several fact entities. Figure 12-1 shows a simple example of
a star schema data model.
FactSales
DimCustomer
DimSalesTerritory
DimDate
DimCurrency DimProduct
DimSalesPerson
Figure 12-1 A simple star schema data model