Question : SSIS ETL package to update/insert staging source to destination ?

Hello,

Could someone please help me here, new to SSIS and ETL stuff, I have raw data being staged to a SQL table, which I then want to move to our production table and update/insert the record set based on a unique key.

The staging table has a row "SEQ" that identifies the record set, based on the SEQ I want to compare the destination table and if the SEQ exists, update the whole record or insert new otherwise.

I am using SSIS to acomplish this as per the image attached.

1. Connect OLE DB Source
2. Lookup transformation Editor, and configure error output to ignore failure.
3. Conditional Split
      - Insert Record Condition: ISNULL(Dest_SEQ)
      - Update Record Condition: (Dest_TYP != TYP) || (Dest_SALEDATE != SALEDATE) || (Dest_INVOICED != INVOICED) || (Dest_WEEKDATE != WEEKDATE) || (Dest_EWK != EWK) || (Dest_ZWK != ZWK) || (Dest_IY != IY) || (Dest_WK != WK) || (Dest_ROMO != ROMO) || (Dest_ROYR != ROYR)

4. New Reords go to OLE DB Destination
5. Updated records go to OLE DB Command as such:

UPDATE dbo.sales
SET
TYP = ?,
SALEDATE = ?,
INVOICED = ?,
WEEKDATE = ?,
EWK = ?,
ZWK = ?,
IY = ?,
WK = ?,
ROMO = ?,
ROYR = ?
WHERE SEQ = ?

The staging table has about 100 columns with data, so I am not sure how appropriate it is to define a Update Record condition for each one of those, perhaps my update record will be very expensive as far as resource when we are talking about 100,000 updates..

Is there a way I can change this so that in a conditional split (or using something else) to check if staging's SEQ exists in destination, if it does, update the whole record.

The above works, but ideally should I be creating a primary key on my SEQ table, and then doing an update/insert based on if the primary key exists or not, how could I accomplish this?

Right now, if I set the SEQ as a primary key on my "sales" destination table, and re-run my SSIS I get an error:

An OLE DB record is available. Source: "Microsoft SQL Native Client" Hresult: 0x80004005 Description: "Violation of PRIMARY KEY constraint 'PK_sales_1'. Cannot insert duplicate key in object 'dbo.sales'.".

But the table is empty, why can't it insert the SEQ key?

Thanks for all your help.

The guide I followed was: http://vsteamsystemcentral.com/cs21/blogs/applied_business_intelligence/archive/2007/05/21/ssis-design-pattern-incremental-loads.aspx

         
           
             1:
2:
3:
4:
5:
6:
7:
8:
9:
10:
11:
12:
13:
14:
15:
16:
17:
18:
19:
20:
21:
22:
23:
24:
25:
26:
27:
28:
29:
30:
31:
32:
33:
34:
35:

           
           
          Staging table layout:

USE [Sales]
CREATE TABLE [dbo].[stg_sales](
	[SEQ] [int] NOT NULL,
	[TYP] [varchar](1) NULL,
	[SALEDATE] [varchar](50) NULL,
	[INVOICED] [varchar](50) NULL,
	[WEEKDATE] [varchar](50) NULL,
	[EWK] [varchar](50) NULL,
	[ZWK] [varchar](50) NULL,
	[IY] [varchar](50) NULL,
	[WK] [varchar](50) NULL,
	[ROMO] [varchar](50) NULL,
	[ROYR] [varchar](50) NULL,
	[YRTD] [varchar](50) NULL,
GO

Destination Table Layout:

USE [Sales]
CREATE TABLE [dbo].[sales](
	[SEQ] [int] NOT NULL,
	[TYP] [varchar](1) NULL,
	[SALEDATE] [varchar](50) NULL,
	[INVOICED] [varchar](50) NULL,
	[WEEKDATE] [varchar](50) NULL,
	[EWK] [varchar](50) NULL,
	[ZWK] [varchar](50) NULL,
	[IY] [varchar](50) NULL,
	[WK] [varchar](50) NULL,
	[ROMO] [varchar](50) NULL,
	[ROYR] [varchar](50) NULL,
	[YRTD] [varchar](50) NULL,
GO

           
         

Attachments:

SSIS.JPG (30 KB) (File Type Details)

Current SSIS Layout

lookup-transorm.jpg (89 KB) (File Type Details)

Lookup Transform

conditional-split.jpg (57 KB) (File Type Details)

Conditional Split

Answer : SSIS ETL package to update/insert staging source to destination ?

Instead of using a conditional split, you might try a Slowly Changing Dimension; once you walk through the wizard, it will create inserts or updates depending on whether your data exists already in the table or not. Plus, you don't have to worry about creating your own queries. :)

hth

valkyrie_nc

Question : SSIS ETL package to update/insert staging source to destination ?

Answer : SSIS ETL package to update/insert staging source to destination ?

Instead of using a conditional split, you might try a Slowly Changing Dimension; once you walk through the wizard, it will create inserts or updates depending on whether your data exists already in the table or not. Plus, you don't have to worry about creating your own queries. :)hthvalkyrie_nc

Instead of using a conditional split, you might try a Slowly Changing Dimension; once you walk through the wizard, it will create inserts or updates depending on whether your data exists already in the table or not. Plus, you don't have to worry about creating your own queries. :)

hth

valkyrie_nc