2012. 8. 9. 11:08

SSIS 데이터 이관 속도

  • ODBC Tasks  : 5 minutes 57 seconds 
  • ADO NET Tasks  : 11 seconds
  • OLEDB Task
    • fast load, table lock option: 5 seconds.
    • table or view option : 2 minutes and 21 seconds
    • sql command :  2.85 seconds.
  • SQL Server Destination
    • oledb destination : 2.5 seconds.
  • T-SQL Tasks
    • 프로시저 생성해서 작업 : 1.8 seconds!   -> 이건 같은 DB 네요.

 

http://www.mssqltips.com/sqlservertip/2684/importing-sql-server-data-using-ssis--which-option-is-fastest/


'Business Inteligence (SSIS)' 카테고리의 다른 글

SSIS::플랫 파일 가져오기 에러.  (0) 2012.02.09
SQL2008에서 SQL2000 DTS열기  (0) 2010.06.07
For컨테이너-무한루프  (0) 2010.06.03
스크립트task사용  (0) 2010.06.03
2012. 2. 9. 18:46

SSIS::플랫 파일 가져오기 에러.

플랫 파일을 Table로 저장 해서 가져오기를 할 때 특이한 값이 가져오거나, 일부 행을 가져오지 못 할경우가 발생한다. 
ssis 버그라고 하는데 SQL R2 버전이면서 win7 x64일 경우 발생한다.


아래는 버그에 대한 설명이고 sp2를 설치하면 된다고 한다.
혹은 플랫 파일 연결자의 속성중  TextQualifier  를 아무것도 없이 지운다. 

http://support.microsoft.com/kb/2576118 
 

here is a bug in SSIS when exporting data from SQL to a comma delimited text file where you specified no text qualifier.

This bug apparently only occurs when you develop the SSIS on a x64 win7 PC and copy the .dtsx file (windows explorer copy/paste) to network path of a x86 SQL server and schedule the job to run from SQL Agent on the same x86 SQL server.

When the SSIS runs, the text file is written out containing text qualifier = "“_x003C_none_x003E".

If you look at " _x003C_none_x003E", it actually means <none>.  x003C = "<" and x003E = ">".

 

If you go into the SSIS package, double-click in the connection manager section to open the flat file connection manager editor and try to clear the text qualifier removing the <none>, the <none> value get added back in.

The only work-around is to NOT open the flat file connection manager editor, but instead make the change using the property window and clear out any value in the TextQualifier field.

 

Other similar problems occur when you actually want to put a real value in the text qualifier.  For explain if you select the double-quote as the text qualifier and copy to a x86 server and run, you end up with a text file containing the value " _x0022" around each field instead of a double quote.

 

In my mind this is a serious bug, I did some research and other people have been having this same issue.

 

FYI, the SQL server is currently SQL2008 with SP2.  I will try to get it updated to SP3/4 this weekend to see that will help.

 

FYI2, when I am developing the SSIS using VS2008 and my local test/run is done through visual studio on my Dev PC (x64).  Everything works fine this way.  I do not have a instance of SQL on my machine so I did not test it running from SQL Agent on my PC.

 

'Business Inteligence (SSIS)' 카테고리의 다른 글

SSIS 데이터 이관 속도  (0) 2012.08.09
SQL2008에서 SQL2000 DTS열기  (0) 2010.06.07
For컨테이너-무한루프  (0) 2010.06.03
스크립트task사용  (0) 2010.06.03
2010. 6. 7. 23:43

SQL2008에서 SQL2000 DTS열기

SQL Server 2008에서는 DTS 패키지에 대한 디자인 타임 지원을 설치하지 않습니다. SQL Server 2008 도구를 사용하여 DTS 패키지를 열어서 보려면 다음 절차에 설명된 대로 이러한 디자인 타임 지원을 다운로드하고 설치해야 합니다.
참고:
다운로드할 수 있는 DTS 패키지 디자이너는 SQL Server 2008 도구를 사용하지만 디자이너의 기능은 SQL Server 2008용으로 업데이트되지 않았습니다.

데이터 변환 서비스 패키지를 위한 디자인 타임 지원을 설치하려면 ·

         인터넷 브라우저에서 Microsoft SQL Server 2005용 기능 팩 페이지를 열고 Microsoft SQL Server 2000 DTS 디자이너 구성 요소를 다운로드하여 설치합니다.
기본적으로 32비트 컴퓨터에서 SQL Server 2000의 이진 파일은 %ProgramFiles%\Microsoft SQL Server\80\Tools\Binn 폴더에 있으며, SQL Server 2008의 이진 파일은 %ProgramFiles%\Microsoft SQL Server\100\Tools\Binn 폴더에 있습니다. 이러한 기본 위치를 사용하면 PATH 환경 변수에서 SQL Server 2000 이진 파일의 경로가 SQL Server 2008 이진 파일의 경로 앞에 올 수 있습니다. 이러한 경우 DTS 디자이너를 사용하려고 하면 오류 메시지가 표시될 수 있습니다. 이러한 오류를 해결하려면 다음 절차에 설명된 대로 디자이너에 필요한 파일을 새로운 위치로 복사합니다.
참고:
64비트 컴퓨터의 경우 다음 절차에서 %ProgramFiles%를 %ProgramFiles(x86)%로 대체하십시오.

SQL Server Management Studio에서 DTS 디자이너를 사용할 수 있도록 하려면

1.       %ProgramFiles%\Microsoft SQL Server\80\Tools\Binn 폴더에 있는 SEMSFC.DLL, SQLGUI.DLL 및 SQLSVC.DLL 파일을 %ProgramFiles%\Microsoft SQL Server\100\Tools\Binn\VSShell\Common7\IDE 폴더로 복사합니다.
2.       %ProgramFiles%\Microsoft SQL Server\80\Tools\Binn\Resources 폴더에 있는 SEMSFC.RLL, SQLGUI.RLL 및 SQLSVC.RLL 파일을 %ProgramFiles%\Microsoft SQL Server\100\Tools\Binn\VSShell\Common7\IDE\Resources\%lang_id% 폴더로 복사합니다. 예를 들어 미국 영어이면 lang_id 하위 폴더는 "1033"입니다.



Business Intelligence Development Studio에서 DTS 디자이너를 사용할 수 있도록 하려면

1.       %ProgramFiles%\Microsoft SQL Server\80\Tools\Binn 폴더에 있는 SEMSFC.DLL, SQLGUI.DLL 및 SQLSVC.DLL 파일을 %ProgramFiles%\Microsoft Visual Studio 9.0\Common7\IDE 폴더로 복사합니다.
2.       %ProgramFiles%\Microsoft SQL Server\80\Tools\Binn\Resources 폴더에 있는 SEMSFC.RLL, SQLGUI.RLL 및 SQLSVC.RLL 파일을 the %ProgramFiles%\Microsoft Visual Studio 9.0\Common7\IDE\Resources\%lang_id% 폴더로 복사합니다. 예를 들어 미국 영어이면 lang_id 하위 폴더는 "1033"입니다


 


1. SQLServer2005_DTS.msi 설치

http://www.microsoft.com/download/en/details.aspx?displaylang=en&id=11988


2.  파일 복사

  FROM : C:\Program Files\Microsoft SQL Server\80\Tools\Binn

   TO : C:\Program Files\Microsoft SQL Server\100\Tools\Binn

3. SSMS 재 시작



.

'Business Inteligence (SSIS)' 카테고리의 다른 글

SSIS 데이터 이관 속도  (0) 2012.08.09
SSIS::플랫 파일 가져오기 에러.  (0) 2012.02.09
For컨테이너-무한루프  (0) 2010.06.03
스크립트task사용  (0) 2010.06.03
2010. 6. 3. 23:54

For컨테이너-무한루프

 

Question
FOR 문을 이용해서 무한적으로 반복해서 실행하고 싶은 작업이 있습니다.
그러면서도 작업과 작업 사이에 대기시간을 5초 정도 주고 싶습니다.
즉, 매번 반복되면서 5초에 한번씩 A 라는 곳에서 select 해서 B 라는 곳으로 Insert 하는 작업이 필요합니다.
대기시간 주는 것은 여기 사이트에 예시도 있어서 가능 할 것 같은데 계속 실행되면서  select 하려면 어떻게 해야 할까요?
작업할때 이렇게 대기와 loop가 필요할 때가 있다 JOB으로 등록했을때는 주기가 1분 마다 밖에 안되기 때문에 초 단위 대기가 필요할때는 이렇게 해야한다.

 

다음과 같이 For루프 컨테이너를 추가합니다.

 

 

 

루프 편집기에서 다른 조건은 설정할 필요 없이 EvalExpression 속성 값에 true 입력합니다.

 

 

이렇게 설정하는 것은 쿼리에서

 

WHILE (1=1)

BEGIN

 

END

 

비슷한 형태입니다.

 

이제, For 루프 컨테이너 안에 필요한 작업들을 설정해서 넣고선, 5 동안 대기하는 것을 구현해야 합니다.

 

이전에 올린 글에 FOR 루프 컨테이너를 이용해서 일정 시간 동안 대기하는 것을 구현한 것이 있습니다. 하지만 방법은 쓰지 마시기 바랍니다. (CPU 과도하게 높이 올라갑니다.)

http://www.sqlleader.com/mboard.asp?exec=view&strBoardID=SS2005SSIS&intSeq=523

 

대신, SQL 명령을 이용하는 것이 가장 바람직할 같습니다.

 

작업들을 구성한 , 가장 마지막 부분에 SQL 실행 작업을 추가한 ,

 

 

DB 연결을 설정하고, 다음과 같은 간단한 명령을 사용합니다.

) 5 동안 대기를 해야 경우

                  WAITFOR DELAY ’00:00:05’

 

 

 

 

 

루핑 마다 마지막 단계에서 5초간 대기하는 무한 루프가 만들어졌습니다.

 

 

이 글은 스프링노트에서 작성되었습니다.

'Business Inteligence (SSIS)' 카테고리의 다른 글

SSIS::플랫 파일 가져오기 에러.  (0) 2012.02.09
SQL2008에서 SQL2000 DTS열기  (0) 2010.06.07
스크립트task사용  (0) 2010.06.03
Debugging and Logging in SSIS  (0) 2010.06.03
2010. 6. 3. 23:53

스크립트task사용

Using the script task to dynamically build SSIS package variables

Written By: Hal Hayes -- 2/27/2008

 

Problem
One of the advantages of using SSIS is the ability to dynamically create tasks that can take different elements and manipulate them in code instead of having to hard code the package to do only one task. In a previous tip we looked at how to use expressions to dynamically build an output file: "Dynamic Flat File Connections in SQL Server Integration Services".  In this tip, we will show you another way to do this using the script task.

Solution
Basically what we will be doing is generating an output file name using a package variable, representing the location and name of an output file and using the Script Task to dynamically build the output filename.

Using this method, modifying variable information is not just restricted to output file information. You can use this method to generate input file locations, complex queries that can be used in your Execute SQL tasks, data for inserts, etc.  The Script Task is a powerful component that can offer developers capabilities not covered in other Control Flow items.

In this solution we will address the simplest case of using a Script Task to dynamically modify a variable that is assigned to an output flat file connection so that the file is saved with a name and location of our choosing.


Create Package

To start, create a new Integration Services project using the SQL Server Business Intelligence Design Studio (BIDS).

  • Add a Data Flow Task to your Control Flow surface
  • In the Connection managers, create an OLE DB connection to the AdventureWorks database
  • Double click the Data Flow task and add an OLE DB Source, set it to any table or view (in this case I used the Purchasing.vVendor view)
  • Add a Flat File Destination and connect the two objects with a Data Source Path (green arrow) as shown below
  • Open the Flat File Destination Editor and create a new connection, choosing the output file type (delimited) with a name of "CSV File Output".  Also, choose a filename which can be anything at this point, this will be dynamically generated again below, but for now you need some filename to complete this step.

 


Add Package Variables

Move back to the Control Tab and right click on any area, but not on an object and select Variables. (We do this to make sure that our variables, which we will design in the next step, are scoped to the entire package.)  Another way to do this is from the menus by selecting View | Other Windows | Variables.

Now add the following variables (within scope of the entire package).

Name Description Type Value
mDirectory Directory for file output. String C:\SSIS\Output
mFilePrefix File name (descriptive name for your organization). String Vendors
mOutFileName Resulting output file name and path that will be used with the flat file connection. Set a default value (use the current filepath used by the flat file connection object). String C:\SSIS\Output\Vendors.txt

Note: Our package is called "WithScripting", so here is an example of our variables after they have been setup.


Adding the Script Task

Now add a Script Task to the Control Flow above the Data Flow Task. Connect the two objects (from Script Task to Data Flow Task) with a precedence constraint (green arrow) as shown below.

Open the Script Task Editor. We will use the defaults for this item, but we need to add the following to the Script tab.By setting our variables in this property window, we make them available to the Script environment. If this step is not done the package variables can not be used with the script task.  Note: the variable names have to match the exact same names that were used, the variables are case sensitive.

ReadOnlyVariables mDirectory, mFilePrefix
ReadWriteVariables mOutFileName

Here is an example of our script variables setup.

Next select the “Design Script...” button. This opens the Visual Studio for Applications (VSA) design environment. By default, you will have a ScriptMain class with a single method, Main.

Replace the Main method with the following text:

Public Sub Main()
'
        ' MSSQLTips: Modify Output File Location
        '
        Dim sFinalFileNameAndPath As String
        Dim sDirectory As String
        Dim sFilePrefix As String

        sDirectory = Dts.Variables("mDirectory").Value.ToString

        If Not (sDirectory.EndsWith("\")) Then
            sDirectory = sDirectory + "\"
        End If

        sFilePrefix = Dts.Variables("mFilePrefix").Value.ToString


        sFinalFileNameAndPath = sDirectory + sFilePrefix + "-" + Month(Now.Date).ToString + Year(Now.Date).ToString + ".txt"

        Dts.Variables("mOutFileName").Value = sFinalFileNameAndPath

        Dts.TaskResult = Dts.Results.Success
End Sub

 

Here is a snippet of what our script looks like after pasting the above code.

  

Discussion of VB.Net coding is beyond the scope of this article, but a brief explanation follows. The above code reads the directory information from the SSIS variable into a string variable, and ensures that it ends with a “\” character for the formatting of our full file name and path. The file prefix is also captured in a string variable. We craft our output path and file name in a string variable, dynamically adding the month number and year to the file name. The file extension of “.txt” is added to the end of the file name.

Finally the SSIS variable, mOutFileName, is set to the full file path. Remember that we set this variable as read-write in the Script task. At the end, the script closes by setting the DTS.TaskResult value to success. The setting of this value is required for proper operation of the script (the value could also be set to failure).

Now you can close the VSA environment.


Modify the Flat File Connection

  • Select the flat file connection object "CSV File Output"
  • Right click and select Properties
  • Select the Expressions property and click on the ellipse "..."
  • In the Property Expression Editor, select the ConnectionString property as shown below.

  • Next select the “…” button to bring up the Expression Builder interface shown below
  • Set the expression to @[User::mOutFileName] by dragging the value down to the Expression window from the Variables window as shown below and select OK.


That is it. Now you have dynamically modified your filename by adding the month and year to the end of it. And you are saving it in the location of your choosing as defined by the mDirectory variable. 

So for our example if this was run on February 27, 2008 the file created would be "C:\SSIS\Output\Vendors-22008.txt".

 

Next Steps

이 글은 스프링노트에서 작성되었습니다.

2010. 6. 3. 23:53

Debugging and Logging in SSIS

 

 

It’s been more than two years since SQL Server 2005 shipped with SQL Server Integration Services (SSIS). However, many companies still haven’t converted their DTS packages to SSIS, possibly because the migration process can be painful or they don’t have the time to learn about a new product.

Those of you who have made the conversion know that SSIS definitely isn’t the “next version of DTS.” Debugging and logging are just two of many areas that have undergone a complete overhaul in SSIS. Let’s take a high-level look at SSIS’s debugging and logging capabilities. If you’re among those who are still using DTS, prepare to be impressed.

Debugging Packages During Development


SSIS is far ahead of DTS in the area of debugging, which is the process of identifying errors that prevent a package from being executed or producing the desired results. In DTS, debugging typically involves using MsgBox statements or the VBScript Stop command to simulate a breakpoint in scripts. Such statements had to be removed from the package before it was put into production. In contrast, debugging is built into SSIS, and nothing has to be removed when the package is moved to a production environment.

The problem with SSIS’s debugging tools is that they aren’t consistent across all types of tasks, so knowing when to use what tools can be a big help. Let’s look at SSIS’s debugging capabilities at the package level, followed by debugging within Control Flow tasks and Data Flow tasks.

Package-Level Debugging

During the development process, SSIS provides red or yellow icons within the SSIS designer that tell you when something is inherently wrong with the package. To view the message associated with an icon in a collection of packages, hover your mouse over the icon, as

Figure_01.jpgFigure_02.jpg

 

 

shows. These messages are typically related to data-source connections or data-type problems. Because it can take time to sift through the messages in the Output pane (which is located below the Error List pane in the SSIS designer) during debugging to see what caused a package to fail, SSIS lets you click the Progress tab from the designer to view an outline structure of the package, as Figure 2 shows, and see where the failure took place.

When you debug a package in a Business Intelligence Development Studio environment, you can see which task is running and how far it’s progressed by the background color of the task: Yellow indicates that the task is running, green indicates that the task completed successfully, and red indicates that the task completed with errors. Next to Data Flow tasks, you also get a count of the rows that have been processed by the task. You might notice that SSIS often runs multiple tasks simultaneously, whereas DTS runs only one task at a time. The pipeline nature of SSIS is one of the core architectural differences between it and DTS.

 

Debugging Control Flow Tasks


Control Flow tasks control the flow of the package. SSIS’s debugging tools for Control Flow tasks closely resemble those available in any respectable development environment. Breakpoints and the Debug windows can be especially helpful in debugging Control Flow tasks.

Breakpoints tell SSIS to pause execution at the indicated point in the package. When processing encounters a breakpoint in Debug mode, processing pauses, and the Debug windows give you access to additional information such as variable values, call stacks, and status messages. You can set breakpoints at the package, container, task, or Script task level. You can even set breakpoints to pause after a given number of encounters with an event, such as on the tenth iteration of a task in a For Loop construct. Figure 3 shows how you can interrogate an object to reveal its property values during a breakpoint session.

I find the Locals window, which is accessible from the Debug/Windows/Locals menu item, to be the most useful of SSIS’s Debug windows. It displays values for all the variables in the package and even lets you modify variable values, which can be useful when skipping ahead to a later point in the package or simulating certain conditions during package development.

 

 

 Figure_03.jpg

 

Debugging Data Flow Tasks

Data Flow tasks control how and when data is manipulated in the package. The primary debugging tool available in Data Flow tasks is the data viewer. I use the data viewer during the development of Data Flow tasks to see what the data looks like in the pipeline as it flows from one task to another—usually just before the task that actually loads the data into its destination.

You can add a data viewer between two Data Flow tasks by right-clicking the connector between the tasks, selecting Data Viewers, and then selecting the data viewer you want to use. You can view the data in a grid, chart, scatter plot, or histogram. I usually view data in a grid, as shown in Web Figure 1, but I recommend playing with all four data viewers to get a feel for when you should use each one. Although I typically remove my data viewers before deploying a package in production, you don’t have to do so.

You can modify columns displayed by a data viewer either as you set up the data viewer or after setup. To modify a data viewer after you’ve created it, right-click the connector, select Data Viewers, highlight the data viewer, then click Configure.

Another Data Flow task debugging technique that I use frequently during package development and debugging is the RowCount task. The RowCount task relates only to Data Flow tasks. The RowCount task isn’t usually billed as a debugging tool but can be quite useful as one. The RowCount task simply counts the rows passed through the pipeline and puts the final row count into a variable. I almost always use the RowCount task as my initial data destination because it serves as a way of examining the data, via a data viewer, without actually loading the data anywhere. Because the Row- Count task carries no measurable overhead, it can also be used for baselining or to diagnose performance problems. For more information about using the RowCount task, see the Microsoft white paper

Integration Services: Performance Tuning Techniques” (www.microsoft.com/technet/prodtechnol/sql/2005/ssisperf.mspx). 

이 글은 스프링노트에서 작성되었습니다.

2009. 7. 27. 10:57

SSIS::Foreach 이용한 장비별 동일 로직 반복 - ADO 열거자

DBA 작업을 하면서 장비별 동일 정보를 관리용으로 수집하고 싶은 경우가 종종 있게 된다.

연결 정보만 틀리고 동일한 로직을 호출하면 되는데 여러개의 데이터 흐름 작업을 만들고, 여러 연결 관리자를 만들게 되면 관리상의 이슈로 힘들어 지게 된다.

 

또한, 간혹 발생하게 될 장비의 IP 변경으로 인해 패키지를 다시 빌드해야 하는 단점도 발생하게 된다.

이런 점을 해결하기 위해 DB의 연결 정보를 파일로 처리하여 Foreach 루프 컨테이너의 ADO 연결자를 사용하여 장비별 동일 로직을 실행하는 예제를 만들어 본다.

 

- 한대성 님의 자료를 도움 받음 -

 

 

1. 관련 변수들을 선언한다.

object 변수와 파일에서 connection string 을 읽어올 변수는 필수이다. (obj_conn, file_conn)

 

 

 

2. conn_file01.txt 파일을 만들고  연결 스트링 정보를 입력한다.

연결정보에 관한 정보는 시스템에 맞게 입력

 

Data Source=ip 정보;User ID=;PWD=;Initial Catalog=db명;Provider=SQLOLEDB.1;Persist Security Info=True;Auto Translate=False;

 

 

 

3. 새 플랫 파일 연결관리자를 만들고 2번에서 입력한 파일을 읽을 수 있게 처리한다.

 

 

 

 

4. 플랫 파일 원본을 선택한 후 파일 연결관리자를 입력해서  connection string이 열로 출력되어 레코드 집합 대상을 만들 수 있게 처리해 준다.

 

 

5.  레코드 집한 대상에 VariableName 에 object 변수를 선택하여 파일에서 읽어온 정보가 해당 변수에 담아지도록 처리한다.

 

 

6. 파일 출력열이 맞는지 확인 한다.

 

 

 

7.  Foreach 루프 컨데이터를 이용해서 불러들인 연결 정보 만큼 동일 로직을 수행하게 작업 처리한다.

 

 

 

8.  ADO 연결자를 선택하고 원본 변수에 데이터 흐름 작업에서 connection string 정보를 담은 변수를 입력한다. 이 변수의 row 수 만큼 loop 작업이 실행되게 된다.

 

 

 

9.  ForEach 컨테이너에서 변수 매핑 을 클릭하고 출력되는 열을 변수에 담는다.

 

 

10.

담은 변수를 연결 관리자를 하나 만든다음에 실행된 DB의 커넥션 변수에 해당 변수를 매핑해 준다.

이로써 DB에 동적으로 연결 정보를 받아 작업이 실행될 수 있다.

 

 

 

 

11 SQL 실행작업을 하나 선택하고 DB를 연결관리자의 select @@servername을 실행하게 하고 출력된 결과를 스크립트 작업을 통해서 메세지 박스로 확인하는 작업이다.

 

해당 장비로 접속하여 실행되는 모습을 볼 수 있다.

 

 

'Business Inteligence (SSIS)' 카테고리의 다른 글

SQL2008에서 SQL2000 DTS열기  (0) 2010.06.07
For컨테이너-무한루프  (0) 2010.06.03
스크립트task사용  (0) 2010.06.03
Debugging and Logging in SSIS  (0) 2010.06.03