Sql Server Integration Services (SSIS) 2016 Review
SSIS 2016 - the Good the Bad, and the Ugly
SSIS is a unique ETL tool for a number of reasons. Most shops already own it but may treat it as a one-off free utility rather than as an enterprise ETL tool. Which is ironic given it beats out many of the supposed enterprise-class tools in areas such as logging, application adapters and parameter support. The lack of respect in our industry seems due to immature beginnings in its former "DTS" days and because it comes bundled "free" with Sql Server installation. But don't let that fool you into thinking it can't compete with the expensive ETL tools. I've used many ETL tools over the years and I'll take SSIS over any of them. I've now used SSIS exclusively for close to a decade and have no regrets.
SSIS is extremely flexible, extensible and integrated with many other Microsoft tools and a multitude of add-ons - both for purchase and for free. It's come a very long ways since its DTS days. It's incredibly cost effective, easy to learn the basics quickly (although like all ETL tools requires the traditional learning curve to get good at) and has an immense user base. There are also endless bits of quality shareware available that seamlessly plug-in and a wide variety of low priced vendor supported add-on products to fill in any out-of-the-box gaps (see my other review of MellissaData Data Quality MatchUp for SSIS). And if you can't find something you need and you know how to code C#.net you can extend the tool yourself. So unlike any other tool on the market, there's always a way to make something work with SSIS.
This comes from the fact that SSIS is merely Visual Studio code and comes with templates to add your own custom components. A large variety of pre-built shareware is available at the codeplex.com website and vendors such as MellisaData and Pragmatic Works provide sophisticated add-on components from advanced realtime monitoring to state-of-the-art data quality plug-ins. BIML shareware allows for automated ETL code generation based on XML templates. Some of the shareware available on Codeplex are very high value such as the MultipleHash component providing very sophisticated hashing to support CDC and SCD operations. If you don't like out of the box functionality (such as the SCD wizard which is largely worthless) then there's likely a worthwhile supported replacement by some vendor for a reasonable price. There's built in support for CDC from many database sources such as Sql Server and Oracle for no added charge which is unheard of among their competitors.
Adding plug-ins such as BiXpress, Task Factory and DocXpress from Pragmatic Works are relatively inexpensive tools that do some really amazing things. BiXpress is a MUST for providing real-time and historical monitoring of ETL including tracking package parameter and local variable value changes both in real-time and historical. I highly recommend MelissaData Matchup for SSIS - you'll never see any other data quality tool as easy to use as that one which seamlessly integrates with SSIS.
Unlike any other ETL tool - if you learn Visual Studio, you gain familiarity with an entire toolset. Its navigation and project organization is common to all other .net development. Along with Visual Studio you get all the source control plug-ins inherent with the tool such as TFS (aka. TFVC in VSO which is cloud based and free!), and Git.
2012 Version introduced very flexible parameter capability superior to most all other ETL tools. Project and package parameters integrate seamlessly into Sql Agent to provide step-level dynamic change runtime values such as source/target connections. They of course can be used with many other job schedulers albeit a little less tightly integrated.
Either via shareware or purchased products. Github provides a huge amount of free shareware code - some of which is very high quality. Vendors such as KingswaySoft and PragmaticWorks and many others provide multitude of inexpensive add-ons from adapters to enhanced components to connect to just about everything. I use SSIS to connect to Microsoft Dynamics CRM, Oracle Netsuite, on-prem Oracle, hosted source using SOAP calls, and Azure SqlServer as well as flat file loads and on-prem SqlServers.
Built in reports show package execution and messages. Logging can also be customized so only what is needed is logged. There is also an excellent logging replacement called BiXpress that provides both historical and real-time monitoring which is more efficient and much more robust than the built-in logging capabilities. And none of this requires custom coding to make it useful unlike many other ETL tools.
Just google if you don't know the answer - from youtube to blogs there's an incredible amount of information out there about this tool. I suspect far more than any other ETL tool.
List of good features...
You have to write push down join & lookup SQL to the database yourself via joins in the data flow source to get very high performance. That said, this is a common complaint for nearly all ETL tools on the market and those that offer an alternative such as Informatica offer them at a very expensive add-on price and even then don't work for all join situations. (My best practice is complex joins go into views of the data lake/landing area tables anyways so the queries are easily audible but I know there are deferring opinions here.)
No surprise here as this is a common issue with ETL SCD wizards. Requires custom/shareware enhanced wizard or one from 3rd party vendor to get good performance. This begs the question why Microsoft can't come up with a better wizard since it's pretty straight forward to code a dynamic merge as a workaround and someone wrote a much better shareware version. Pragmatic Works also sells a much better and supported version in their toolkit along with many other tools.
Here's a list of constraints or potential gotchas...
10. Logging significantly impacts performance. You can customize logging however starting with 2016 version.
SSDT is still 32 bit!
Yup...and you thought this was the year 2020 and everything is 64 bit. Apparently Microsoft doesn't know that yet. Combine that with its tendency to not release memory and its not difficult to hit out of memory errors when doing SSIS development.
Here we go with more not-so-pretty "features"...
Development environment and deployment wizard becomes unstable with larger projects
It is not unusual to get "out of memory" errors IF you use the default deployment wizard which is 32 bit on even medium sized projects. However there is a 64 bit version that eliminates this issue but you have to realize that using it isn't the default.
SSDT (the development tool) keeps grabbing more memory as you open new solutions so you have to exit at least once a day to free up memory. SSDT is unstable if you open more than about 30 packages at a time (such as when you're applying framework code to a bunch of packages - you have to limit how many you do at a time).
Containers that help group tasks have several very annoying bugs. For example, sometimes if you attempt to resize the container it will make the diagram tool move about wildly and out of control. A task within a container sometimes becomes detached and you can't get it back into the container. This is common with sequence containers when you try to add a new task. The new task seemingly disappears but is actually behind the container. The workaround is to cut and paste it in but you may start to scream before you figure that out.
And if that's not enough, here's a very special feature for you to enjoy...If you change the "show annotation" on a precedence constraint when the constraint is using a package parameter, Pennywise the clown slaps you in the face, laughs and then SSDT dies. There is fortunately a workaround. You can make the constraint something generic like 1==1, change the show annotation again and then put the real constraint into it. In the end you walk away with satisfaction knowing you found a way to slap it back.
Prior to 2012 not recommended!
Prior versions had many issues including debugging instability with large parent-child package call volumes and .com locking issues when running many parallel threads. It's largest drawback however was it was WAY overly complicated with its configuration XML file method of passing data between packages. That said, it was still superior to ETL tools that require passing parameters via just files (such as Informatica)! But these issues were resolved with 2012 when they introduced project and package parameters and they also improved memory management for parent/child package calls.