WhereScape RED Room for Improvement

GaryM
Data Architect at a non-profit with 1,001-5,000 employees
In my opinion, the tool requires a much different architectural approach to be effective. It has fundamental architectural flaws that prevent it from being a practical solution for even moderate volumes of data. The largest issue is that it violates the most basic database data-movement physics through its reliance upon the infamous RBAR or "row by agonizing row" cursor operations within Tsql. In contrast, efficient database apply operations to whole sets of data rather than row by row achieved through Sql joins and set based queries. Note that this is a common problem with code generators as they end up trading scalability for flexibility. Ironically this also makes them unable to take advantage of the much touted "push down" operations that is one of their big selling points. This tool avoids the traditional lookup ETL lookup cache component issue (sometimes referred to as blocking transactions) - that much is true. However it ends up doing something similar but just within the database and with no easy way to override it. Experienced ETL developers know not to do any blocking transactions or RBAR if they can possibly help it. And with traditional ETL tools you either take advantage of built-in push-down capabilities or pre-join the data in views or data sources to avoid this kind issue. Other constraints include... - Repository only supports a single target database - No support for change data capture or delta detection - that must be custom coded - Must be used to stage data or it requires a redundant copy of staged data. This ended up more than double our storage - Relies 100% upon target database server with no ability to scale ETL to separate server - Relies 100% on the target database engine to do all the work (along with everything else the database must do like reporting) - Cannot acquire data from sources other than flat files. It relies upon the use of linked servers (which many shops don't allow) or calling an ETL tool such as SSIS in the background for data movement. In that case you're still relying on an ETL tool for the ELT portion so then you're actually using two different tools - Wherescape plus a 3rd party ETL tool instead of just one tool. Specific list of areas for improvement: * Performance (this is the area that needs the most improvement) * Efficient storage utilization * Logging * Count validations (data movements don't validate counts which is 101 basics to prevent data leaks) * Scheduler reliability and transparency * Ability to separate data acquisition/staging processing from target schema population * Providing automatic record count validations * Support for change data capture * Support for in-line transformational procedure language that extends beyond database procedures * Documentation lineage accuracy and completeness * Support for sharing metadata repository across multiple targets * Improvements in staging architecture requiring fewer copies of source data * Supporting transparency into post-gens * Providing encapsulated pre and post events for custom code that doesn't get wiped out when you re-generate supported code from the tool. * Better support for task orchestration in a top-down way that can be easily understood. * Support for direct parameter passing from job schedulers. View full review »
Kevin Marshbank
BI Architect at a healthcare company with 5,001-10,000 employees
Since WS supports Data Vault 2.0, I would like to see automated support for hash keys on the appropriate Data Vault tables. WS does have a table trigger that can be manually added to the WS repository that will accomplishes this but it isn’t built in to the IDE. I would like to see multi-database (creating/storing tables in different databases based on type, e.g. staging, data mart, etc.). Similarly to the hash keys, it can be accomplished (I'm using multiple databases) it just requires making a manual entry in a RED repository table for each database. Again, having it officially built in to the IDE would be great. The only other thing I would note is that the IDE isn’t as intuitive as I would like. Like most tools, after several weeks of use it makes a lot more sense…but for new users there will be a moderate ramp up time. View full review »
Ken Flerlage
Business Intelligence Architect at a university with 1,001-5,000 employees
As with any product, there are things that can be improved, but most are fairly minor, including things like greater flexibility in ordering job tasks. I’d also like to see increased support for NoSQL data sources. View full review »
Jeff Christen
Manager Data Warehousing at a university with 10,001+ employees
I would love to see a GUI interface for defining dependencies between build processes. RED provides a spreadsheet like interface for defining the dependencies between builds. Once the dependencies are defined, RED can produce a nice dependency diagram to give a visualization of the dependency tree. It would be easier to define complex dependency relationships if the dependency diagram were interactive. Our legacy ETL tool provided this GUI dependency definition via a drag and drop diagram which was very useful. The solution provided by WhereScape does work, and the dependency diagrams generated are helpful. It would just be nice to have the ability to define dependencies via a diagram, since dependencies relationships are much easier to understand via a diagram. View full review »
John Stassi
Senior Director, Software Development at a consultancy
Self-Documentation capabilities while good, are an area we would like to see continued improvement. View full review »
Kenneth Edwards
Data Technology Analyst at a tech services company with 1,001-5,000 employees
There are some newer features that haven't been included in the training materials yet so they're a little outdated (although the help documentation is very good). There are also some more complex data transformations that aren't supported and so need to be done in multiple steps. View full review »
Jim Mihalick
Sr. Technical Consultant/Architect with 11-50 employees
Here are some things that could improve the overall experience: * A tool to diagnose what is missing from the environment for WhereScape to install successfully. * Improve the object renaming ability (it works, but it could be more automated). View full review »
Daniel Sprague
Director Information Systems & Software Development at a healthcare company with 501-1,000 employees
Jobs cannot be deleted via the deployment package. When deploying from dev to QA or production, a job has to be retired. The job has to be manually removed from the target environment. View full review »
Chris S
Software Developer at a healthcare company with 1,001-5,000 employees
The scheduled jobs which are run by the WhereScape scheduler seem to be a strangely separate animal. Unlike all other WhereScape objects, jobs cannot be added to WhereScape projects. Also, unlike all other objects, jobs also cannot be deleted using a WhereScape deployment application. View full review »
Dave Wells
Sr. Research Analyst - Data Management Practice Head at a tech consulting company with 51-200 employees
* Data discovery would be more powerful with machine learning features * Big data functions could include some of the features of the advanced self-service data preparation tools View full review »
Angel Johnson
Director, Corporate Systems with 5,001-10,000 employees
A more robust support center. It has been a bit difficult to find solutions to problems that are out-of-the-box. It would be nice to have user groups available on their website. The ability to execute SSIS projects within WhereScape would be nice because we have a lot of packages that are too cumbersome to recreate. View full review »

Sign Up with Email