BI Tool Evaluation Criteria Rating Matrix -- anyone have one they've used in making a tool selection?

5100
33

Does anyone have a BI/BA tool Evaluation Criteria Rating Matrix or RFP they used in making a tool selection?


Several users submitted their BI matrix to share with the community. You can download them here.

If you have a matrix or RFP to share with our community of tech buyers, please email it to tools@itcentralstation.com

Anonymous avatar x30
Guest
As seen in
Logosasseenin

33 Answers

Gp nath li?1427467863
GP NathReal User

You can get a comprehensive comparison from Panorama website, we have used this, it is very informative

http://www.panorama.com/bi-comparison-tool/

Like (1)19 December 15
Andras belicza li?1414334055

First of all the Gartner - well structured in-depth analysis, forecasting

Like (0)15 December 15
93673352 7766 4460 8bc5 ffedd103b68b avatar
JhornberReal UserTOP REVIEWERELITE SQUAD

KLBaxter: Partly because we were focused mostly on the dashboarding side of the equation, and also because we already utilize a combination of SSRS and SAP Business Objects for canned and ad hoc reporting respectively, with no plans to replace those products in the near term. Had we been looking to find a single solution to cover most, or all our reporting and analysis needs, I'd likely have explored some additional vendors, especially because canned & distributed reporting is not one of Qlikview's strong suits. However, as I'm sure you and many others here have also come to realize, finding one vendor that is a leader in all things BI is a very lofty ambition. The phrase "Jack of all trades, master of none" comes to mind. ;)

Like (0)01 April 14
Kirsty baxter li?1414331025
KLBaxterReal User

Jhornber - was there a reason Cognos wasn't evaluated? All my initial scoping keeps leading me to QlikView but I'm determined to give the others a fair trial!

Like (0)01 April 14
93673352 7766 4460 8bc5 ffedd103b68b avatar
JhornberReal UserTOP REVIEWERELITE SQUAD

Ultimately Qlikview was selected, as it seemed to best meet our particular needs.

Like (0)31 March 14
Ddp 8148
Ariel LindenfeldCommunity Mgr

Thanks for sharing Jhornber! Which BI tool did you end up selecting?

Like (0)30 March 14
93673352 7766 4460 8bc5 ffedd103b68b avatar
JhornberReal UserTOP REVIEWERELITE SQUAD

I've posted my BI tools selection criteria here:
http://www.itcentralstation.com/articles/i-evaluated-business-objects-tibco-spotfire-tableau-qlikview-sisense-prism-and-pentaho

Let me know if you’d like to know anything else!

Like (0)30 March 14
Ddp 8148
Ariel LindenfeldCommunity Mgr

Jhornber - Can you share your criteria here with the community? I'm sure everyone will benefit from reading it.

Like (0)27 March 14
Anonymous avatar x30
Joanne MorganReal User

Yes, I would be interested in seeing your criteria.

Joanne.morgan@canforpulp.com<Joanne.morgan@canforpulp.com">Joanne.morgan@canforpulp.com>

Like (0)27 March 14
Anonymous avatar x30

Jhornber, it will be nice to share it.

Like (0)27 March 14
Anonymous avatar x30

Please share your thoughts...

Like (0)27 March 14
93673352 7766 4460 8bc5 ffedd103b68b avatar
JhornberReal UserTOP REVIEWERELITE SQUAD

I just completed an evaluation of (and ultimately a final selection from) several vendors and identified about a dozen high level evaluation criteria (using Gartner capabilities as a starting point, and adding those specific to our focus). We were primarily concerned with an efficient, competent dashboarding tool with a secondary focus on ad hoc analysis. If you're interested in seeing my criteria, let me know. Happy to share if it helps.

Like (0)27 March 14
Arnab li?1414330651
arnabReal UserTOP 20

We have noted that one single BI does not meet the requirements of large organisation. I would suggest some changes in the template shared.

Like (0)27 March 14
Larry keller li?1414328940
Larry KellerConsultantTOP 5POPULAR

Brevity is rewarded - the operative word is "report" and not analysis visual or otherwise

Like (0)27 March 14
Reviewer75321 li?1414329885

Hi, before responding to this question let me be open and declare a bias towards certain types of BI tools because I work for Tibco Spotfire, a data discovery and visualisation vendor.

Leaving product aside therefore lets establish some principals before starting down the road of a matrix - firstly, what needs to be in that matrix?
What do the business (and technical) users want to be able to do? There is a wide variety of capabilities across the vendors and building a matrix on what every vendor does is going to be a time consuming exercise to both build it and then administer it, especially if it is not all relevant to what the business needs.

For instance, if the key requirement is to burst out a pixel perfect report every Monday morning to 300 recipients then tools like Qlikview, Tableau, and Spotfire are unlikely to be your best bet.

If on the other hand you want to empower users to explore data themselves (in order to minimise the IT overhead), mash-up data from disparate systems, follow a train of thought to identify new threats and opportunities, and collaborate across the enterprise with colleagues on the insights you are discovering, and quickly prototype new dashboards - then the new generation of data discovery tools is likely to be more useful. That said, according to Gartner, this is where all of the traditional BI vendors are trying to take their products. They just haven't got there yet. So, timescales are important.

The fact is that extensive matrices are a tick box exercise, perhaps okay for a first cut, but largely a waste of time. All vendors (or their marketing departments) will try to put as many ticks in the product boxes as possible. Just because a vendor says their product can do something does not mean they do it very well and there are only two ways to find out the truth of it - buy it and build it, or talk to someone who has.
If you are clear about the capabilities you need (and I emphasise 'capability' not product 'functionality') your short list comes together pretty quickly. At that point, you start on the detail, and you will find your matrix more focused and easier to manage and you will be able to get to a conclusion a lot more quickly.

If you do pull down a generic matrix, use it as a guideline to getting to something specific that will meet the needs of your business, not someone else's a few years ago.

Like (0)27 March 14
Anonymous avatar x30
rsh0303Real UserTOP 10

Thanks for all the helpful comments. Please send me the full matrix from example 1. Thanks, Steve

Like (0)25 March 14
Anonymous avatar x30

Starting with customer acceptance criteria before going to a Evaluation scorecard would be my advise. Ask them what they want and go have a look on how they work. If you choose the "best" tool but the rampup for learning how to use it is too big, customer want simply reject. Usual they know what they want, you just Need to translate it into a Feature & Performance set.

Disclaimer: I work for Microsoft

Like (0)24 March 14
Anonymous avatar x30
Privilege MatizanadzoConsultantTOP 20

Quite an interesting discussion and eye opener.

Like (0)24 March 14
Rusty isaac li?1414329905
Rusty IsaacCommunity Mgr

Very useful, thanks.

Like (0)23 March 14
Ddp 8148
Ariel LindenfeldCommunity Mgr

Very interesting discussion! We've added several examples of BI matrices submitted by the community. See them here: http://www.itcentralstation.com/articles/bi-evaluation-criteria-matrix

Email matrix@itcentralstation to get your complete copy.

Like (0)23 March 14
Anonymous avatar x30
reviewer85482Real User

I would recommend Gartner's BI Magic Quadrant see http://www.gartner.com/technology/reprints.do?id=1-1QLGACN&ct=140210&st=sb . Hope this helps.

Like (0)23 March 14
Anonymous avatar x30
reviewer93888Real User

The problem with tool selection criteria is the way they are applied and the lack of any environment-specific filters. Check out Gartner and then go with what they see as the future.
I personally advocate Cognos for technically competent organizations. Government agencies do not fit this criteria.

Almost without exception, organizations(especially government orgs) do not have the expertise to implement BI tools properly. Microsoft had a terrible false start with Performance Point Server a few years back. They actually had people lined up to buy their BI tools and had to back out of the market until they got their tools matured. BI is not trivial.

To wring performance from BI tools requires a few people with very solid data modeling abilities and exceptional data/db skills. These people are hard to find and hard to keep.
Another thing that causes havoc when buying BI tools is any organization's structure. If the org has an ETL group, are they going to do the BI ETL or is the BI group going to do that also? Will your organization let the BI group build report-specific databases? Who will maintain the reporting databases; The enterprise or the BI group or both?

BI was crippled at a large aerospace company because the organization considered all databases to be equal. This meant that an enterprise warehouse counted one and the tiny, by comparison, BI report-specific cubes and stars were also counted as one each. The company bought Cognos without the ETL/DB building tool "Decision Stream/Data Manager" and suffered because of it. Finally after many years of failure or slow progress, it was acknowledged that BI requires its own databases to gather and stage the data. You cannot ask a meta-data modeling tool such as "Framework Manager" to do everything and you cannot ignore ETL. A BI environment will have dozens or hundreds of stars and cubes for specific purposes and report requirements.

I work at a government agency right now and would rate them a 1 on a scale of 1 to 10 for technical ability. There are sharp, up-to-date people but they are the exception. There are maybe 5 out of 100 with current skill sets.

Like (0)23 March 14
Anonymous avatar x30
reviewer93135Real User

I have one I used, but frankly I found that the "can you do this" feature-specific type matrix doesn't get to the real crux of why one tool is better than another.

A good deal of tool selection has to be based on what will fit with your company: are you aggregating data from multiple sources? If your database organized such that building views will be fairly easy? Are you migrating an existing reporting system for which you already have all the SQL and you don't want to re-invent the wheel? Do you have the resources to spend 6 months getting an "end-to-end" tool implemented? (And there are some good ones out there).

What about in-memory technology and performance? Caching? How does the cache get refreshed? Can I use my native SQL database, or do I have to build a star schema?

I find feature lists don't uncover these kind of implementation/design criteria.

Like (0)23 March 14
Anonymous avatar x30
boconne27Real UserTOP 20

I don't have access to an evaluation matrix. I worked for a very large company that had many BI applications at my disposal. There are a kit of considerations to be made. The most important considerations surround cost and functionality. After the budget is identified and the right applications are identified the next steps would be to gauge the support available from the vendor and then identify internal resources available to help in that effort. Some application licenses are more expensive but they also provide great support.

Like (0)23 March 14
Anonymous avatar x30
imabiproReal User

I would suggest checking in with Cindy Howson at the BI Scorecard. She typically has the tool area covered nicely. She could have just the info you are looking for.

Like (0)23 March 14
Nurul haszeli li?1414329996
Nurul HaszeliReal UserTOP 20

Yes. I can help out. You can send me a message.

Like (0)23 March 14
Anonymous avatar x30
Jonathan FordReal User

When it comes to BI, unless the business sees the need, it will forever say no!

When it does come around to saying yes, to make sure the win is certain, and the picture is rosy clear, I suggest the following approach:

Clearly define your requirements - (every step of the way):

In a rather common sense approach, satisfy the following questions/aspects:

1. What's my target?
How can the business benefit? Is it just for a specific unit/department, or the entire business? Can I take a modular approach - add on over time?

2. Functionality:
Who are my target audience - endusers, SMEs, hybrid (mix of both), executive management?
What kind of training will they need?
How many users - now and in the future? - plan for growth
Is the tool for internale use only, or both internal and external? Will it be accessed outside the firewall?
Do executives or salespeople need mobile BI capabilities?
Will dashboards, data visualization tools, etc. be implemented as well?
How important is real-time BI for operational workers, like customer service staff or tech support?
What kinds of ad hoc queries must the tool be able to handle?
Browser-based? Specific Application?

3. Licensing:
What pricing/licensing options are available?
How are support and maintenance contracts set up?
How many users will be querying the system - concurrent users, named users?
What kind of licensing scheme?

4. Off the shelf vs. customization:
What can the product do?
How does it meet our requirements?

5. Data makeup and placement:
Will you need to build a new data warehouse?
Heterogeneous environment... or separate data marts?
Centralized data, or all over the place?
Can you identify you data sources?
What is the technical environment that this BI software will need to integrate into (source systems, databases, preferred hardware platforms, etc.)?
How will your BI software integrate into our current environment?

6. Data volume and growth:
What are my data volumes today? What will they be in the near and far future?
What are your software update cycles like? How easy is it to add on new functions/features?


7. Training:
How will BI users and administrators be trained?
All-inclusive training class?
Or will you train only administrators and have the administrators teach the others?
8. Testing:
Do you want to test the product in your environment? Can we get a proof of concept?
Do you want a Proof of concept -- in your environment with your data? Or use Vendor's test environment?

9. Time frame for implementation?
Corporate deadline?
How long will implementation take? Will you need consultants or vendor services? In-house team?
Who's going to be on my BI team?
Who's driving the process (evaluation & purchase) - IT? Or by the business? Or both?

If you've made it this far, you've answered the biggest question - Do we need BI?

Now you can finish with the last 2 questions:

1. Should we write a request for proposal (RFP)?

2. What's my budget?

3. Who can we talk to to compare LOE (level of effort)? Find someone in the same field/industry.

Like (0)20 March 14
Anonymous avatar x30
Phillip MarshReal User

A review of the gartner magic quadrant is a good starting point. There also might be something for use on the TDWI website

Like (0)20 March 14
Reviewer74586 li?1414329857
reviewer74586Real UserTOP 20

Gartner probably does the best job of all the rating organizations, but Klas also has a pretty good reputation for healthcare products.

Like (0)20 March 14
Anonymous avatar x30
Luke_AlonsoReal User

This is a pretty comprehensive overview with a Magic Quadrant chart.

http://www.gartner.com/technology/reprints.do?id=1-1QLGACN&ct=140210&st=sb

Good luck.

Like (0)20 March 14
Anonymous avatar x30
Rikard MajcenReal User

Gartner BI-MQ-2014 can help

Like (0)20 March 14
Gustavo ramirez perez li?1414330186

The best and most known evaluation Matrix is Gartner´s Matrix. It also explains all the criteria and the strengths of the most important BI tools in the market

http://www.gartner.com/technology/reprints.do?id=1-1QLGACN&ct=140210&st=sb

Like (0)20 March 14
Larry keller li?1414328940
Larry KellerConsultantTOP 5POPULAR

Yes - but it addresses only data visualization and analysis vendors

Like (0)20 March 14
As seen in
Logosasseenin

Sign Up with Email