2018-07-22 09:41:00 UTC

What's the best way to trial reporting tools?


We all know that it's important to conduct a trial and/or proof of concept as part of the buying process. 

Do you have any advice for the community about the best way to conduct a trial or PoC? How would you conduct a trial effectively? 

What mistakes should be avoided?

Guest
99 Answers
ResellerTOP 5LEADERBOARD

Hi Rhea,
This is a great question. Just like Alberto, this is my opinion, so please take it with a grain of salt. Here are some tips I would suggest for any group looking at purchasing reporting tools.

* Define your current business challenge - What kind of decisions is this reporting supposed to support? ...number of users involved? ...how are they organized into teams? ...what are the steps in the business process?

* Decide on some metrics in key categories - If you decide on some metrics before looking at all the alternatives, it can help make comparison conversations more clear. Here are some examples

A) Ease of Use - {Training required, Usable from Excel, Usable from Web, Usable from Mobile, ...}
B) Aesthetics - {Visuals, Responsiveness, Look and Feel}
C) Functionality - {Ease of development, Business rules, change management, Output}
D) Support - {Community Support, Vendor Support, Partner Support, Inhouse Support}
E) Effectiveness - {Solves the business problem, Flexibility for change, Transparency, Accountability}

* Ask for a proof of concept - Vendors are happy to build a proof of concept with your own data. They know showing you your own data will prove (or reject) the idea of the software helping the organization. After the vendor builds a proof of concept, ask multiple users from the organization to play with it.

...as for mistakes to be avoided, here are some common pitfalls I've seen folks encounter.

* My buddy uses is, so it must work - Please don't assume that something that worked for someone else works for your organization. Test all the key functionality in the PoC or at least talk through it with your vendor/partner.

* We need something now, so let's just do this quickly - By skipping through the selection process, there is so much risk added. If you don't have the time to invest in the selection process, just wait.

* We want everything in the PoC - PoCs are not meant to be full solutions with everything. Ask for the most important features to be developed as part of the PoC. Even if it does have everything, you'll want to participate in the development process to ensure you know how to change things without having to go back to the vendor.

2018-08-28 17:02:28 UTC28 August 18
Real UserTOP 5LEADERBOARD

Hi Rhea,
This is my personal opinion and shouldn't be taken as a best practice manual, but regarding PoCs, the better way I found so far:
- Having very clear what you want to "test" in the tool. Lets say that your "actual problem" with data is volume, please, don't test only volume handling (you risk not to assess other functionalities that may become the "new problem". in the future).
- Ask the vendor/partner to help you with the PoC. This is the better way to avoid trouble solving specifics on the tool, other than testing it, and can be very frustrating, biasing your judgement on the tool.
- Use a specific business case/business question to tackle the PoC and involve the business user in the PoC (at the end, he is the SME)
- Do not hold ANY questions to the vendor, he is there to sell, but once the buying decision is made, he should be there to provide you support (and enablement, and training, and etc..

Hope this helps you.
Best,
_AG_

2018-08-28 16:38:49 UTC28 August 18
Vendor

Hi All,

Excellent answers below. A few things I would add. Full disclosure...I am on the vendor side of an embedded BI platform. I ran our pre-sales engineering program for several years. Hoping that sharing a few things from the vendor's perspective can help you find the right platform.

First off, the comments about using your own data, coming up with well-defined requirements, including example reports and dashboards, etc. are all spot on. Make sure you know what your "must have's" are versus your "nice-to-have's". No software can be all things to all people.

One thing I didn't notice mentioned already is to evaluate the *team* you'll be working with as well. Entering into a new vendor relationship is just like any other, and you'll want to make sure you feel comfortable working with the team. Everything is bright and glowy and sunshine-y during the sales process. But sometime down the road, you'll need something. Do you feel like the vendor is vested in your success? Or are they just trying to close a deal?

If you haven't done so already, I'd make sure to check out online reviews of both the product and the company on sites like Gartner Peer Reviews, G2 Crowd and Glassdoor. Down the road the people you are working with will be just as important as the features of the software. Do you want to be supported by a bunch of people who hate coming to work every day?

By the same token, talk to reference customers. They will of course be hand-picked by the vendor so take their responses with a grain of salt. But don't be afraid to ask about vendor weaknesses as well. If they tell you there aren't any, that should be suspect.

Overall be realistic. Every vendor solution will have its trade-offs and if someone appears to be perfect and promises you the earth the moon and the stars something's amiss. At the end of the day you want to find the right combination of tradeoffs that will provide you the most value with the least pain.

As someone who's been through a number of these I'm happy to talk with you directly if I can be of any further help. I'm easily findable on LinkedIn.

2018-08-29 12:17:14 UTC29 August 18
Real UserTOP REVIEWERELITE SQUAD

All good guidance so far! A few things I'd add:

1) Before you even start a POC, put together a core set of requirements, and ask vendors to complete an RFI (Request for Information) - basically a check list of capabilities as they pertain to your use cases. This will help you identify early show-stoppers and rule out some vendors from the beginning should they not be able to provide some of the core functionality you're looking for. For example, when we last evaluated tools, automated report generation and distribution was a requirement, that several vendors could not meet. Likewise, more specifically, we needed a tool that could display Image thumbnails within a table. Again, many vendors were unable to do this and we were thus able to cross them off our list without out any more time evaluating the solution.

2) Echoing what Bob said, don't rely only on the vendor to prove something can be done by going away and doing it for you. Be sure you fully understand what it takes to do it yourself. i.e. Is it more or less out-of-the-box functionality, or does it require a lot custom coding, extension building, etc? A lot of solutions can accomplish nearly anything with enough time and technical expertise, but you probably don't want to rely on that for your primary use cases. Look for something that meets most of your core requirements while still enabling rapid dev and deployment.

3) As much as possible, make sure you're testing it under real-world conditions to properly gauge performance. I agree that data volume shouldn't be the only thing you assess, but do use one of your largest data sets rather than a sample and make sure the POC is taking place in an environment that has relative parity with your production environment. A lot of products demo well on a sample data set and vendor's architecture, but fall short once fully deployed. Performance is so crucial to adoption. You can build all the insightful, cool data tools in the world, but if they're slow, nobody's going to use them.

2018-08-28 19:57:07 UTC28 August 18
User

I’ve always approached BI/Analytics tools evaluations/POCs with the mindset that the POC needs to demonstrate the tools ability to satisfy the requirements of both the IT organization and business teams ( who will be the primary consumers of the insights generated out of the tool and interacting with the technology most frequently). This is especially important when the POC is focused on tools for self-service data discovery and visualization. I have always started the process with a jointly defined set of evaluation criteria (evaluation matrix) that is weighted and can be stored as part of the POC evaluations. Another best practice I have used is to take actual business use cases and incorporate them as part of the POC. These are usually of simple to moderate complexity, can be implemented quickly in the POC, and can demonstrate business value. I have also made sure to incorporate IT specific use cases such as data connectivity, metadata capture, account setup/creation, security, and administration. Each of these can be tied to the evaluation matrix. I’ve found that this really demonstrates which tool best fits the needs of the organization.

I’ve also relied frequently on the vendors to create the POC environments. This eliminates the need for internal IT to provision temporary infrastructure and can significantly accelerate the evaluation process.

What mistakes should be avoided?

The biggest pitfall I’ve seen is not including the business in the process. Another mistake is to not validate the fit with the existing technical landscape and systems portfolio.

2018-08-28 13:40:12 UTC28 August 18
Real UserTOP 20

Most of a reporting tools are exact same.
Details is a subject to add to a question about what is better to use?

Major steps to select a right product:

1 Estimate how many potential users you have
2 Estimate how much you gonna pay for a license
This will help to narrow down a list of products offered.

3 If possible - check with a vendor a report about at least 5 gb size to see a reports performance

4 Figure out which back end can be used and how effectively as of a connectivity to use a tool as well as a price of a different back end if required together with a migration price.

5 Figure out a culture level of users if you are going to implement reporting as a self serve. Who at least can work with Excel and knows what is a pivot table?

6 Agree on a trial test for a most advanced users - again using a mid size data sets.

7 Figure out about how complicated is a security configuration and its requirements.

8 Does this product requires an administrative 24/7 maintenance? How long takes a refresh or reboot?

9 What is a back compliance policy? as new version upgraded -> may have crap all existing stuff

10 Working in a cloud environment is too different from a local net?

11 .. many other things basically related to a reports functionality and advanced capabilities.

Cheers,
Art

2018-09-21 13:36:29 UTC21 September 18
Real UserTOP 20

First of all check the reporting tool requirements and if it works with your database or not. Install all the necessary frameworks and then make a connection to your database to see if it works or not. After the connection is made you can make a sample report with a test query.

2018-08-29 09:52:33 UTC29 August 18
User

What's the best way to trial reporting tools?

The best way to trial reporting tools is by doing a hands-on PoC at your office. The people in charge of making reports should be able to recreate simple reports on their own in the tools alongside an expert from the company.

Do you have any advice for the community about the best way to conduct a trial or PoC? How would you conduct a trial effectively?
My advice would be to recreate 4/5 reports that are made manually currently in the BI tool. We found out certain features were lacking when we tried to recreate the same report in different tools. For our internal comparison, we used SAS VA, Power BI, Qlik and Tableau. We created the same report in all 4 tools starting with the absolute raw data. We created all calculated columns/flags in the tool itself.

What mistakes should be avoided?
Do not go by the demos the Pre-Sales guy shows you while demonstrating the product. The data used has gone a lot of transformation and the data you have may not be as clean as theirs. Always use your data (dummy/masked) for the PoC.

2018-08-29 05:43:04 UTC29 August 18
User

To reiterate what is already been stated, have a strong business case established to use for the PoC. Don't make it up as you go along. Also, recommend having a scoring methodology/matrix for each tool you are testing. This takes some of the subjectivity out of the mix so you can base your decision on how each tool meets your needs.

We recently went through this process and having the scoring really helped.

2018-09-04 13:57:38 UTC04 September 18
Find out what your peers are saying about Tableau, Microsoft, Windward Studios and others in Reporting Tools. Updated: November 2019.
378,570 professionals have used our research since 2012.
Sign Up with Email