We all know that it's important to try out software as part of the buying process. Do you have any advice for the community about the best way to conduct a trial or POC? How do you conduct a trial effectively? Are there any mistakes to avoid?
Your questions are about 'how to trial effectively' and 'avoid mistakes' so will assume that after a lot of research, you have narrowed down the tools that fit your organization's specific needs. Also assumed is that you will be able to procure free trial license for the time you want which almost all tool vendors are happy to comply.
- First task is to form a small team with lead(s) to own the evaluation effort. This can be tricky but most organizations have top tech folks at various levels who are open to change in tool direction. For APM tool solution trials, team members should come from engineering, performance, capacity planning and development/devops. Engineering include platform, network, infrastructure and service desk SMEs. Team would be a handful with hands on ability to install,configure, script, test, analyze and present findings.
- Next task is identifying sandbox environment for trials and naturally would not be production or replication environments. Performance or staging would be ideal and the footprint of tool coverage across the CIs (Configuration items) should be minimal but representative of the APM operating environment.
- A load / performance test is almost always a requirement for APM tool evaluation and should be used for trial. Any existing tool would suffice as the objective is APM tool trial not test tool capability.
- Some criteria for effective evaluation are :
a. that resolved issues could be captured by the new tool when the fixes are removed (satisfies current capability)
b. one or more unresolved issues can be unearthed during the poc and corroborated (new capability)
c. time to resolution meets or exceeds current state
d. tool instrumentation is easy and dynamic or dynamic enough
e. tool footprint on CIs are known and scalable
f. tool integrates well with ITSM, platform specific tools(e.g. vm, SAN, DB)
- Some guidelines to avoid mistakes during APM tool trial:
a. focusing too narrowly on application areas can make choice look better than what it actually is
b. not differentiating monitoring and debug instrumentation can overstate bau efforts
c. during trials, it may be difficult to determine the false positives and negatives received/not received at ITSM so have to keep an eye the number and quality; it is difficult to be objective
d. all IT teams have a chance to independently review from their vantage point
The suggestions here are not on how to select an APM tool.
Because there are so many APM solutions available, you will need to do some preliminary research first. Consider your current APM solution that is in your environment. Review articles and evaluations such as those found in Gartner’s software evaluations. Make note of the good and bad of the tool being used now in your environment. From your preliminary research, narrow your perspective trial candidates to perhaps 5 possible solutions. Sign up for the trail offers and take advantage of the demos and WebEx’s that will engage the support and sales staff of the products. Typical you want to setup any downloaded software in a test or developer environment. If the trail is agentless, then you should have no problem. Consider the most problematic applications that you currently have in your current APM solution to be the ones to focus on in the trail or demo.
Obviously there are a number of other things to consider i.e. firewall rules, authentication, security, etc., but I hope this post help to answer your question.
We conducted a POC within our development/test environment where we ran a select set of UAT test cases manually after we had installed the APM solution. This so as to assure ourselves that the APM solution would not have any negative impact to our application’s functionality. I would have preferred to have run a complete regression test and would suggest that be done where possible and especially if the regression test suite has already been automated via an automated testing tool such as HPE UFT or Selenium. I would watch out for the ability to monitor and improve the performance of batch jobs as different APM solutions provide support for batch jobs to different degrees. Lastly, if you are selecting an APM solution for the enterprise, you might want to look into how the APM solution should integrate with the ITMS solution as well as any IPM (infrastructure performance monitoring) solutions and also whether it would be suited to monitoring the different technologies your applications are implemented upon.
- Identify the critical application(s)
- If you use an APM already, figure out the shortcomings and/or identify new requirements which are not fulfilled currently by already-in-use APM
- Having an APM in production environment has its own side effects. So, build your performance environment if you don't have one. Is it OK for your organization to run APM in production on a trial version? I strongly believe answer is NO, but it is still an option
- Simulate the production load in PT environment which is a key challenge as at many places identifying the right user mix is always an issue
- Enable application or configuration changes
- Get the new APM configured in your PT environment
- Configure dahsboards with relevant metrics as you require
- Run few performance tests including stress, production like load test
- Look at the dashboards. Are you convinced? Do you require any further tweaks or new metrics? If yes, get the help of product team or tool expert and make sure you are satisfied
Are you happy with the tool? If yes, go ahead with the commercial talks with the APM vendor.
Best way to conduct a trial or POC for APM.
- For the POC, identify YOUR solved issues to see if the proposed APM solution can identify and pinpoint the problem. Duplicate the issue to see the proposed solution can find the problem.
- Make sure the solution can be integrated to your environment. Different organizations have different operational environments. Make sure it meets the expectations for all concerned departments.
- Load test. The best load test is the production, real world, load test. MOST enterprise organizations won't allow this, and this is where it matter the most.
- Talk to the technical advisors directly. Understand that some APM solutions will offer different methods of delivery or solution. For example, SaaS, On-Premises, On/Off Premises. Make sure you know YOUR environment first so that you can best explain your requirements.
- Don't mix Sales and Technical teams together. Make sure the right persons are communicating with each other in order to minimize miscommunication or misunderstanding. I've had sales team overselling their product just to make a sale. I've had sales team lacking technical details. Sales team are there to sell and negotiate pricing, not give technical advice. In the end, everything will be contractual.
- Don't be shy to ask for product limitations, this will come after discussing in detail with their technical advisors about your environment and requirements (this can be contractual), good technical advisors will tell you of product limitations and possible workarounds. Good technical advisors will want to have good reputation, thus will always give the best honest answers, and they are the ones you want to be with.
As per my experience, if you are able to show the number of concurrent user hits at URLs and pagewise response time as well as include these things in reports format in POC.
It will be very helpful if we segregate the code, database, and infrastructure by going straight method after integration with unified infrastructure management.
The best method of shortlisting a tool is to list down all your requirements in a sheet, give weight to every requirement based on its criticality for your business and then comparing the tools based on your requirements. The tool which gives you best result in terms of meeting all your critical or must have requirements is the one most suitable for your business.
Best way to complete POC, is to select 2 - 3 application and expected parameters or measurement type.
You need to experience 'living with' the software - so it needs to be a trial exercise in your environment, driven by your people and stakeholders. The most important element is to exercise an application via load testing, in order to see what a 'real' application is doing, in normal and high load situations. If you don't have sufficient licenses to generate a heavy load, it is often enough to run the app on undersized hardware.
I would also recommend you make a modest investment in "APM best Practices - Realizing Application Performance Management" https://www.amazon.com/APM-Best-Practices-Application-Professionals/dp/1430231416/ref=sr_1_1?ie=UTF8&qid=1529343857&sr=8-1&keywords=APM+best+practices
Chapter 6 lays out all of the considerations for a successful pilot exercise - plus you get additional insights in how to justify, deploy and operate an APM solution. The best practices described are vendor-neutral.
I'm still a newbie to monitoring tools like APM.
Are there any Application Performance Management (APM) certifications for tools like CA APM, AppDynamics, Dynatrace, or any single/general certification?