What is our primary use case?
My use cases are typically working in conjunction with our business partners. For the most part, I get questions from the business as to what our conversion rates are on the eCommerce website. E.g., what the typical user journey looks like, especially when they're doing AB testing. They like to sort of double verify that in Dynatrace with a user session tracking to see which users are taking which path.
I often get diagnostic questions about things like latency. Somebody on the business side will perceive some latency on one of our pages, then give me a call to use Dynatrace to go in and do a waterfall analysis of the page load to see if it is in fact loading more slowly than it has been in the past.
How has it helped my organization?
Since we are a 24/7 support shop, our primary job is to make sure that revenue keeps going in through the site and the site itself works. We have categorized issues into three tiers: Priority 1, 2, and 3. Depending on where a given issue falls, the Dynatrace alert is generated and sent to my team, then prioritized into one of those three categories. The response then to that issue will fall into a bucket for that type of priority.
My primary use of the tool is to keep revenue coming into the business and to use it to help our business team in running their site analytics and web performance tools. They have things like Adobe Analytics that provide them with one layer of data. We use Dynatrace as another railroad metric to both confirm the Adobe Analytics data and enhance it in certain places where Adobe won't give us the answers that we need. In terms of metrics, we've had roughly about 120,000 unique sessions per hour on our website. So, we're capturing a lot of session data and real user data, and all of that data is kept in user sessions. We can look this information up by user ID to tag any given session that we want to find by date/client. E.g., if the user said that they had an issue last Thursday at 11:00 PM, then we can just do a search on their email address, go through all their sessions, and find the one that they mentioned, then dig directly into that one.
It has completely transformed the way that we do eCommerce from a couple of different perspectives. The first one is that we are really tied in with our business users to a much greater extent than we were in the past. A lot of the Dynatrace data is data which not just the technical team wants to see and to work with, but it is data that the business team wants to see as well. It has sort of facilitated a better communication stream between my business partners and my team to allow us to go back and forth on different aspects of the website. We didn't have that in the past.
It has also built more solid relationships between the application development team and my support team because they will often have questions after new releases as to how those new releases affect different areas of the site. So, we have constant sessions with those guys in standing meetings on a biweekly basis to go through different aspects of what a release has done to the website, how we can use Dynatrace to track differences from one release to the next, and see if there is any latency as a result.
What is most valuable?
There are about 24 dashboards built, not only for my eCommerce support team, but I've also built dashboards for our development, business, analytics, and senior management teams that allow them to log into the product. Without having to know much about Dynatrace, they can just click widgets on the dashboard that I've customized for them to get to the information that they need. This saves me a lot of time because I don't have to go in and investigate every single issue that crops up. I can go in and just ensure that there's a dashboard available for diagnosing that type of issue and point the user who is interested in it to that dashboard.
After the dashboards, the Davis AI engine is fantastic. We're able to set thresholds within the Dynatrace application for what acceptable load times are for our web pages or API callback times. Davis actually monitors what those thresholds are and notifies me, not only when the thresholds are violated, but when they are either over or under the threshold. Then, it makes suggestions for tuning those thresholds based on what it sees in real user actions. Therefore, I'm never dealing with outdated data. Every day Dynatrace is updating what the user experience looks like and letting me have that compared to my benchmarks, which is super useful.
We use both Session Replay and synthetic monitors:
- A Session Replay is primarily for our customer support operation. We sometimes have customers call in to complain of issues with the website, and it's really useful to be able to look a customer session up and replay it in its entirely so you can see exactly what happened during that user journey. You can even find the point that the user is calling in to raise an issue, so you can dig down and resolve it. We record every single user session on the site. We don't have a limit on the number of sessions we capture because it's just so useful for our customer service folks to be able to do this. It's worth the trade off for us.
- Synthetic monitoring is set up on a couple of different levels. My primary synthetic runs every five minutes, every day, 365 days of the year. It is a simple, single page pane that tells me whether the website is up or down. If it is down, then an email distribution gets emailed and a pager text goes out to whoever is on call at that time to let them know that the site is down. We also use synthetics for the user journey testing. When new features go in from development, as part of our QA process, we'll often set up Dynatrace synthetic that simulates what that user journey should look like. We will then allow the synthetic to run every given set of minutes (whether it's 10 minutes, 15 minutes, or half an hour), to collect data on what that user journey looks like. This allows us to go in and run our reports against that synthetic module rather than against real user search.
What needs improvement?
For how long have I used the solution?
We have been using it for about 18 months. We first installed it last year in January.
What do I think about the stability of the solution?
Dynatrace itself doesn't really break, but the website does on occasion lose connectivity with a given API vendor, whether it's a payment processor or one of our other API pieces. Dynatrace is very good at alerting us when that happens, but we're not using any sort of self- healing capacity for that. What we do is we get alerted when those APIs aren't available or when a web page has an issue. My team gets alerts on a 24/7 pager basis, then we go in and investigate to resolve them.
The solution has given me a better view into uptime, in terms of how much downtime our website has. However, the Salesforce Commerce Cloud solution is remarkably solid. We rarely have downtime. Even during the holiday season for Black Friday and Cyber Monday, we've had zero percent downtime for the last two years. From one perspective, it does let us know when the site is down using the synthetic monitor. The good news is that it has not had to give us much data because the site just doesn't go down.
What do I think about the scalability of the solution?
We have 96 users logging into Dynatrace right now.
Our primary eCommerce environment is Salesforce Commerce Cloud. That is where we have the agent list SaaS solution implemented. We also have internal API servers within Azure, where we've implemented user agents to track them. We have not encountered any limitations in scaling to cloud-native environments with Salesforce Commerce Cloud. We've had remarkably smooth deployments.
We have not scaled up to any other environments at this point.
How are customer service and technical support?
It is fantastic. You don't even have to pick up the phone. You can submit a Jira ticket from directly within Dynatrace to the support team. Those Jira tickets are categories based on the component of Dynatrace that you're looking at. You actually get a live agent chat within the tool, so you're not only submitting a ticket and getting a case number, but you have the support rep right there in the chat session to walk you through it. It's the only product I use that has a similar interactive of an online health system.
Which solution did I use previously and why did I switch?
We have used other monitoring applications, like SolarWinds and Gomez, in the past. However, they have all been replaced by Dynatrace at this point.
How was the initial setup?
From the infrastructure perspective, what we have installed is the user agent on our API servers. So, we have six API servers set up in an Azure load balancing pool. There are three active at any given time. It was super neat when we installed the agent, because we actually went out and had lunch after the installation. When we came back, Dynatrace had generated the Smartscape view of not just the API and the different services they connect to, but it had crawled our entire network and found everything that it recognized from SQL Server databases to .NET Servers and API services. All of that stuff showed up in a sidenav automatically without our having hands on anything whatsoever. It provides a good quick view in the morning when you come in and just flip to that view right away, because it will flash in "red" for any given service or platform that is having a problem, then you can zoom to that problem and look into it right away.
What was our ROI?
Being able to capture every single user session on the site and work with our customer support team has been a huge return on investment for me. Of course, the additional support on top of Adobe Analytics to be able to try things, like website conversion, is also a huge return on investment.
The time to diagnose has decreased on average by 34 percent in its implementation. That is primarily because Dynatrace not only alerts you when something goes wrong, but the Davis AI also gives you suggestions as to why it may have gone wrong. This gives you a head start on triage and resolution.
Because the Davis AI gives us such a head start on problem identification, this leads into triage and diagnosis. Our diagnosis time has gone down significantly. We can find, identify, and get problems triaged more quickly than we could in the past by approximately 25 percent.
What's my experience with pricing, setup cost, and licensing?
The only limitation with scaling to cloud-native environments is licensing. It all depends on how many DEM units you're willing to license. The more of DEM units that you purchase, the more user data you can collect.
Which other solutions did I evaluate?
We did compare it with several other products in the market when we did our due diligence before purchasing an APM and SaaS solution. Dynatrace came out just leaps and bounds beyond the pack. We're very happy with the results we're getting with it today.
We compared Dynatrace with AppDynamics, Opsgenie, and New Relic.
What other advice do I have?
We do not use the solution for dynamic microservices within a Kubernetes environment. It was on our development roadmap for this year, but I think COVID-19 has probably pushed it to next year. While it is something we will be doing, we're not doing it now.
We have not yet integrated the solution with our CI/CD and/or ITSM tools, as it was on our roadmap for this year. We are a GitHub and Jenkins shop, and Dynatrace has plugins for both of those tools. One of the very next things we want to do with the tool is plug it into our CI/CD process so we can have sort of a hands-free built. We want to allow our builds to run through the entire pipeline and be managed by these three tools, then allow Dynatrace to do the reporting on the deployment and the resulting difference in the web application based on that new format.
My advice would probably be to start with the SaaS implementation to get a feel for Dynatrace, what it does, and what it can deliver. Then, based on results with the SaaS platform, evaluate installing the onsite on-prem solution. They both have their advantages and disadvantages. They obviously work best when you use them together, but there are some instances where our firm does not need an on-prem solution and may need just the SaaS application. Vice versa, there may be some firms that just need the on-prem solution and don't need the SaaS cloud based solution. In my opinion, it is best to start with SaaS, then based on what you discover with SaaS, decide whether you need on-prem.
I would give Dynatrace a solid eight (out of 10). It's beyond the expectations that I had when we purchased and installed it. As I went along and learned more about Dynatrace after the implementation, I was impressed with how much the tool does. Another aspect is not just how much it does, but how easy it is to do it. The AI engine runs 24/7/365, providing input. The dashboards make it super easy for my users to use as well as myself.
The analytics that it provides are very easy to read. You can present them in pie charts, bar charts, or single table data. There's just a myriad of ways you can display the data that you get from Dynatrace to make it more consumable for users.