Over the years I’ve come to rely on information radiators during testing to get immediate (or as quick as possible) feedback from the systems I’m testing.
Firebug, log files, event logs and many other sources of information are all very useful to a tester. They can give you insights in to what is happening in the system under test.
We’ve just taken this a step further by rolling out NewRelic on our test servers.
NewRelic is what’s termed a “Application Management Solution”.
I’ve been talking about this internally as a system that can give us three distinct insights:
I’ve probably over simplified the tool and doing an injustice but it allows me to clearly explain the value we’re seeing from it.
User Experience Information
NewRelic gives us all sorts of data around how the experience is for end users when they use our product.
We can use this to ascertain how our product is being experienced by our customers, but we can also use it to understand how the experience is stacking up for our testers.
If we are testing and we observe a slow down we can check whether it really was a product slow down using NewRelic and more importantly; what’s actually happening on the stack.
We can use NewRelic to work out what browsers are being used across all of our environments. We can see the test coverage we have across browsers and we can also see what browsers our own business use from our pre-production test environments (where we test all kits before live deploy).
We can also then see which browsers are faster than others. We can see which versions are used and which browser is our most heavily used. Interesting stuff to help guide and tune our testing.
NewRelic monitors the actual servers giving all sorts of information such as memory, CPU, process usage etc etc. This is great information on our test servers, especially during perceived slow downs or during a load test.
We have other mechanisms for measuring this also so this is the least used function in NewRelic when testing.
Product Performance Information
For me, this is the greatest information tools like NewRelic offer; they show you what the product is actually doing.
It includes what pages are being dished, how fast are they being dished, where they may be slow (in the DOM? Network?), what queries are being run, what part of the code is running them and how often they are being called.
When we dig around in the data we can find traces that NewRelic stores which give an amazing level of detail about what the product is/was doing when the trace was run.
It’s going to become a testers best friend.
In a nutshell what it allows us to do is provide an accurate picture of what the product is doing when we are testing. This means we can now log supremely accurate defect reports including traces and metrics about the product at the moment any bugs were foud.
The programmers can also dig straight in to any errors and be given the exact code that is generating the error.
We can see which queries are running meaning that if we encounter an error, a slow down or something worth digging in to we have the details to hand.
It’s still early days using the tool but already we’ve had deep insight in to how the product runs in our environments which I’ve never been able to get from just one place.
It’s immediate also. Test – check NewRelic – move on.
Imagine how powerful this could be on your live systems too.
Imagine the richness of information you could retrieve and imagine how fast you could get to the root cause of any problems. It’s powerful stuff. Expect to hear further posts on how tools like this can inform tests, provide a depth of supporting information and provide help to performance testing.