Over the years I’ve come to rely on information radiators during
testing to get immediate (or as quick as possible) feedback from the
systems I’m testing.
Firebug, log files, event logs and many other sources of information
are all very useful to a tester. They can give you insights in to what
is happening in the system under test.
We’ve just taken this a step further by rolling out NewRelic on our test servers.
NewRelic is what’s termed a “Application Management Solution”.
I’ve been talking about this internally as a system that can give us three distinct insights:
- User Experience Information
- Server Information
- Product Performance Information
I’ve probably over simplified the tool and doing an injustice but it
allows me to clearly explain the value we’re seeing from it.
User Experience Information
NewRelic gives us all sorts of data around how the experience is for end users when they use our product.
We can use this to ascertain how our product is being experienced by
our customers, but we can also use it to understand how the experience
is stacking up for our testers.
If we are testing and we observe a slow down we can check whether it
really was a product slow down using NewRelic and more importantly;
what’s actually happening on the stack.
We can use NewRelic to work out what browsers are being used across
all of our environments. We can see the test coverage we have across
browsers and we can also see what browsers our own business use from our
pre-production test environments (where we test all kits before live
deploy).
We can also then see which browsers are faster than others. We can
see which versions are used and which browser is our most heavily used.
Interesting stuff to help guide and tune our testing.
Server Information
NewRelic monitors the actual servers giving all sorts of information
such as memory, CPU, process usage etc etc. This is great information on
our test servers, especially during perceived slow downs or during a
load test.
We have other mechanisms for measuring this also so this is the least used function in NewRelic when testing.
Product Performance Information
For me, this is the greatest information tools like NewRelic offer; they show you what the product is actually doing.
It includes what pages are being dished, how fast are they being
dished, where they may be slow (in the DOM? Network?), what queries are
being run, what part of the code is running them and how often they are
being called.
When we dig around in the data we can find traces that NewRelic
stores which give an amazing level of detail about what the product
is/was doing when the trace was run.
It’s going to become a testers best friend.
In a nutshell what it allows us to do is provide an accurate picture
of what the product is doing when we are testing. This means we can now
log supremely accurate defect reports including traces and metrics about
the product at the moment any bugs were foud.
The programmers can also dig straight in to any errors and be given the exact code that is generating the error.
We can see which queries are running meaning that if we encounter an
error, a slow down or something worth digging in to we have the details
to hand.
It’s still early days using the tool but already we’ve had deep
insight in to how the product runs in our environments which I’ve never
been able to get from just one place.
It’s immediate also. Test – check NewRelic – move on.
Imagine how powerful this could be on your live systems too.
Imagine the richness of information you could retrieve and imagine
how fast you could get to the root cause of any problems. It’s powerful
stuff. Expect to hear further posts on how tools like this can inform
tests, provide a depth of supporting information and provide help to
performance testing.
Some notes:
- There are alternatives to NewRelic.
- It’s still early days but tools like this are proving invaluable for
accurate and timely troubleshooting and information gathering.
- I’m not affiliated to NewRelic in any way – I’m just a fan.
Disclosure: I am a real user, and this review is based on my own experience and opinions.
