Part I: The Pilot
A/B testing is part of our company’s DNA; we test every change to our platform and games. When we were small, this was easy, but as we grew into the tens and hundreds of millions of users, query speed ground to a halt. (Familiar story, right?)
So in 2011 we piloted Vertica for our A/B testing suite. Our nastiest query used to take up to 22 hours to run on [name of old vendor - but don't want to mention them and be mean]. On Vertica, it ran in… 800 ms. That’s right, a scan and aggregation of over 100 billion records could be done in under one second. We were hooked!
Part II: The Rollout
Yeah we rolled it out. Boring. No interesting story here.
Part III: The Impact
Not having to worry about speed or data volume changes you. Suddenly we began logging and reporting on everything. Where did users click? How long between clicks? How long does it take to type in a credit card number when you’re ready to pay? How much free memory does an iPad 1 have, and how does that change every second?
Like all software engineers, we solve problems under constraints, and we had conditioned ourselves to think of logged data volume as a constraint. Suddenly that was no longer a constraint, but I would say it took us a full year to fully appreciate how powerful that was.
Part IV: Today
Today we record every customer interaction with our games and platforms – on phones, tablets, Facebook, and the web. Every department at the company consumes this data.
Marketing: Monitor ad campaigns in realtime, and throttle campaigns up/down based on performance of the users who are acquired via those campaigns.
Game design: Monitor game difficulty and tune in realtime.
Operations: Monitor for changes in customer service volume, exception logging, etc.
Creative services: Test different artwork and themes and monitor impact on game KPIs
Finance: How much money did we make in the last 60 seconds? (Bonus tip: finance gets very happy when they see this, and a happy finance department makes for a happy company. Me: “Hey Bob, can I buy an Oculus Rift for my team to play with?” Bob: “Hold on let me check the reports… whoopee! Sure thing, request approved”.)
Part V: Conclusion
We love speed, unlimited data, and Vertica!