2 Comments »This is a guest post by Bill Grosso, the Chief Technology Officer of Twofish.
As the social application space has started to mature, more and more application developers have adopted a “metrics driven” approach to game design. Driven partially by the availability of data gathering and analysis tools, partially by the distributed nature of the internet, and partially by the quick turnaround times that are possible if you analyze customer behaviors in real-time, it seems like almost every aspect of application design has gone under the analytical knife.
Which is why I looked forward to last week’s session on metrics at the Social Gaming Summit. The SGS is at the forefront of social game design, and the speakers, David King and Siqi Chen, are outspoken proponents of using metrics to guide every decision, and of making metrics a “core cultural value.” I was looking forward to learning about the latest and most up-to-date thinking from people who are on the forefront of metrics-driven design.
And make no mistake: Siqi and David gave a great talk.
But, somehow, it felt a little bit like a great talk about last year’s metrics. In a conference where every other talk or panel discussion prominently mentioned virtual currencies and virtual items and virtual gifting, it felt a bit odd that the metrics talk was focused on user acquisition, virality, and page views.
What makes it even odder is that virtual currencies and virtual items are natural candidates for quantitative analysis. At the end of the day, people are spending currency to buy things. As a culture, we’ve had 100 years of MBA’s thinking hard about sales reports: about what to measure, how to report on it, how to do drilldowns and user segmentation, and tactics for increasing sales once user behavior is known.
What I really wanted to see, and what I hope to see at the upcoming Virtual Goods Summit, starts with David King’s last slides. In those slides, he called out “Monetization by Gender” as a key piece of data, and split out age as a key bucketing device. This is a great start.
But to my way of thinking, it’s not even half the question. Once you know that a user group is spending money, the key question is what did those people spend money on? Are there items or types of items that sold well to particular demographics? Can you break down your sales by demographics, spot purchasing trends, and upsell effectively? Using a ‘transactional graph‘ doesn’t just lead to more effective monetization, it leads to a better user experience — or, at least, to a more engaged and invested user.
And for those users who have money but aren’t spending it, can you figure out why? Something as simple as “Once they’ve been using the application for 45 days, women over 30 spend 60% less money than during the first 15 days” is a powerful fact that can guide game design and item catalog revisions.
At Twofish, our recently released analytics framework focuses on exactly these sorts of questions. We think crossing user engagement and demographic data with sales data and information about user spending takes metrics to the next level for social application developers.
Speaking as a long time game developer who’s made the transition into social game development, I must admit that I find the obsession with metric-driven development a little hard to swallow.
What I love about social games is the way that it can connect you with customers, but at the same time I think it’s important to actually have a game idea in mind before you get to the metrics. It’s a very web-development centered idea to think metrics first but the trouble with that approach is that it’s entirely reactive and it leads to a lot of copied product.
What’s bad about the social game space as a result is how poor many of the games actually are as games, and secondly how quickly (and in many cases shamelessly) even the big names are falling to copying one another. This troubles me because (aside from the ethics of it, which is a debate that should be had) it’s creating companies with high revenues but low value.
By this I mean if everyone has essentially the same Mafia game (or whatever) then there is essentially no product differentiation and that ultimately makes the space limiting and creates a series of companies that seem exactly the same. Metrics are not really a substitute for originality and overly obsessing on them leads companies to unimaginative and ultimately short-term positions.
For example: I read that Zynga killed a $2m project because it wasn’t performing according to metrics. That seems crazy to me because it’s really just writing off a whole lot of investment on the basis of what sounds suspiciously like short-term ideas of metrics and a lack of patience to develop the game properly. In the same vein, they then rushed to “welcome” FarmTown (in Mark Pincus’ words) which seemed to me again to show very short term thinking.
What I’m afraid of is that the metrics-culture is essentially producing a reputation for social games generally of low-quality products. Low-quality short-term products is exactly what led to the Great Videogame Crash for Atari, and so far I see only Playfish managing to stand apart from what is increasingly looking like a race to the bottom.
It’s not really about who-did-what so far or how various companies have arrived at their positions. It’s about high value intellectual property, thoughtfully built applications and game quality into the future. Companies can flame out just as quickly as they’ve meteorically risen if the audience grows tired of their output, as happened to Atari, who really messed up a commanding position by developing a reputation for shoddiness. Speaking as a nascent social game developer myself the most important thing facing social games is to ensure that that fate doesn’t happen to us.
I am by no means a game designer or associated with the game industry. I work in the web and mobile space building websites and apps for both platforms.
We have also seen this metric-driven approach take firm hold of our industry. This approach has come under great scrutiny as of late and exemplified in Doug Bowman’s post about departing Google as Lead Designer:
http://stopdesign.com/archive/2009/03/20/goodbye-google.html
“Yes, it’s true that a team at Google couldn’t decide between two blues, so they’re testing 41 shades between each blue to see which one performs better. I had a recent debate over whether a border should be 3, 4 or 5 pixels wide, and was asked to prove my case. I can’t operate in an environment like that. I’ve grown tired of debating such minuscule design decisions. There are more exciting design problems in this world to tackle.”
Doug doesn’t believe Google is doing anything wrong per say but perhaps not the best environment for creative individuals.
I personally have always believed that optimal measurement comes from both quantitative (data-driven) and qualitative (user-driven) inputs. If you are not listening to the users actual feedback and just looking at the numbers, the numbers can lead you astray on a wild goose chase for a performance metric that simply wasn’t there to begin with.
- Nick Finck
Director of User Experience, Blue Flavor
http://www.blueflavor.com