At Twitter, we found that if you visited Twitter at least 7 times in a month, then it was likely you were going to be visiting Twitter in the next month, and the next month, and the next month. And we decided this was enough initially to be “really using it”, though of course I think Twitter gets even better when people use Twitter every day or more.
Corporate control of data could give preferential access to an elite group of scientists at the largest corporations.
When you talk about data-driven decision making it’s not necessarily about the everyday metrics. It’s about trying to figure out behavior in the data. And that can come through looking at actual data in your databases, or it can come through interviewing users and trying to figure out, “This person’s a really engaged user of my product…. Why?”
Ashish Thusoo joined Facebook in 2007 when the company had 50 million users. He left when it had some 800 million. During that time he managed Facebook’s internal data analytics team…
”We invested in making our tools more and more collaborative so that users could share analysis with each other and discover data by getting connected to expert users of a data set.” With Facebook’s hyper-growth and data that was changing all the time, a collaboration approach “worked better than creating knowledge bases around metadata.”
The user growth team works very closely with the data science team to ask the right questions about what drives growth, instrument logging to collect data that may answer that question, analyze the data when it comes in, define a test that is aimed at improving/manipulating some data set, measuring the results of that test, and then doing it all over again.
What does Mixpanel do that Google Analytics is incapable of doing?
• Retention (true cohort analysis)
• Very sophisticated querying / segmentation
• Retroactive funnel analysis (not just from time of setup)
• You can integrate in any language
• Real-time (every part of Mixpanel, not just one dashboard): Everything in Mixpanel is to-the-second real-time instead of 12 hour delays you will often get with GA
The HEART framework
While helping Google product teams define UX metrics, we noticed that our suggestions tended to fall into five categories:
Happiness: measures of user attitudes, often collected via survey. For example: satisfaction, perceived ease of use, and net-promoter score.
Engagement: level of user involvement, typically measured via behavioral proxies such as frequency, intensity, or depth of interaction over some time period. Examples might include the number of visits per user per week or the number of photos uploaded per user per day.
Adoption: new users of a product or feature. For example: the number of accounts created in the last seven days or the percentage of Gmail users who use labels.
Retention: the rate at which existing users are returning. For example: how many of the active users from a given time period are still present in some later time period? You may be more interested in failure to retain, commonly known as “churn.”
Task success: this includes traditional behavioral metrics of user experience, such as efficiency (e.g. time to complete a task), effectiveness (e.g. percent of tasks completed), and error rate. This category is most applicable to areas of your product that are very task-focused, such as search or an upload flow.
Social scientists have long realized that human behaviors are too complex and subject to too many variables to rely solely on quantitative data to understand them.
Engagement & Retention
• Although testing and iterating is how social devs have found success on Facebook, rapid iteration is not so easy or fluid on mobile platforms like iOS. It can take from three days to two weeks to receive approval for each new version from the App Store, making it difficult to fine-tune to optimize on the fly.
• Many users don’t update their apps regularly, resulting in outdated, or “dirty,” data from the combination of “old” and “new” app version users. This makes it challenging to iterate and improve your app based on user data without being misled, unless your data tools easily filter users by app version.
• Design the app to use data-driven back-end systems to control user experience in-app when a connection is available on the device. This way, developers can dynamically tweak play balance, economy balance, etc., and then perform split-testing to further optimize customer economics.
• Use an analytics solution that allows filtering data by app version, location and device type (for Android). This is very important in addressing the dirty data issue, and in determining which platforms and devices in which to commit time, effort and resources.
All of this is shored up by the tremendous analytics [Twitter] has. You have ways of deeply understanding the varieties of engagement and the levels of engagement that people have … And those folks are embedded with each of the teams, so each of the teams can really understand whats going on.
The growth team was created in 2007 and was the first example of data driven design at Facebook … A simple visual design change on registration page resulted in 9M more people per year joining. 3% increase. For growth this is a big deal.
A few months ago (2011), Facebook created a team called engagement – modeled after the growth team. Thought it would operate the same way. First tried quantifying success by looking at number of read/writes. One of the early ideas was comment liking. Saw an increase of writes in the system 7% up when this feature was rolled out.
Though writes went up 7% it was the same people creating writes. 85% of reads/writes of Facebooks are generated by 20% of users. Same people were doing the activity. Had to go back and reset metrics for the engagement team.
Current metric for engagement at Facebook: L6/7 number of users that come back to Facebook 6 out of 7 days a week.
I’m just one of several hundred research scientists on Google’s in-house research team, which stretches across everything from Gmail to Google+. My particular focus is on identifying key developments and trends in user behavior and designing systems to support them – backed up with big data analytics and a comprehensive understanding of how social systems function.