Analytics Won’t Get You To Product Market Fit

all-you-need-to-know-about-big-data

In the new world of big data, everyone in the startup community seems obsessed with collecting data and making data-driven decisions. For the most part, this is a healthy obsession. New analytics platforms like KISS, Mixpanel and Localytics are providing great tools for product managers and marketers to better understand what’s happening inside their apps and web products.

We’ve also developed great processes for defining success from a metrics standpoint.  Aside from K-factor for viral apps and standard retention metrics for paid services, we have frameworks like Dave McClure’s Startupp Metrics For Pirates that help any startup with customers think about their product-market fit goals from an analytical view.

These innovations have been net positive for business building, but they can also enable a bigger problem:  they allow entrepreneurs to analyze  metrics as a replacement for talking to real customers.  in truth, it’s way easier to look at a Mixpanel screen than it is to talk to a person about your product. First, no one wants to face judgement directly, they’d much prefer to have it distilled into a bad retention metric. Second, talking to users is noisy – it’s really hard to pull actionable insights from a handful of conversations.

Analytics can provide us a window, but they don’t give us a narrative for how people think about our products, how they think about competing products, or even what they think of our value proposition (assuming they even know what our value proposition is).  The narrative around a given product will inevitably be the leading indicator of success or failure, but analytics will only give us a sliver of insight around it.

Depending on your product type (consumer and enterprise startups should handle this problem differently), there are a few tactics you can employ to get more narrative-based feedback on your product.  None of these will cost you much more than time, but they can save you months of incorrectly deploying your resources.

1. Follow Michael Margalis’ Quick and Dirty Consumer Research. Michael is a partner at Google Ventures’ Design Lab.  In this video, he offers a wealth of information about how to find and interview users. The video is 90 minutes and every single minute is worth watching. I cannot recommend this video enough for the discovery phase of your product development.

2. Have Users Test Competitive Products. If you want to build a great product, see where people are getting tripped up with your competitors’ products. The best part about this is that you don’t have to build anything to test competitors’ products.  Aside from identifying UX / UI opportunities, you’ll quickly understand how users feel about products in your category, and how they fit them into their lives. Most technology products are used to satisfy a need – understanding how people  think about their needs and solutions is as important as any metric in your analytics dashboard.

3. Talk To Your Existing Users.  There are two primary ways to get in front of your existing users –  phone calls and email surveys. I’ve found that both are great for different goals. Survey’s tend to get you ‘crowd-sourced’ style data points, general sentiment towards your product (e.g. satisfaction scores, net promoter scores, etc.). They can also help influence your product roadmap.  Interviews are better for the narrative questions – how do your users think about your product, when do they use it and what need(s) does it satisfy?

You Probably Dont Need a Growth Hacker Yet

GHC

I was lucky enough to attend the Growth Hackers Conference this week in San Francisco. I’m not typically that interested in conferences, but I could not miss this speaker list and I wasn’t disappointed. Product marketing and growth folks from PayPal, Facebook, LinkedIn, Twitter, Tagged and 500 Startups spoke on their experiences driving growth from early to the late stages of their respective companies, offering their insights and suggestions. It was a pretty priceless experience set. Here a few things I gleaned :

Growth Hackers Are Not Valuable Before Product Market Fit: There’s a common misconception that growth hacking (like regular hacking, I guess?), happens early in product development. It makes sense that people are confused. There’s been a lot of conversation about lean startup in the scene lately, so I guess people think of growth hacking as lean’s marketing-y sister.

It isn’t.

Growth hacking is experimenting with a pretty big data set, and that data set is a user base in the multiple millions of users / profiles / content pieces. Yes, the multiple millions. Think about it, if you’re going to hack your way into additional growth, you need to start with something to hack into. That pool is a large group of users coming through your door every day. It allows you to run multiple experiments per day and the scale means that even small optimizations will move the needle in big ways.

When Growth Hits, Build A Dedicated Growth Team: Growth hacking isn’t different from marketing, but it’s building a more technically-inclined team. I heard from a number of veterans today that growth hacking teams are typically comprised of a product manager, engineer, designer and data analyst. Everyone had a slightly different composition of talents in mind, but this was roughly the composition of the standard team.

Don’t have 10mm users yet? If you’re in a consumer business there is no question that exponential growth is important to you, so here are a few things to consider now while in the early stages

Build a Growth Culture: Baking growth into the organizational decision engine is a choice. It’s not for everyone, some companies want to focus on brand and having a growth culture can create tension with a brand focus. Keith Rabois noted that one of the best performing emails of all time at LinkedIn had a grammatical error in the subject line. At Square, he never could have gotten that email sent because the branded experience is so important to Square.

Set Measurable Goals: Facebook famously set a goal of getting new users to add 10 friends within 13 days of signup. At some point they figured out that users who add 10 friends within 13 days were far more likely to stay active, so that’s the metric that they optimized against.

Focus On Core Metrics: There used to be a school of thought that encouraged product managers to “track everything”. Instead, it’s better to define what your core metrics are and chase those. It minimizes distraction and keeps focus on the drivers of the core business

Experiment, Experiment, Experiment: There’s nothing more poisonous at an early stage company than creating a culture that punishes failure. Stan Chudnovsky said at PayPal that he might run five experiments in a given week and that they would all fail. Elliot Schmukler similarly noted that at LinkedIn they would actually repeat failed experiments letting other people run them. #awesome

Get An MBA, But Not Until You’re Useful

As an MBA working in early stage tech, I’m ambivalent about the current conversation in the community about the value of these degrees. On one hand, I’ve met enough MBA founding teams to understand where the vitriol comes from, but I also believe that my graduate business education has been the most rewarding investment I’ve made to date.

Why Many MBAs Are Bad For Startups

An MBA education teaches students to analyze markets and assess opportunities from a high level [e.g. crowd-funding is going to be a huge opportunity and disrupt traditional business financing, an $xx bn. annual market in the U.S. alone!]. That thinking is fine, but it can only get you so far in the early stage.

The best consumer-facing products satisfy an unmet need. Consumers don’t care where a particular market is headed,  they often don’t even know how to articulate the problem that they want fixed. Nothing in an MBA program really equips students to develop products that solve real world problems, so MBAs tend to create products that wedge themselves into what they view as a market opportunity.  That approach often produces low quality products that don’t solve real problems.  The best products start with an acute, definable consumer ‘problem’ and grow into market-disrupting companies, it’s never the other way around.  While the difference in approach might sound trivial, it results in products that are usually worlds apart in execution.

Also, in the early stage there are very few day-to-day activities where having an MBA is actually useful.  At most early stage companies, people are either building the product or selling it.  Neither of those functions require an MBA, so the other skills become ‘nice to have’, but they don’t necessarily drive the business forward.

Why MBAs Can Be a Really Useful Tool

On the other side, I view MBA programs as a safe place in an unsafe world.  I think they’re safe in two ways:

First, MBA programs tend to lag trends in the business world and that can be a good thing for your career arc, especially if you’re working in an industry that’s very fast-paced.  Skills like strategic thinking, data-analysis and an understanding of the broader business world and global economy don’t run out of funding, they don’t exit and they don’t become irrelevant when the market is disrupted.  MBA programs have curriculum in place that helps students navigate the business world for decades.  That’s extremely valuable in the long term.

Second, MBA programs are a place to learn things that you’ve never understood and that the real world isn’t going to let you learn on the clock.  If you’re an engineer, there are very few on-the-job opportunities to understand option values, venture financing, accounting, macro economics or product marketing.  While it’s true that you can learn many of these things on your own, having a curriculum and a classroom are incredibly helpful aides to the process.  Just having to show up with your homework done will propel you forward.

MBAs Are Useful After You’re Useful

Everyone has a unique set of inputs that should drive their decision about a graduate business education, but my broad advice to most young people interested in early stage tech and considering an MBA is to go for it, but only if you’ve already developed practical skills that will make you invaluable an early-stage company.   An MBA alone will never be enough.

Don’t Be Busy (self-help post)

I’ve had a pretty full calendar lately. I’ve been trying to accomplish a number of important goals since this year started, both personal and professional, and I’m finding that my to-do list is spiraling out of control.

One of the things I learned to do in graduate school was focus on the important things on my lists, and deflect urgent issues whenever possible. In practice, this is incredibly difficult to do. The hardest part of my day is trying to prioritize what’s most important to accomplish, and then making sure that I do those few things and manage expectations around everything else. We all have multiple stakeholders in our lives and all of them demand attention, usually more attention than we have time to give. Without higher level planning, we end up burning ourselves out doing ineffective things for others, which doesn’t help us in the longrun.

I’ve been working on honing my prioritization and effectiveness chops for years, as I’m sure we all have.  These are the things that work for me:

1. Try to be mid-term goal driven: The best thing about a new year is the opportunity to reflect on accomplishments form the previous year and define goals for the next twelve months. Every three months or so I try to think about where I’d like to be in a year or two.  I then course correct and engage in activities that I think will get me there.  For me, thinking a year or two into the future works really well.  Some people take a longer view, and some people are very focused on the immediate.  I find two years is a good horizon because I can build an actionable plan around that. Anything further out and there’s too much uncertainty.

2. Make smart lists / be organized: I am a huge fan of keeping a list.  I have one in evernote that I update every morning. I try not to let it go over 5-6 items, but it invariably does.  I also try to keep everything on the list something that I can push into a new version within the week.  For example,  If I’m trying to build a sales department for a startup, I might add something like 1.) build and populate sales pipeline 2.) create draft of sales deck for one week, then run a cycle and change them to fit the next phase.

3. Use technology to help you and fend off time wasters:  I practice Inbox Zero, which is a fancy way of saying I try to keep my email inbox empty.  To do this, I use (and live by) a product called sanebox. Sanebox pre-sorts my emails and only puts what’s important in my inbox.  Everything that makes it into my inbox gets read, responded to and archived.  I cannot begin to explain what a game changer this has been for me.  Use it.

 

The second part of this, fending off time wasters, is getting harder every year. I try not to respond to push notifications on my phone, I try not to bring my phone into meetings and I try to keep my browser tabs to a minimum. I fail pretty miserably at all of this, but I do make a concerted effort.

4. Know when to zone out: This is what I use to justify my failure to filter techno-noise. If I’m running around the city this is a non-starter, but if I’m working on a longer project I take regular breaks and zone out for a couple minutes.  It keeps me from getting distracted when I’m actually working,  I try not to do this for more than five minutes at a time every hour or so.

5. Eat healthy food: If I eat a lot of sugar or carbs I find that I get really sluggish. I try to avoid both during the day.  I also ty (and fail) to limit my caffeine intake so I dont crash.

6. Get enough sleep: This one is easier said than done, but I can’t seem to function if I dont sleep six-seven hours.

7. Exercise every day:  Another difficult promise to keep, but I find that if I can get in 30 minutes of cardio I am a much better person to be around. I bet you’re the same.

Content Is King Again

1322538843_burger-king

The never-ending power struggle between content and distribution seems to be writing a new chapter around the film and tv worlds.  Some very big things have already happened in the first quarter of this year in the media business.  These are just a few happenings that have rocked the media biz recently:

– Comcast speeds the acquisition of NBC

Twitter acquires Blue Fin Labs

Liberty Media Purchased Virgin Media

Netflix debuts exclusive premium content

HBO Go becomes available on Apple TV

Amazon & CBS Expand A Content Deal

– Time Warner is selling off a huge piece of Time Inc.

What’s Happening

I look at the value chain in the old cable television industry as such: Studios -> Networks -> Operators -> Consumers

In a simplified model of cable television, studios pay to create content (or networks would pay advances), the networks buy rights to that content and sell advertising against it , and then cable operators pay networks for the content that they pipe through their lines to reach consumers in their homes.

You’ll notice that Apple, Netflix, Hulu, Google and Amazon aren’t really an obvious part of that chain.

Consumers want content, hate their cable bills and are increasingly willing to watch the content that they want online. This is particularly true of  younger consumers.  Online streaming services can now circumvent networks and operators, cut deals with studios and deliver content directly to consumers over the Internet.  This is rapidly changing the dynamic of the media and entertainment industries and shifting the value away from distributors and towards content creators.

It’s not immediately obvious how this is going to affect consumers. One the one hand, disintermediation should lower costs for consumers, but it’s unlikely to do so in this case because most of the value in this chain is on the content, not the distribution.  Because there’s no marginal cost to delivering the N+1 unit of Breaking Bad or Downton Abby,  the price for the content in a perfect market will be set at whatever price the creators can profit the most at.  The cost of delivery isn’t really part of the equation.

Another stakeholder group that’s going to be hugely affected by this change is advertisers. Currently prime time television CPMs are probably somewhere around $50, but pre-roll video ad eCPMs are probably between $5-$10 (nowhere near the value of tv).  As the value of advertising declines and as customers increasingly purchase their content via subscription and ala carte models, advertisers are going to find it difficult to get in front of audiences at scale with high value brand messaging. Additionally, ad supported content is going to become increasingly rare and the demographic composition of consumers watching ad-supported content is going to shift.

What 2013 Will Look Like

These shifts will increase in speed over the next year. I think we’re going to see studios and networks potentially change hands.  Operators are going to struggle to backwards integrate into content  and Internet services are going to bid up the price of premium content and pass those costs onto customers in the form of ala carte premium pricing.

TV ad dollars will at last  begin to decline as advertisers realize that their audiences have moved online. These dollars will flow into online ad channels (video being a big one), social advertising and, to a lesser extent, mobile.

Diego-San, Humanoid Robot One Year Old

I found this on Singularity and wanted to share it. This is Diego-san, a new humanoid robot who will be developed to help researchers better understand child development and motor skills.With 27 moving parts in its face and cameras in its eyes, Diego-san is able to produce a range of infant-like facial expressions.  Pretty amazing work:

Signal in the Social Noise

Eric Schmidt famously said in 2010 that every two days we generate more data that we have up to 2003. To clarify:

  • dawn of civilization -> 2003 = X amount of data
  • this past weekend = X amount of data

That’s a mind-bending statement.

For a long time, middle management in large organizations has lamented the tornados of data that it’s faced with processing on a near daily basis.  Middle management’s primary responsibility in large organizations is processing and interpreting vast swaths of information in order to report to upper management for strategic decision-making. We had to invent middle management in the business world just to process data.

And if that wasn’t enough, now we seem to have as much data in other spheres of our lives: social, personal, professional.  Everything is being documented and quantified now. The silver lining, of course,  is that our ability to process huge amounts of data is also in its infancy. As storage costs continue to decline and as more data stores are hosted on cloud-based systems, the possibilities are nearly infinite for us to track, process, analyze and interpret larger and larger pools of data than we ever have previously.

Facebook’s new graph search is an early attempt to address the need for a new kind of search, one that’s built on the people we know.

For me, the followup question is about untapped opportunity.  What haven’t we started to process that can help change the way we do business and the way we live our lives?  Is there any signal left in the social media noise that we haven’t found? I assume that graph search will produce a ton of value that hasn’t been unlocked: restaurants, travel, workplace searches will all become more interesting. But are there insights that it won’t pick up?

Here’s another question. If we use Google to search information about the world and we will presumably use Facebook’s graphsearch for information about our friends, what will we use in the future to search for information about ourselves?  Who is going to store our personal data, process it and feed insights back to us and our doctors?  I doubt that Google or Facebook are going to provide that service. I think we’re going to want that information in a ‘closed’ system, or at least a one way street (pull data from other services but never let my weight history, blood history, dietary habits, emotions, measurements or other personal information out into the real world).

Not that this is a new idea, but I believe that we’re on the cusp of tremendous innovation around data processing, and I’m personally excited to be here for the ride.

Future Perfect

Image

I just finished reading Future Perfect by Steven Johnson and wanted o recommend it, as well as get the salient points across in words to help me better understand the reading.

Johnson takes the reader through vignettes in history, and describes various social structures to make the point that decentralized, or distributed networks of decision-making are generally superior to what he refers to as centralized, top-down decision-making structures.

The most visual representation of the dichotomy that Johnson creates is the story of the evolution of the rail systems in France and Germany. Referred to as the Legrand Star, the French developed a rail system in the 1800’s that focused most of its traffic on main lines, all of which ran through Paris.  The entire system was designed from start to finished and was highly centralized. This made a train system that could reach very high speeds, and get people in and out of Paris from all over the country.  The Germans, on the other side of the Rhine, developed their system with no centralized design, the final product was mostly a messy network of small linkages.  When the French and Germans went to war in the late 1800’s, however, the German rail system ended up being the better solution because it was more flexible, while the French had to move all of their soldiers through Paris before getting to the front on the eastern border. Because the germans could get more troops to the line faster, they ended up winning the war.

The book has so many great, optimistic ideas about how distributed networks can make a better future. I think we’ve seen some of this at work with Kickstarter, SOPA’s defeat and a thousand amazing geopolitical events that have been amplified through social media. While some of his vision is a bit too optimistic for me, I agree that we’re at the dawn of a new way of interacting and connecting with one another.  While the growth of distributed communication platforms has amplified certain risks (the new borderless terrorist organizations, for example, thrive in distributed networks). The benefits are only beginning to become evident.

Future perfect is a great read, check it out if you have the chance.

 

Facebook Stock is Undervalued (or is it?)

Someone asked me to take a look at Facebook’s stock this evening, so I took a page from Aswath Damordaran’s valuations class and did a quick equity analysis, borrowing a number of his assumptions. You can download my worksheet here (Dropbox Link).

My analysis generated an implied market cap of about $48 Bn.  The stock closed today with a market cap of $41 Bn. implying that the stock is slightly undervalued by the public markets.

So is it?

I’ll let you decide, but regardless it’s a very risky stock and the future for the company is hazy due to the nature of its business (online and mobile advertising) and the newness of the media (social). The same factors that make it a game-changer generate significant risk.  To me, doing a 10-year discounted cash flow analysis on Facebook is like trying to guess what cars will look like in 50 years,  you might have an idea but so many things can happen between now and then there’s really no telling.  With that being said, it’s helpful to ground the speculation with revenue and margin estimates. A few big questions to ask related to this:

  • Is the team going to figure out mobile advertising?  Effective CPM rates on mobile (the price to advertise)  are significantly lower than desktop
  • Is corporate management and communication going to level expectations with the public markets?  The ‘story’ has much to do with tech valuations.
  • Will users get bored and go elsewhere?

I’d love any comments on the approach, or any thoughts on Facebook’s stock value.

Mobile Advertising Sucks and That Needs to Change

I was not surprised to see a recent report from Trademob finding that 40% of mobile ad clicks are either accidental or fraudulent.

When I interned for Jerry Neumann, he had me take a look at the mobile vs. display ecosystem, which I did. Back then I was interested in the differences between mobile and desktop traffic with respect to audience buying.  My conclusion from that exercise was that existing DSPs were best-positioned to figure this out when mobile ad buying demand was big enough, so the risks of creating a mobile DSP probably outweighed the opportunity.

But after reading the recent Trendmob report on click results, I’m starting to think that we haven’t been looking at the ecosystem in the right way.  I previously thought of mobile the way a lot of people look at mobile: a smaller version of our desktop experiences with some nuanced differences in cookie technology. I’m building some new theories about mobile advertising.  I’m starting with the statement that mobile advertising fundamentally sucks, but also that it’s important for  it to eventually not suck.

Related to my point above about DSPs, even if we get better at audience targeting on mobile, we haven’t solved the engagement problem. Selling clicks on mobile is not an indicator of anything other than a bot or my fat thumb accidentally hitting a banner.  What’s worse: more and more, we’re consuming our content on mobile devices.  If your primary revenue driver is advertising and your customers are increasingly consuming your product in a medium that’s a bad format for advertising, you are going to have a big problem on your hands in the near term.

According to Mary Meeker’s 2012 Internet Trends Report,  here are a few disparate data points that, in my opinion, spell disaster for the ad-supported media industry:

  1. Mobile Internet Usage surpassed desktop internet usage in India this year.  I’m betting that much of the world is moving in the same direction.  While mobile traffic won’t replace desktop access in more developed countries, it’s going to become an increasingly bigger piece of the pie.
  2. eCPMs on mobile are $0.75 vs. desktop which are about $3.50. Based on The Law of Shitty Clickthroughs, this isn’t a great start to the relationship between advertisers and consumers on mobile.  It will likely improve, but probably not by much.
  3. Mobile monetization levels could surpass desktop in three years (but not through advertising).  Most revenue in the mobile ecosystem is commerce-based, whether through game in-app transactions, app purchases, etc.  So, while monetization levels will increase, there’s no indication that media companies will take part in that.

I see this as an ominous sign for publishers, as well as an opportunity to create value.