More data beats better algorithm at predicting Google earnings

Readers of this blog will be familiar with my belief that more data usually beats better algorithms. Here's another proof point.

Google announced earnings today, and it was a shocker -- for most of Wall Street, which was in a tizzy based on ComScore's report that paid clicks grew by a mere 1.8% year-over-year. In the event, paid clicks grew by a healthy 20% from last year and revenue grew by 30%.

In comparison, SEM optimizer Efficient Frontier released their Search Performance Report on their blog a few hours ahead of Google's earnings call. EF manages the SEM campaigns of some of the largest direct marketers, handling more SEM spend than anyone in the world outside of the search engines themselves. Their huge volumes of data give them more insight into Google's marketplace than anyone outside of Google.

EF reported a 19.2% increase in paid clicks and 11.2% increase in CPCs at Google Y-O-Y. Do the math (1.192*1.112 = 1.325), that's a 32.5% Y-O-Y revenue increase. That's the closest anyone got to the real numbers!  And this quarter is not a flash in the pan: in January, EF reported a 29% Y-O-Y increase in SEM spend, with 97% of the increased spend going to Google: that is, about a 28% Y-O-Y revenue increase for Google. That compares very favorably with the actual reported increase of 30%.

As Paul Kedrosky points out, this is a huge indictment of ComScore's methodology (ComScore's shares are trading down 8% after-hours post the Google earnings call). ComScore sets a lot of store on their "panel-based" approach, which collects data from a panel of users, similar to Nielsen's method of collecting data on TV viewing using data from a few households that have their set-top boxes installed. ComScore has been in this business longer than anyone else, and has arguably the best methodology (i.e., algorithm) in town to analyze the data. They're just not looking at the right data, or enough of it.  Some simple math using the mountain of data from EF handily beats the analysis methodology developed over several years using data from a not-so-large panel.

To my mind, this also puts in doubt the validity of ComScore's traffic measurement numbers. For websites where I personally know the numbers (based on server logs), both Quantcast and Hitwise come far closer to reality than ComScore. The latter two don't rely as heavily on a small panel. ComScore's value today is largely driven by the fact that advertisers and ad agencies trust their numbers more than the upstarts. Advertiser inertia will carry them for a while; but a few more high-profile misses could change that quickly.

Disclosure: Cambrian Ventures is an investor in EF. However, I don't have access to any information beyond that published in their public report.

Comments

[RT]

So, how does Efficient Frontier collect the additional data? Hitwise sits in the middle of the traffic (not at the client-end, not at the server-end - but in the middle - at the ISPs) and therefore does not need any panel based collection approach. It does not appear from their website that EF sits in the middle of the traffic, so how does it get "more" data?

[Anand Rajaraman]

Think of the SEM ecosystem as a big pipe: at one end are advertisers, at the other end Google, Yahoo, and Microsoft. EF manages the accounts of some of the largest advertisers, amounting to a statistically significant fraction of Google ad revenues. Also, EF's clients are worldwide, not just US, so they have a great sample of the overall search advertising revenue stream flowing into Google, Yahoo, and Microsoft.

The nice thing, EF has both the bid data (dollars and cents), as well as the ad clickthrough data. Hitwise, comScore, etc can measure only clickthrough data; they don't know the value of each click.

[Scott]

This is a great analysis ... can you provide a similar analysis of AdGooroo's research report which indicated that advertisers were coming back to Google? This report was released about a day before earnings and (to me at least) indicated an earnings surprise may have been in the works.

http://www.adgooroo.com/google_gains_advertiser_share.php

What would be really interesting is to put together a model with all of these different data points and try to predict their revenues 10-15 days before earnings. You'd have a least one more avid reader if you did that...!

[Srinivas]

Have you seen this interesting response from Comscore?

http://www.alleyinsider.com/2008/4/comscore_google_s_us_business_does_stink

There is a still a fair amount of discrepancy between Comscore's US numbers (-9%) vs. Googles (0%), but the gap is smaller. Of course, your overall point is still valid though - Comscore didn't have non-US data while EF did which turned out to be pretty useful. In any case, I dont trust many of these panel approaches either! On the TV ratings measurement side, TiVo is trying to upset Nielsen's TV ratings by collecting data from lot more people (TiVo users) and at finer granularity. (Disclosure: I worked on that project at TiVo).

[Anand Rajaraman]

Srinivas, thanks for the pointer. I'll take a look and post an update.

[david solomon]

I totally agree with Scott ...that would be some real great piece of help if you could do that !