Nine Months of Search Engine Data of My GoogleNumber

It takes nine months for us to grow inside the womb and apparently it takes nine months to really analyze how search engines crawl and get results.

Every week for the last nine months I have done a search for "Auren Hoffman" in eight major search engines (Google, Lycos, All the Web, Teoma, Gigablast, Wisenut, Yahoo, and A9). I recorded only the raw number of results as my objective was to understand the reach of each search engine rather than the accuracy. Since, as far as I know, I'm the only "Auren Hofman", the results are fairly finite and easy to define. (a chart and the raw data is below)

This experiment was born of my fascination with other people's fascination about their "Google Number" (see What Is Your Google Number). When I started, my Google Number was only 794. It climbed to 2880 in March. Today, it is inexplicitely only 2220 -- which means that either:
(a) about a six hundred pages with my name on it have dropped of the face of the earth in the last five months
(b) Google changed the way it crawls and stopped caching certain sites
(c) google misrepresents its raw search results (we'll get to that later)

My observations:

1. Fluctuation in weekly results
Some search engines have results that fluctuate widely every week and others are very consistent (see graph below). Google fluctuates widely each week even though it is obvious (from just looking at the search results) that Google spiders new pages very quickly. Search engines like Gigablast and Wisenut fluctuate very little -- and it also seems they rarely add newly crawled information to the search engines.

2. Numbers that end in zero
I noticed that for results above 1000, all Google and Yahoo results end in a zero. Maybe they are rounding to the nearest ten or maybe they are just making an approximation (which might be the reason that results vary so widely). If you follow every result, they both seem to overestimate the number of pages.

3. There is a huge discrepency between the search engines
Even if the results from Google and Yahoo are a bit inflated, they are still vastly higher than the other engines (though All the Web) indexes a good deal of pages too. That means that purely on who indexes the most pages, there is a big difference between the engines.

And indexing the most pages is very important. Now everyone always seems to talk about relevance. And relevance is very important. But so is mass. Especially when you are looking for something more archane.

What is your Google number??

See Larger graph here

	Google	Lycos	All the Web	Teoma	Gigablast	Wisenut	Yahoo	A9
12-Nov	794	649	636	582
11/16/2003	808	402	636	582
21-Nov	866	652	636	582
12/7/2003	949	419	406	540	540	956
12/13/2003	1020	418	404	537	267	984
12/18/2003	1070	411	404	539	268	1001
12/23/2003	1060	411	408	539	269	1001
3-Jan	994	409	402	539	269	1001
10-Jan	993	418	409	539	269	307
24-Jan	2390	693	699	537	272	1001
31-Jan	2700	677	693	557	273	1001
8-Feb	1810	696	680	557	282	1001
14-Feb	1570	693	703	509	285	1001
27-Feb	1890	719	703	509	280	1001	2070
5-Mar	1920	668	655	509	278	1001	2190
13-Mar	2770	690	674	509	278	1035	2140
3/20/2004	2880	649	483	568	278	1035	2190
3/26/2004	2970	636	307	568	278	1035	2400
4/6/2004	2160	275	361	568	278	1035	2180
4/10/2004	2010	278	421	567	278	1035	2100
4/18/2004	2200	290	335	643	278	1035	1580	295
4/23/2004	2330	274	290	643	278	1035	1790	297
4/29/2004	2380	268	293	643	271	1035	1730	320
5/9/2004	2390	271	276	680	293	1035		303
5/14/2004	2250	255	258	680	300	1035	1590	304
5/21/2004	1890	246	217	684	306	1035	1410	297
5/28/2004	2010	252	238	683	314	869	1390	303
6/6/2004	2080	261	335	706	317	869	1410	292
6/11/2004	1910	254	334	712	325	869	1350	299
6/18/2004	1670	250	439	712	336	869	1350	295
6/24/2004	1400	248	451	728	361	869	1340	284
7/2/2004	1430	256	484	728	363	664	1310	288
7/8/2004	1370	254	484	728	360	664	1290	278
7/16/2004	1390	253	1220	741	361	664	1250	305
7/23/2004	1510	251	1190	741	361	664	1230	309
7/29/2004	1550	251	1200	742	361	664	1240	321
8/6/2004	871	251	1210	758	361	664	1230	341
8/13/2004	1640	366	1170	756	361	664	1310	345
8/18/2004	1720	378	1180	133	361	664	1330	328
8/25/2004	1950	367	1230	765	361	664	1330	302
9/10/2004	2220	332	1210	765	361	660	1240	300