How Much Time Does It Take Googlebot To Spider 29.5 Million URLs?

Sep 12, 2007 | 1,660 views | by Navneet Kaushal
VN:F [1.9.20_1166]
Rating: 0.0/5 (0 votes cast)

How long for Google to spider 29.5 million pages? asks a forum in Webmaster World.

dberube posts:

"We launched a new site about 2 weeks ago that has about 29.5 million unique, search engine friendly, URLs. Since we launched Googlebot has visited us 22 times and has hit just over 2,000 files. My question is, how long do you think it would take Google to spider the entire site and get them into their index?"

Some of the answers are:

Pageoneresults writes, "PageRankâ„¢ is the determining factor when working with that volume of pages. Your best bet is to release groups of pages for indexing as you garner more PR. Block all but the most important ones right now. Without the PR, you're going to have 29.49 million pages in the Supplemental index at the end of those 56.52 years."

mbennie writes,

"I have been helping a friend with a new site that has 6.8 million uri's of reasonably original content. The site went live 3 weeks ago. G-Bot started grabbing about 1K pages/day after 10 days. It kept on that schedule until a few days ago when it increased to 30K pages/day. I expect that speed will increase once again before too long. The fact is that G-Bot could crawl the entire site in less than a day if it wanted to. G-Bot is very conscious of whether or not it is going to crash a server and from what I have seen it won't take more than about 150 pages/second at peak – but right now its throttled down to 1 page/2 seconds. I suspect it also grabs a small set of pages and checks them for spam/duplicate content before deciding to crawl a site more aggressively."

Discussion continued at Webmaster World.

4.thumbnail How Much Time Does It Take Googlebot To Spider 29.5 Million URLs?

Navneet Kaushal

Navneet Kaushal is the founder and CEO of PageTraffic, an SEO Agency in India with offices in Chicago, Mumbai and London. A leading search strategist, Navneet helps clients maintain an edge in search engines and the online media. Navneet's expertise has established PageTraffic as one of the most awarded and successful search marketing agencies.
4.thumbnail How Much Time Does It Take Googlebot To Spider 29.5 Million URLs?
4.thumbnail How Much Time Does It Take Googlebot To Spider 29.5 Million URLs?