In a recent blogpost Gary Illyes has explained crawl budget is, crawl limit, crawl demand, and factors that affect crawl budget.
Gary says, there is no single term to define crawl budget and it’s not something most publishers have to worry about. Prioritizing what to crawl, when to crawl and how much resource the server hosting site can allocate to crawling is more important to bigger sites or to the sites that auto-generate pages based on URL parameters.
- Googlebot is designed to crawl the website pages without degrading user experience. Crawl rate may go up and down depending upon crawl health and limit set in Search Console.
- Crawl demand means how much Googlebot crawls your pages depending upon the popularity of your website and the staleness of the URLs in the index.
- Crawl rate and crawl demand together determine the Crawl Budget. “Taking crawl rate and crawl demand together we define crawl budget as the number of URLs Googlebot can and wants to crawl.”