In the recent Webmaster Help video released yesterday, Matt Cutts, Google's Head of Search Spam, explains the difference in how Google treats 404 and 410 error status codes. The question posed was:
“Does Google do anything different when encountering a 404 versus a 410?”
Matt answered that both 404 and 410 are request errors and imply that the page is not available. 404 denotes that the server can't find the requested page. The server often returns this code if the page in question doesn't exist on the server. Whereas, 410 code is returned when the requested resource has been permanently removed.
Matt confirmed that Google treats both these status codes differently. In case of a 404 error and a few other status codes falling under the 4xx category, Google protects the page by not marking it as removed for around 24 hours. On the other hand, in case of a 410 request error, Google marks the page as “gone”, immediately after the error is detected by Googlebot.
According to Matt, Googlebot goes back and checks both status code responses later to ensure that the page is really not there. It keeps on checking the codes for several times over the years, to see if the page is brought back again.
If you don't have a robots.txt file on your site, you will see the 404 status on the blocked URLs page in Google Webmaster Tools. But if you see this status, although you have a robots.txt file, then it is likely that your file is named incorrectly or is in the wrong location. According to Google, the robots.txt file should be at the top level of the domain and named robots.txt.
The 410 code is sometimes used in place of 404 for pages that existed earlier but no longer do. Google also recommends in its Help section that if a page has been permanently moved, you should use a 301 to specify the new location.