Panelists explored how you can ensure that large sites with complex databases and lots of content are efficiently crawled by Google.  The session covered everything from canonicals and crawl priority to pagination topics.

Conquering Today’s Technical SEO Challenges: #SMX New York 2014, Day 2!


Alex Bennert  (@alexbennert), SEO Consultant, Independent

Q&A Coordinator:

Jason White (@sonray), Director of SEO, DragonSearch


Mark Munroe (@markemunroe), SEO, SEORadar

Anthony Piwarun (@apiwarun), Manager, Search Engine Marketing, Zeon Solutions

Matt Storms (@matt_storms), Independent Consultant, Independent Consultant

Conquering Today’s Technical SEO Challenges: #SMX New York 2014, Day 2!

Matt Storms

Matt Storms came up with tips for Infinite Scroll and AJAX SEO issues.

Infinite Scroll

It is where you can keep on scrolling and never reach the bottom of the page. When you scroll downward, it would load more content, providing a never-ending amount of material.

He mentioned a few famous sites like Twitter, Facebook, LinkedIn, Chicago Tribute, LA Times, FindNSave, Pinterest and Google Images and asked which one of these was truly an infinite scroll site.

Breaking content into chunks

Take stock of the content that you wish to infinite-scroll and break it up into chunks. Make sure that the users are able to find whatever they are looking for without the scroll going unreasonable. If the user has to keep on the scrolling for long, it will only turn out into disappointment.

Using JavaScript and Ajax

Create a URL structure which accommodates infinite scroll. Avoid two things as you structure the URLs:

  • Relative-time-based URL parameters
  • Code-based language in the URL

Keeping with best practices

Ensure that the pages have optimal load times and do not duplicate the content of other pages. Each page should include rel=“next” and rel=“prev” within the <head> tag.

PushState Method

It is an HTML5 method which dictates the load process. It will draw from the state object (anything serialized) and the URL to the load the content in sequential manner as determined by the pagination.

PushState directs the browser what to load and display. This is the technique Twitter uses to implement infinite scroll.

ReplaceState Method

ReplaceState method is used for modifying the history entry or responding to user-scrolling behavior. The method would respond to scroll actions and allows the user to view pages which were loaded in the pagination history.

Testing infinite scrolling page

Make sure that every page loads accurately and in the right time frame. Follow it with creating the best and cleanest content possible.

Build better sitemaps

Matt advised to build better sitemaps. He attempted to take home his point with analogy of building blocks.

Track bots crawling your site

He asked the audience who has used New Relic to show hands. Using custom attributes to track Googlebot, Bingbot and others can be effective. Find how the search engines are crawling and what can be done for the change.

Setting up analytics

You can choose from Google Analytics, Adobe Analytics among many others.

Page parameters

You need to monitor your page parameters. Matt explained when Google detects duplicate content such as variations caused by URL parameters, it groups the duplicate URLs into one cluster. It would then select what it determines to be the best URL to represent the cluster in search results.

Properties of the URLs in the cluster, such as link popularity, are consolidated to the representative URL. The process often provides users with more accurate search result.

There are advanced features and you must play with them on your own site before using them on client sites.

Issue with Googlebot Indexing

Matt explained how he dealt with Googlebot indexing issue. He worked on pagination and canonical.

Show as many pages which make sense. For instance,Prev, 1, 2, 3, 4, 5…& Next. Spiders will be able to see the next page with Javascript turned off.

Matt changed the canonical of page #2, #3 and so on.

They did not want to index the Infinite Scroll pages with JavaScript. It will send a signal to the spiders that the page is different, encouraging them to scroll it.

Conquering Today’s Technical SEO Challenges: #SMX New York 2014, Day 2!

Keeping low quality pages out

To prevent spammy looking duplicate pages from rendering in the indexes, Matt and his team incorporated the noindex, follow tag on each of the /?page=2/pages.  They did not have to worry about the page, but more about the items.

Clear signals

These include Canonical, Prev& Next, Noindex, Follow

Load times

You need to ensure the pages have optimal load times and do not duplicate the content. He also asked the audience questions like:

  • What is an ideal load page time?
  • How many of you (audience) had an Infinite Scroll site?
  • Do you like sites that use it?
  • Do you feel this is the future of sites?

Bonus Tip

Conquering Today’s Technical SEO Challenges: #SMX New York 2014, Day 2!

People are unsure whether 301, 404, or 410 is better for product sites. He explained:

301 – old page to new page

302 – should never be used

404 – the page is not found, but search engines will keep coming back to find it

410 – will kill the page within a week

If there are links to the product page, choose 301.

If no links are there, or you do not care, select 410.

Matt concluded by informing that 2.2 billion page were indexed in less than 4 months.

He termed infinite scroll sites the future.

He gave an example of LA Times. You begin reading an article and reach to the bottom, where you find another article similar to what you just read. It is likely to keep you on the site longer.

Mark Munroe

Mark Munroe also provided useful insights into the technical challenges faced by SEOs.

Q&A Session

During his session on technical SEO at SMX, Matt Storms asked attendees, "How many of you consider infinite scroll to be the future of web pages?" But, nobody raised their hands and Storms was surprised. So, a debate began where SMX experts weighed the pros and cons of the infinite scroll, the popular site design.

In favor of Infinite Scroll

The first thought most of the attendees had was the infinite scroll sites are annoying. It was a notion popularized by xkcd. But, it's a layout that appeals to the preferred web behavior of people. Here's the pros aspect covered in the discussion:

The most popular non-commercial websites are doing it, so the average user knows it.

Matt said, "When you use Facebook, you scroll infinitely to see the friends who are doing – till you get bored. This can be up to 30 minutes".

It's mobile friendly

Alex Bennert, a consultant formerly of The Wall Street Journal said, "If we think about how frequently people are searching the web on mobile, it makes sense when they use a hand to literally scroll".

Promotes endless content consumption, especially in the case of ecommerce and news sites

Matt said, "he believes that infinite scroll is the future for news sites. It is a great way of showing endless related content for more time on the website and engagement. The same can be said for ecommerce sites.

Moreover, infinite scroll can help in building KPIs for publishers. Since, it keeps people on the site for longer, the impressions can get a boost.

Can create seamless experience, when done right

Matt said that if the design and the tech integration are solid, people will prefer infinite scroll as the option. It's modern and also where the web is right now, said Storms.

Against Infinite Scroll

Performance issues including the load time

Marcus Tober of Searchmetrics that that a lot of ecommerce websites are turning back to regular designs from infinite scroll, which he says could be because of infinite scroll. Even Storms agreed that it could be an issue. He said, "I admit, the biggest problem with infinite scroll pages is getting the load time under a few seconds".

Navigation nightmare

Mark Munroe of SEORadar said that sometimes people want to look at individual product pages and jump to them quickly.

No inclusion of Google News

"Infinite scroll exceeds the file size for Google News accepted pages", said SEO consultant Alex Bennert.

The Conclusion

As far as user experience is concerned, there's no right answer, unless it comes from the visitor. The core of any design decision should be the customer, else the experience will suffer no matter how great it looks.


Navneet Kaushal is the Editor-in-Chief of PageTraffic Buzz. A leading search strategist, Navneet helps clients maintain an edge in search engines and the online media. Navneet is also the CEO of SEO Services company PageTraffic which is one of the leading search marketing company in Asia.