In his latest webmaster video, Matt Cutts said that responsive design of a website will not hurt the SEO, as long as the site is fast & working well for users.
The question asked was: Does a site leveraging responsive design "lose" any SEO benefit compared to a more traditional m. site?
Whenever you have a site that can work well for regular browsers on desktop as well as mobile phones, there are a couple of completely valid ways to do it.
Matt Cutts addresses responsive design to be one valid way. A responsive design means that the page works totally fine when you access the URL with a desktop browser or whether you access the URL with a mobile browser. Things will rescale, page size will be taken into account, and everything works fine.
Another way to do it is depending on user agent that is coming, you can do a redirect. So, a mobile phone for example might get redirected to a mobile. version of your page, which is totally fine. Google has guidelines and best practices for things like having a rel=cannonical from mobile version to desktop version etc.
Matt Cutts said that in general, he would not worry about site that uses responsive design losing SEO benefits because by definition, you get the same URL. So, in theory, if you have a mobile version of your site, if you don't handle that well by not doing rel=cannonical and all, you might in theory divide the PageRank between those two pages. But, if you have a responsive design, everything is handled from one URL. So, PageRank doesn't get divided. Everything works fine. So, you don't need to worry about SEO, drawbacks of responsive design and all. You have to ensure that your page works well for user, its fast, and not about responsive design hurting you in terms of SEO.Responsive Design Doesn't Hurt in Terms of SEO - Matt Cutts!,