My company’s SEO team lead came to me with an interesting problem and I thought I share it with everyone.
It so happens that his team were doing SEO related work on an e-commerce website and they were seeing a lot of the website pages indexed in Google which had the text
The page you're looking for doesn't exist. If you followed a link from elsewhere in the site, please contact us. Return to home page
One look at the content on these pages and I assumed that it was their custom 404 page however Google thought otherwise. It was treating all these pages as duplicate content pages and the website ranking was getting affected by this.
As the website had not been developed by us we contacted the concerned technical team for that website and the client and informed them regarding this issue. They came back to us a couple of days later saying that these pages are 404 pages and this is a non issue for them. To put it short they did not believe us that their 404 pages could be the cause of the duplicate content problem.
I then decided to check the status that was being returned when anybody browsed the website and voila we got our answer. All the so called custom 404 pages returned a 200 status meaning they really existed. The web server should have sent out a 404 status if they really were 404 pages. This would have informed the bots which could then term those pages as ‘non-existent’ and the problem would have been resolved.
The pieces started to come together.
Google considered all these pages as actual pages and due to the same content being displayed on all of them the website was being penalized for duplicate content. The problem was more to do with the e-commerce software they were using to run their website as it did not send out the 404 status. It was simply displaying a page and saying that the page was not found.
This incident should be an eye opener for website owners, SEO’s and web developers alike as a simple mistake by your software could bring down your website rankings like nine pins.
Hope the above helped. Comments?
Check for 404 status with online 404 checker utility I have developed.