Conquering Crawl Errors and Indexing Issues
Ok time to get down and dirty with the geeky shit. Google has web crawlers that look at websites, they analyse them by looking for what keywords they have on the page, what images you use, which websites your pages link to, and a whole host of other things that I won’t bore you with.
The point of all this is to help Google rank your website. Along the way of crawling your website, Google will encounter errors that either stop it from crawling your website in its entirety or perhaps if the problem is severe then it may not be able to crawl your website at all. In this article, I’ll go through the main issues that will hinder your SEO efforts.
Conquering Crawl Errors and Indexing Issues
Ok time to get down and dirty with the geeky shit. Google has web crawlers that look at websites, they analyse them by looking for what keywords they have on the page, what images you use, which websites your pages link to, and a whole host of other things that I won’t bore you with.
The point of all this is to help Google rank your website. Along the way of crawling your website, Google will encounter errors that either stop it from crawling your website in its entirety or perhaps if the problem is severe then it may not be able to crawl your website at all. In this article, I’ll go through the main issues that will hinder your SEO efforts.
Site-Level Errors
Site-level SEO errors can be something like your DNS settings not being correct, this means that your domain isn’t connecting to your hosting provider. This could be for a number of reasons such as a lack of bill payment leading to service suspension (check your emails regularly), a problem with the hosting server, or your web designer hasn’t connected the domain and the host properly. Most DNS problems will be blatant because your site just isn’t showing at all when you type in your www.site.com
So your domain is like your phone number and your hosting provider is where the files for your website are hosted. Some domain providers offer domain registration, which usually costs around $10 per year. Some domain providers offer domain and hosting together in one package costing around $10 per month for one website. I have found that buying a domain with a domain provider and then buying your hosting from a specific hosting-only provider is good for getting the best price.
Robots.txt Fetch Errors:
A robot.txt is a .txt file that lists out places crawlers can look and where they can’t look, this also tells which crawlers are allowed to look at the website. Some SEO’s will often block certain web crawlers from a website. The reason you might want to block a crawler from your website is that you may not want your competition to know your backlink profile, which is useful for competitors as they can get the same links and possibly outrank a website. If when crawler can’t see a robot.txt it might not even look at the site.
Websites like AHrefs, SEMrush, Magestic SEO, will use web crawlers to gather information about your website which you can then access through their website. Websites like AHrefs will then offer you some of your data for free, more if you connect the Google search console to both your website and AHrefs. Ahrefs once connected to your site will offer you a much needed analysis and actually will give you a free report on the errors that are on your website. The errors that I’m talking about in this very article will be revealed by tools like AHrefs.
Site-Level Errors
Site-level SEO errors can be something like your DNS settings not being correct, this means that your domain isn’t connecting to your hosting provider. This could be for a number of reasons such as a lack of bill payment leading to service suspension (check your emails regularly), a problem with the hosting server, or your web designer hasn’t connected the domain and the host properly. Most DNS problems will be blatant because your site just isn’t showing at all when you type in your www.site.com
So your domain is like your phone number and your hosting provider is where the files for your website are hosted. Some domain providers offer domain registration, which usually costs around $10 per year. Some domain providers offer domain and hosting together in one package costing around $10 per month for one website. I have found that buying a domain with a domain provider and then buying your hosting from a specific hosting-only provider is good for getting the best price.
Robots.txt Fetch Errors:
A robot.txt is a .txt file that lists out places crawlers can look and where they can’t look, this also tells which crawlers are allowed to look at the website. Some SEO’s will often block certain web crawlers from a website. The reason you might want to block a crawler from your website is that you may not want your competition to know your backlink profile, which is useful for competitors as they can get the same links and possibly outrank a website. If when crawler can’t see a robot.txt it might not even look at the site.
Websites like AHrefs, SEMrush, Magestic SEO, will use web crawlers to gather information about your website which you can then access through their website. Websites like AHrefs will then offer you some of your data for free, more if you connect the Google search console to both your website and AHrefs. Ahrefs once connected to your site will offer you a much needed analysis and actually will give you a free report on the errors that are on your website. The errors that I’m talking about in this very article will be revealed by tools like AHrefs.
URL-Level Errors
404 Errors:
A 404 error is where a page links to another page that no longer exists. Pages are deleted on a regular basis on websites as it’s a part of web development. Web developers need to be aware of 404 errors as these hurt SEO.
The best thing to do if you delete a page and re-purpose the same content to another page is just to create a 301 redirect to the new page. This allows the SEO juice that the original page was created to flow into the new location. This also helps in situations where an outside website is linking to a page that no longer exists on your website. When a user clicks on a link on another website and then visits your website, they will get to a 404 error page on your website this leads to a bad web experience, Google is keen to avoid these instances.
500 Errors:
A 500 error indicates a server-side issue preventing the search engine bot from accessing a webpage. This can be caused by server misconfigurations, resource limitations, or programming errors in the website’s code.
URL-Level Errors
404 Errors:
A 404 error is where a page links to another page that no longer exists. Pages are deleted on a regular basis on websites as it’s a part of web development. Web developers need to be aware of 404 errors as these hurt SEO.
The best thing to do if you delete a page and re-purpose the same content to another page is just to create a 301 redirect to the new page. This allows the SEO juice that the original page was created to flow into the new location. This also helps in situations where an outside website is linking to a page that no longer exists on your website. When a user clicks on a link on another website and then visits your website, they will get to a 404 error page on your website this leads to a bad web experience, Google is keen to avoid these instances.
500 Errors:
A 500 error indicates a server-side issue preventing the search engine bot from accessing a webpage. This can be caused by server misconfigurations, resource limitations, or programming errors in the website’s code.
Duplicate Content
Let’s say that you’re writing an article, and you’re stuck for ideas on how to write a certain part, so you decide to just copy and paste some content from another website, sure you can do this, your readers probably won’t even notice, but this is bad, and also a little unethical.
Websites like Copyscape can find content that you have copied and pasted and tell you where it has been copied and pasted from, Google can also do this and it is against websites that copy and paste from other sites, so keep your content original and add your own spin a popular topic. If you can’t write this content off the top of your head, then hire a writer on this topic that does know the topic well.
Legal issues
As well as Google not condoning copying and pasting from other websites, other businesses that wrote the content in the first place will also not like this as it is copyright infringement so against the law. Always be original in your content, and before publishing always run your written content through a site like copyscape, because it is still possible that although you think your content is original, it still might not be and Google can’t tell if your content is genuine or not.
Low-Quality Content
Google has some issues with low quality content. Low quality content to Google will be pages that only have perhaps 20 words on the page. Although Google prefers pages that are around 1200 words of content, they won’t penalise you if you’re the only one writing about “under water wicker weaving competitions” and you only happen to write 200 words. If you’re the only one writing about a topic then you’re most likely to rank anyways for this keyword.
Google also sees content that doesn’t offer anything over and above the existing content as low quality. Perhaps you work as a digital marketer and you’re writing about “What is SEO” then you’re most likely not going to be able to offer anything new that hasn’t already been discussed in regards to “What is SEO”. While you won’t get a ranking penalty for writing about a subject that had already been written about a million times, you probably won’t rank well for this keyword. While there are new things that come up in SEO all the time, you need to include these over and above the basic content.
Low-Quality Content
Google has some issues with low quality content. Low quality content to Google will be pages that only have perhaps 20 words on the page. Although Google prefers pages that are around 1200 words of content, they won’t penalise you if you’re the only one writing about “under water wicker weaving competitions” and you only happen to write 200 words. If you’re the only one writing about a topic then you’re most likely to rank anyways for this keyword.
Google also sees content that doesn’t offer anything over and above the existing content as low quality. Perhaps you work as a digital marketer and you’re writing about “What is SEO” then you’re most likely not going to be able to offer anything new that hasn’t already been discussed in regards to “What is SEO”. While you won’t get a ranking penalty for writing about a subject that had already been written about a million times, you probably won’t rank well for this keyword. While there are new things that come up in SEO all the time, you need to include these over and above the basic content.
Meta Tags and Directives Issues
Meta tags are needed to help search engines place your content in the search results. While Meta Tags are being phased out as search engines get smarter by using machine learning in how they analyse your website, they are still needed. Meta tags consist of a Meta title, Meta description, and meta keywords, these go into a page’s back end. If you don’t have these on your pages and have only realised now it’s not a big issue, get them onto all your pages, especially you home page, and should see a booth in rankings.
Duplicate metadata, it’s important that you don’t duplicate any of this data across your pages. Duplicating meta data can confuse search engines in how they places you in the search results. Make sure each webpage has a unique, descriptive, and keyword-optimized title tag that accurately reflects the content of the page.
I know these issues look techy and only a few people can understand them, taking the time to research these issues and correct them will help your website’s visibility, these problems are easier to correct than you think. Learn the ins and outs of errors, fix ’em, and watch search engine visibility soar. Stay savvy, use cool tools like Google Search Console and AHrefs, and keep that organic traffic coming!