One of the most important things to keep in mind when building a new website is making it crawlable. Crawlability is one of the most important factors in determining how quickly your site is indexed in the search engines. After all, if they can’t crawl your site then they can’t index it.
Below are a few tips to help you make your site more crawlable and therefore get indexed more quickly:
- Do away with dynamic URLs. Static URLs are more crawlable than dynamic URLs. If you have a content management system then this could be an issue for you. Dynamic URLs tend to have more special characters in them such as ? and =. They may have user IDs in them or session IDs. All of that is fine for tracking purposes, but make sure you have one static page without all of that information so that search engines can crawl your site more easily.
- Use a robots.txt file to block the search engines from pages they can’t crawl anyway. Pages like contact forms, login pages, and shopping carts slow down the search bots and won’t get crawled anyway, so block them.
- Get rid of duplicate content. If you have more than one page with the same content then use the rel=canonical URL tag on your page.
- Ensure you have a good site architecture. Is there a logical flow to your link patterns internally?
- Use sitemaps and register them with the search engines.
- Make sure you have at least one inbound link to your site; otherwise, it will never get crawled.
Making your website more crawlable isn’t difficult. It just takes a little planning and some know-how.