Remember how fun treasure hunts were as a kid? Remember how fun finding the stuff you need on the Internet is? (The answer to both those questions is YES!, in case you were wondering). A good website is the treasure of the online community, whether the gain is in information, products, or entertainment for the user. And, just like any other treasure, you always need a map.
In this case, the map is for the Google bots that crawl the web. They find your treasure (website), evaluate it, and then make it accessible to users by placing it somewhere in the line-up of other treasure. This is called getting indexed.
I’m going to be focusing on getting indexed by Google, but in reality, if you are doing all of these steps, you are more likely to get indexed by any search engine.
What does it mean to be indexed?
Being indexed means that search engines like Google know your page exists and that it comes up in a search. Sometimes only some of the pages for a root domain are indexed, so it is important to check all of your pages that you want to be found.
How can you check to see if you are indexed?
Checking is really easy! In the Google search bar, type in site:yourdomain.com, and if it has been indexed by Google, it will come up. If not, you will get a message saying that it cannot be found.
If you want to check when your site was indexed, search cache:yourdomain.com, and at the top, you will find the last date that the site was indexed.
What do you do if you are not indexed?
There are a few different reasons that your page might not be getting indexed.
Problem: Something is telling the bots not to index your page.
Solution: Search the source code for “noindex” and “nofollow” commands that might be telling Google not to index your page. Control “u” pulls up the source code and control “f” will pull up a search bar to help you search for these terms.
Also check www.yourdomain.com/robotstxt to make sure there are no rules that are blocking your page from being indexed.
Problem: You have an old page that is stealing your new page’s glory, and by glory, I mean search results.
Solution: If your URL or page is a replacement for an old URL or page, make sure you have a 301 permanent redirect. This will automatically direct traffic and bots from your old site to your new site.
Problem: You don’t have a sitemap.
Solution: While having a sitemap doesn’t guarantee that Google is going to immediately index your page, it makes it a lot easier for the bots to find all of your pages because you will have them all listed in one spot. Check your site to see if you have a sitemap already by typing in www.yourdomain.com/sitemap.xml. If you don’t have a sitemap, check your CMS (Blogger, WordPress, etc.) to see if you can add one through there. You can also download an XML sitemap generator like GSiteCrawler and create a sitemap yourself.
Be sure to update your sitemap every time you add or subtract pages from your site. You wouldn’t change the location of your treasure without updating your treasure map, right?
Problem: You have a sitemap, but it’s complicated.
Solution: Sometimes your site map needs to be cleaned up a bit. Check over it and take out old or duplicate URLs. Duplicate URLs are URLs that might have minor differences but are designed to take you to the same page. For example, www.yourdomain.com and www.yourdomain.com/ are considered duplicate. If this happens, check which URL works and delete the extras.
Problem: Your website might have content that is duplicated somewhere else on the web. Search engines want unique information to present to their users, so they will rank duplicate content lower or even eliminate it from their index completely.
Solution: A 301 Redirect is the most common way to deal with this issue. You also might end up rewriting content if you find that it is duplicated on a page that you don’t own.
Problem: You’ve done everything right, but it’s still not indexed.
Solution: In this case, it might just be a matter of time. While you wait, you can be making external links. Talk about your pages (and link back to them) on social media sites.
Going through these steps shouldn’t take too long, but you can make all the difference in whether your website gets found by treasure-hunting bots and users. An additional resource to ensure your pages are indexed is Google Webmaster Tools, an extremely useful tool available for FREE to everyone. You are now experts on directing to your treasure—happy map-making!
