It is imperative that the search engines index all of your webpages, so that they are aware of each pages content and are in a position to serve them in the search engines results pages.  If you own one of the many websites where only certain pages have been discovered, then this Joomla SEO article will help you get the major search engines to index all of your webpages.

How to find out which pages are indexed

The first step in getting the search engines to index all of your webpages is to find out which pages they have indexed and which pages they have not yet found.  The easiest way to do this is to use the site:domain search parameter, this works with Google, Yahoo and Bing.

Here is how you go about doing this.  Lets assume your domain name is mywebsite.com, open the search engine that you wish to check which pages it has indexed and type the following, without quotations, in the search box "site:mywebsite.com".  The resultant page will list all the pages that that particular search engine has indexed on your domain.  If you have pages that are not in this list and have not been blocked intentionally via your robots.txt file, the search engine in question probably has no knowledge of its existance.

How to ensure the search engines index your entire site

When you find a page that is not indexed by the search engines, here is what you need to do:

Check and ensure that the search engines are not blocked from indexing the missing pages

Your first step is to have a close look at your robot.txt file and make sure that it adheres to the rules of a robot.txt file.  Many webpages get inadvertantly blocked due to errors in robot.txt files.  This is the most common reason for pages not being indexed.  Additionaly, check that you do not have a "noindex" meta tag on the page concerned, if you do, remove it as this is a clear instruction to the search engines not to index the page.

Create a sitemap using the correct protocol

Google, Yahoo and Bing all support a sitemap file.  This is not the search engine friendly Site Map that you see on most websites, including seojoomla.co.za, which are designed for humans, allowing them to find the pages they are looking for.  The sitemap we refer to here is not designed for humans at all and uses the sitemap protocol.

You can read more on the specific format that sitemaps need to adhere to at sitemaps.org.  You do not need to make use of all of each aspect of the specifications on that website, if all you need to do is ensure that the major search engines index all of your pages, keep reading, we will cover how to do this later on in this article.

Edit your robot.txt to allow auto discovery of your sitemap

The search engines have agreed to an extension of the robots.txt file to accommodate the sitemap protocol.  Once you have uploaded your newly created sitemap file to your websites root, add the following line to your robot.txt:

Sitemap: http://www.mywebsite.com/sitemap-file.xml

Changing "mywebsite.com" to your domain name and "sitemap-file" to the name of your sitemap file.

The search engines will always look at the contents of your robot.txt file before spiddering your website.  When they see your sitemap, they will load it to discover what is inside and will follow all of the links, indexing all of your pages.

How to create a sitemap file

To create a sitemap file, use any ASCII editor, if you don't have one and use Windows, Notepad, in the Accessories menu, works well.  Be sure not to use any of the Word Processing packages like MS Word.  Here is an example of what your sitemap file might look like:

<?xml version="1.0" encoding="UTF-8"?>
<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
  <url><loc>http://mywebsite.com/</loc></url>
  <url><loc>http://mywebsite.com/joomla-seo-specialists.html</loc></url>
  <url><loc>http://mywebsite.com/joomla-seo-packages.html</loc></url>
  <url><loc>http://mywebsite.com/joomla-seo-problems.html</loc></url>
</urlset>

Notice that the sitemap file starts with these two lines:

<?xml version="1.0" encoding="UTF-8"?>
<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">

and ends with:

</urlset>

This never changes, all sitemap files need to start and end this way.  Notice that every page that you want indexed is listed in the following format:

<url><loc>http://mywebsite.com/joomla-seo-specialists.html</loc></url>

You need to replace "http://mywebsite.com/joomla-seo-specialists.html" with your own domain and page names.  Save the file as sitemap.xml, in the root directory of your website and remember to add the sitemap line in your robots.txt file, as discussed previously.

A sitemap file needs to be smaller than 10MB and cannot contain more than 50 000 web addresses, if you need to accommodate more, you will need to create multiple sitemap files.