An XML sitemap is an XML document that is submitted to the search engines (Google, Yahoo, etc) that essentially tells them where all your pages are located. It assists them in the crawling of your website. Google and Yahoo used to have their own XML format, but now there is a standardized protocol that can be used for both. Check out sitemaps.org for more details.

The format of the XML sitemap is the least of your worries, its fairly simple and basically is just a list of urls with frequency of updates and last modified dates. For a small site there are many tools available, both offline and online that make building a sitemap a piece of cake. Building sitemaps for websites with thousands of pages is when it gets tricky.

Google has a list on their website of all different types of tools. For a simple website that isn’t too big, I would recommend trying XML Sitemaps or Site Map Generator (yes, they all have very unique product names). They are both online tools and work pretty well, for the smaller sites that is (under 1000 pages).

For larger sites, with thousands of pages, there are the a few things to keep in mind:

1. It can take a long time to index a large site (hours, maybe days) - the site map generators spider the site like a search engine do
2. Your servers response time dramatically affects how long it takes to generate
3. Google only accepts 50,000 URLs per sitemap and files less than 10mb - if you have more than 50,000 URLs you have to break it into multiple sitemaps
4. Building a sitemap that spiders your site with multiple spiders (threads) can speed up the process, but can slow down the site for other users

I have tried several of the 3rd party downloadable tools on Google’s page, and nothing has impressed me. So far I have tried

• CoffeCup - simple to use, good for small sites
• A1 Sitemap Generator - just froze when I tried to start spidering
• GSiteCrawler - worked pretty well, a little complicated, more for the technical minded individual

Next up is Unlimited Sitemap Generator by xml-sitemaps.com.

More to come. Stay tuned.