Look how old this is! I post at SearchCommander.com now, and this post was published 19 years 1 month 18 days ago. This industry changes FAST, so blindly following the advice here *may not* be a good idea! If you're at all unsure, feel free to hit me up on Twitter and ask.
The Google sitemap program may still be in beta, but it’s 100% effective, for them and for you, the owner of a website that wants to be found. You can easily get your entire site regularly crawled by the most powerful search engine the world, that is probably not doing it now. Why wouldn’t you? Oh, and did I mention that it’s free?
They are not charging anything for this, because Google wants to become the most information filled resource of RELEVANT information in the world. That’s great for them and for you. They earned my business over 5 years ago and in my opinion, they’ll soon dominate over 50% of the search results.
Since Google started it’s sitemaps program, they come back MUCH more often, and they don’t bounce around my site when they do. ( I know this because my web stats show robot traffic) All my new pages get visited immediately, and continue to get checked for any changes.
The other search engines will not do this without help from you and a product like Robot Manager but Google is smart. They actually WANT your information to get to them, and they’ll reward you for using their free service by sending you customers via the search results.
The result of my “deep and frequent robot visits” is that I now rank on page 1 for “technology consultant” out of 71 million results. AND I’m on Page 2 for both “computer consultant” AND “internet consultant” with over 40 million results. That isn’t because I have the largest or most relevant website out of 70 million sites out there! It’s because the search engines know I’m here, and they can see exactly what’s on my website.
It may not be worth much right now for your site, but it will be over time if you add content. For growing sites with frequent additions it’s free insurance for frequent and consistant Google crawls. How long can it be before someone else says “okay, name it “MSN.xml.gz” and we’ll follow it too.
If you, manage, promote or maintain your own website, you should go set up a Google account, it only takes a minute. Then, assuming you are technically capable, follow these directions. If you have a webmaster, show them the directions. It’s not hard. Then, within 12 hours of submission, your site will be crawled by Google on a regular basis, until…well, until… Until Google says so.
* No, this has nothing to do with getting new sites out of the sandbox (which doesn’t officially exist)
