Look how old this is! I post at SearchCommander.com now, and this post was published 16 years 11 months 25 days ago. This industry changes FAST, so blindly following the advice here *may not* be a good idea! If you're at all unsure, feel free to hit me up on Twitter and ask.
Am I the only person that didn’t know this? I just “accidentally” discovered that an entire website wasn’t ranked for ANYTHING in MSN. They had #1 ranking for the primary phrase in Google and Yahoo, yet could not be found anywhere in the top 200 results on MSN. After scrolling through 5 pages of SERPS, I guessed that it just wasn’t in the index, but I was wrong…
It was in the index alright, but only one page for the domain name, which was 301 redirected at the root. When I did a site search in Live, I was greeted with just one page, and when I hit the “cached page” this is what I saw:
So domainname.com had a 301 redirect on it to domainname.com/directory/, but instad of following the 301, the MS Live bot just completely ignored the server directive, and cached the server page. How dumb is that? Is this just common knowledge among SEO’s that I somehow missed?
I’m not here to debate the wisdom of redirecting the main page in the first place, but my point is that if the site owner decides to, for whatever reason, the bots should follow, and the majority (Google, Yahoo, Ask etc.) do just fine.
How to solve this dilemma? Well I’ll have to get to that in another post, (because I’ll have to figure it out) but for now I have a question…
Why does Microsoft bot not follow 301’s? Is it an inherent technical inability to do so, or is it actually a conscious decision? It REALLY seems stupid to me…
*Update next morning –
I’m out of town all week, working from a laptop / cell phone connection, and things are painfully slow, but here’s that I’ve found out so far…
1. This is not a new problem, and others have seen it before, going back all the way to 2004.
2. In 2005 someone else noticed it, but nobody even responded.
3. Nearly 2 years later, it was still unresolved, and now, 3 years later it’s still not.
I wish I would’ve known this before I changed my permalinks last week, since now ALL 250+ of my blog pages will likely drop completely out of MSN / Live / Microsoft, or whatever we’re supposed to call them….
This is a very interesting post and I truly think that this is something to look into.
Just like yourself, I really did not notice any problems with MSN not caching the right page when the live bot runs through the page.
But now that you mentioned this, I will definitely pay attention for now on.
This can also be seen as a warning for all of those marketers that use 301 redirects to stay away from trying to improve your pages MSN search engine ranking if it uses a 301 redirect.
yeah, this is crazy. I lied awake in bed last night thinking about how you could make the spiders go where you want them to go with robots.txt, but that doesn’t do anything for you if you’re changing page names or file structure.
The way you do things “correctly” is with the 301 redirect, and everybody knows that…
Now thats pretty weak. I’ve been using 301 redirects for a while, but that explains a few things about some of the links showing up in my backlinks. Cheers.
I am pretty sure MSN does follow 302 redirects..just noticed on a website a few days back..it was showing both versions in the cache.
And the same wasn’t indexed in Google (just the home page of the redirected version in fact) for more than 2 years.
(Jeez, that’s the hardest anti-spam math question I’ve ever seen on a blog.) 🙂
Scott — MSN is far and away the worst, but neither G nor Y are exactly great at dealing with 301s. In my experience, the bigger the site and the more 301s in place, the less likely the engines are to handle it successfully.
Thanks BC, for commenting
Matt – so in a case like this where an entire domain is missing from the Microsoft search engine, what can you do?
Actually I did decide to try sticking the path to the site map in the robots.txt file. That was something that had not been done on this domain so I figured I’d give it a shot
Ha – it actually did help cut down the spam comments to add the zeros!
I wonder if it’s dependent on how the redirects are implemented. For example:
redirect 301 / http://www.domain.com
as opposed to a full mod_rewrite?
Mike – Interesting… 302’s get followed but not 301’s? huh.
Utah – I suppose that’s possible, but they should follow the standard that others do, wouldn’t you think?
On THIS domain, (pdxtc.com) it appears as if the permalink change of a couple of weeks ago didn’t stump MSFT so –
I just edited the robots.txt on the domain which is the subject of this post to point to the sitemap – we’ll see if it now indexes the pages. – tick – tock – waiting.
MSN seems to have found out what 301s are. At least for my website 😉
oh yeah? hmm – nope, not seeing it yet on my end – thanks though, I’ll keep an eye out.
If Google and Yahoo got it right who cares.
Just a million unique visitors monthly, multiplied by even a potential 1 percent MS Live referrals is still a fair number of potentially lost visitors.
However, it appears that MS got the URL’s now, and the cached 301 Apache image is gone. cool 😉 Every little bit helps…
That’s a fair amount of traffic to consider. Glad it worked out for you.
MSN seems to have found out what 399s are. At least for my website 😉
Wow and i have been using 301 redirects for the longest