Search Engine Optimization for Site Owners and Small Businesses

Just over a year ago I posted on the dangers of hidden text and concluded with the advice "…don’t use hidden text to try to improve your rankings".

Here is a practical example of what may happen if you do.

Yesterday John Frost who runs the very popular Disney Blog posted that his blog had been delisted from the Google index and sure enough it had:

Such is the power of popular blogs that within a couple of hours of John’s plea for help their was an explanation and a resolution from none other than Google Engineer and spam fighter in chief, Matt Cutts. He explains in a diplomatic and friendly comment that hidden text was responsible for the ban. Specifically this page code:

<h2 id="banner-description">Informing Disney Fans the World Over with the latest news and updates from all Disney companies, divisions, and related stories. Disney World, Disneyland, Disney Cruises, Disney Animation, Pixar, ESPN, and more are covered in as much detail as I can muster.</h2>

With this in the external CSS file:

#banner-description
{
overflow: hidden;
width: 0;
height: 0;
margin: 0;
padding: 0;
text-indent: -1000em;
}

As it happens this appears to be a generic Typepad problem in that when you set up a Typepad blog you are asked to enter a Weblog description which ends up being hidden by the CSS. However after Matt had pointed it out and John had removed the text, Matt helpfully submitted a reinclusion request.

Matt has gone off to talk to Six Apart the Typepad developers and The Disney Blog will be back in the index sometime next week.

The moral of the story is still the same - don’t use hidden text to try to improve your rankings.

“The physician can bury his mistakes but the architect can only advise his client to plant vines”. Frank Lloyd Wright (1869 - 1959), New York Times, October 4, 1953.

If Frank Lloyd Wright were alive today I wonder what he would say about web designers’ mistakes. I get to see thousands of prospective clients and their competitors’ websites over the course of a year and although web design is improving I am still left thinking that 95% of web designers and web design firms just don’t understand the basics.

I have had to become an expert in diplomacy while explaining to prospective clients that the website for which they have paid hard earned money is (to put it politely) not as good as it might have been.

There seem to be five web design and build failures that come up again and again that require discussion with website owners. I rarely if ever get to talk through these points with the designers so I have listed them here as questions.

If you are thinking of having a new site or revamping your existing site you may want to make sure that these questions will be unnecessary before you appoint someone to carry out the work.

Here are the five questions for web designers:

1. Why don’t you learn what goes in the HEAD element?

Just because your client is unlikely to peruse the HEAD element doesn’t mean you should ignore it or fill it with garbage.

2. What’s so difficult about producing search engine friendly urls?

Dynamically generated urls can cause problems for search engine crawlers and may be ignored. Why not generate search engine friendly, human readable urls instead?

3. Why large logos?

Logos that take up 25% of the home page are a waste of valuable real estate. Users want to see what they came for not pictures of models staring up at the camera.

4. Do you leave blank alt tags for a reason?

Alt tags really do have a purpose. They are for the many users who use talking browsers, screen readers, text browsers or browsers on small devices.

5. Why don’t you use web standards like W3C?

Did you know that separating structure from presentation makes it easy for alternative browsing devices and screen readers to interpret the content? Or that using semantic and structured HTML makes for simpler development and easier maintenance? Or that less HTML means smaller file sizes and quicker downloads? Or that a semantically marked up document is easily adapted to alternative browsing devices and print? Or that if you use standards and write valid code you reduce the risk of future web browsers not being able to understand the code you have written?

“If we value the pursuit of knowledge we must be free to follow wherever that search may lead us”. Adlai E. Stevenson II from a speech at the University of Wisconsin, Madison, October 8, 1952.

This is a compendium of our experiments and the experiments of others to determine how the major search engines currently treat the rel=”nofollow” attribute.

A few months ago I placed a rel=”nofollow” on one of the existing test pages that we had used in the past to determine the search engine indexing behavior of keywords in urls. The link was placed to a new test page with style=”text-decoration:none” to reduce the possibility of someone clicking it and signaling the existence of the new ‘linked to’ page as a referrer. Here is a partial screen shot of the page in Firefox using the SearchStatus extension which highlights rel=”nofollow” links. There are no other links to the new test page.

Google, Yahoo and MSN are now showing a recent cache of the page and we can see how they handled the link.

We know that Google and Yahoo follow rel=”nofollow” links in the sense that they will visit the ‘linked to’ page. Valentin Agachi reported this in detail some time ago in his post Does rel=nofollow work? So for our own experiment and starting with the simplest behavior first:

MSN appears not to have spidered and certainly has not indexed the ‘linked to’ page:

Yahoo has spidered and indexed the ‘linked to’ page:

Yahoo also shows the page in the serps at 14/64 for an exact search on the anchor text.

Google has spidered but not indexed the page:

Mark Barrera in his post “nofollow” - Does it Really Work Like Google Claims? has shown that if the ‘linked to’ page is in the index already then Google will rank the page for the anchor text. Google will also acknowledge the link on the cached page with “These terms only appear in links pointing to this page”.

Here is a summary of all our findings:

What we can’t know for sure is if the search engines are completely disregarding the rel=”nofollow” in their algorithms. Google says in the Official Google Blog “When Google sees the attribute (rel=”nofollow”) on hyperlinks, those links won’t get any credit when we rank websites in our search results”. MSN appears to disregard rel=”nofollow” links in every aspect and Yahoo seems to treat rel=”nofollow” links the same way as any other link but they are probably disregarding them for ranking purposes.

I have been asked several times in the last few weeks, are paid directories worth the money? In my experience there are only two essential paid directories but depending on your product/service there may be more in the form of niche directories.

• Yahoo!

  The Yahoo directory has a $299.00 non-refundable, recurring annual fee per submission, and is the only paid directory specifically suggested by Google.

  Find the appropriate category for your site in the Yahoo Directory (normally where your competitors are) and click on the ‘Suggest a Site’ link on that page. You will need a Yahoo! ID to complete the submission but you can sign up during the process. After ticking various declarations and agreeing to the terms of service you will be taken to the submission form itself. Fill in the form following the guidelines for completion making sure that your site description does not exceed 150 characters.

• Business.com

  Business.com directory has a $199 fee for the first year which is renewable thereafter at $149. Included with the site listing are up to four additional links which allow you to list deep site content. Annual directory inclusion and standard listing account set-up can be found here.

• Niche Directories

  There are three basic ways to find relevant niche directories for your site.

  1. Look in the Business Directories section of DMOZ.
  2. Enter your competitors sites into Yahoo Site Explorer and look to see if they have links from any niche directories.
  3. Search with Google for your industry keywords plus the word directory
     e.g |”lawn care” directory|.

One last suggestion if you find that a large number of your competitors are using a paid general directory other than Yahoo or Business.com then you may wish to consider joining them.

Are you validating your site’s code and checking your site for broken links? You should be and until now you have probably used the W3C Markup Validation Service and something like Xenu’s Link Sleuth.

Now there is a new online tool that performs batch (X)HTML validation, link checking and more. It is called Nikita The Spider and is the brainchild of Philip Semanchuk. Philip found himself managing a growing number of Web pages and wanted to keep those pages valid and low on link rot without having to check each individual page and link. He couldn’t find a tool that would do that and since he is a programmer decided to build one himself.

Currently it is in Alpha test so sometimes you may have to wait to use it but register your interest by sending Philip an email (address on Nikita’s home page) and you can be one of the first to give it a try. (At the time of writing you can start a crawl without waiting).

The link checker will find broken internal and external hyperlinks and references to missing documents such as a missing PDF or MP3 file and uses the same validation parser engine as the W3C Validator. You get well organized HTML reports that summarize what Nikita has found on each page of your site and if you want to analyze your site in a way that these reports don’t you can have XML versions and reorganize the data yourself.

Here are a couple of screen shots:

Nikita can spider your whole site or just a portion of it and it has an online interface for ad hoc queries. For example you can show a list of pages that are delivered as text/html, pages that are larger than 100k or pages that use a certain doctype.

There are other features too. It allows you to control the speed at which your site is spidered and you can define a custom user agent so that you can filter Nikita’s visits from your Web logs. Also you receive statistics about your site such as a list of the doctypes you use, average page size and URL length.

Philip is very open to suggestions as to future features and even has a page where you can vote for the ones you like. After beta obviously power users will have to pay however he intends to always offer a free version with a limit to the number of pages per crawl.

This tool is from the top drawer and has a very bright future indeed.

Philip has also written an excellent article about protecting email addresses from spam with the results of a most interesting experiment.