Search Engine Optimization for Site Owners and Small Businesses

Are you validating your site’s code and checking your site for broken links? You should be and until now you have probably used the W3C Markup Validation Service and something like Xenu’s Link Sleuth.

Now there is a new online tool that performs batch (X)HTML validation, link checking and more. It is called Nikita The Spider and is the brainchild of Philip Semanchuk. Philip found himself managing a growing number of Web pages and wanted to keep those pages valid and low on link rot without having to check each individual page and link. He couldn’t find a tool that would do that and since he is a programmer decided to build one himself.

Currently it is in Alpha test so sometimes you may have to wait to use it but register your interest by sending Philip an email (address on Nikita’s home page) and you can be one of the first to give it a try. (At the time of writing you can start a crawl without waiting).

The link checker will find broken internal and external hyperlinks and references to missing documents such as a missing PDF or MP3 file and uses the same validation parser engine as the W3C Validator. You get well organized HTML reports that summarize what Nikita has found on each page of your site and if you want to analyze your site in a way that these reports don’t you can have XML versions and reorganize the data yourself.

Here are a couple of screen shots:

Nikita can spider your whole site or just a portion of it and it has an online interface for ad hoc queries. For example you can show a list of pages that are delivered as text/html, pages that are larger than 100k or pages that use a certain doctype.

There are other features too. It allows you to control the speed at which your site is spidered and you can define a custom user agent so that you can filter Nikita’s visits from your Web logs. Also you receive statistics about your site such as a list of the doctypes you use, average page size and URL length.

Philip is very open to suggestions as to future features and even has a page where you can vote for the ones you like. After beta obviously power users will have to pay however he intends to always offer a free version with a limit to the number of pages per crawl.

This tool is from the top drawer and has a very bright future indeed.

Philip has also written an excellent article about protecting email addresses from spam with the results of a most interesting experiment.

Following on from the recent post SERPs Position and Clickthroughs and after looking more closely at the AOL research data I thought it would be interesting to provide a tool to calculate the percentage increase or decrease in clickthroughs that can be expected in moving from one position in the SERPs to another.

The AOL data contains 36,389,567 search queries with 19,434,540 clickthroughs. Obviously not every search will result in a click through but the reason why the overall rate is so low is because of the way AOL displays the results. AOL’s organic results, which are essentially Google’s, are top and tailed by sponsored listings. So many of them in fact that only 54% of AOL search queries are followed by a click on the organic listings. (See addendum below).

However the clickthroughs recorded in the AOL data can be considered as a proxy for the Google SERPs and you can use this tool to see by how much your traffic will increase or decrease when you move from one position to another.

Examples:

1. Your current position in the Google SERPs is 15 and you want to know how much more traffic you would get if you moved to position 5.

Current Position in SERPs 15
New Position in SERPs 5
Click ‘Calculate’
Percentage Increase/Decrease in Clickthroughs 936.8%
Your Google traffic will increase by 936.8%

2. Your current position in the Google SERPs is 8 and you want to know how much more traffic you would lose if you moved to position 11.

Current Position in SERPs 8
New Position in SERPs 11
Click ‘Calculate’
Percentage Increase/Decrease in Clickthroughs -78%
Your Google traffic will decrease by 78%

3. Your current position in the Google SERPs is 9 and you want to know how much more traffic you would lose if you moved to position 10.

Current Position in SERPs 9
New Position in SERPs 10
Click ‘Calculate’
Percentage Increase/Decrease in Clickthroughs 5.1%
Yes it’s true, your Google traffic will increase by 5.1% when you go down one from position 9 to 10! This is because the ninth position gets less clickthroughs than the tenth position, which you can see from the charts here.

Thanks to Richard Hearne at Red Cardinal for providing the data on which this tool is based.

Addendum June 19, 2007

In a recent research study Different Engines, Different Results by Dogpile.com, in collaboration with researchers from Queensland University of Technology and the Pennsylvania State University, it is noted that “….a separate study conducted in conjunction with comScore Media Metrix found that between 54 – 62 percent of all searches on the top four search engines are converted to a click on the first result page”. This of course means that 38 - 46 percent aren’t!

A previous post in the Tutorial on Reciprocal Links gives examples of good and bad reciprocal links. It concludes by saying “that reciprocating a link will not automatically make a good link bad or indeed make a bad link good, what is important are the individual links themselves”. This post emphasizes the dangers of reciprocal links.

Google has deployed a new infrastructure and software upgrade of their crawling and indexing. Codenamed Bigdaddy it started to roll out in February and was fully deployed by the end of March when the old system was turned off. Matt Cutts has been addressing some of the anomalies that SEOs have noticed in a long post on his blog.

He mentions reciprocal links four times in the post and here are the relevant paragraphs excerpted:

1. The sites that fit “no pages in Bigdaddy” criteria were sites where our algorithms had very low trust in the inlinks or the outlinks of that site. Examples that might cause that include excessive reciprocal links, linking to spammy neighborhoods on the web, or link buying/selling.

2. As these indexing changes have rolled out, we’ve improving how we handle reciprocal link exchanges and link buying/selling.

3. I think this is covered by the same guidance as above; if you were getting crawled more before and you’re trading a bunch of reciprocal links, don’t be surprised if the new crawler has different crawl priorities and doesn’t crawl as much.

4. Some folks that were doing a lot of reciprocal links might see less crawling.

The message could not be clearer, don’t use reciprocal link exchanges as the basis of your link building campaign.

December 15, 2006 As if to underline what is said above the Official Google Webmaster Central Blog has a post in which the following is written: “To sum up, even though improved algorithms have promoted a transition away from paid or exchanged links towards earned organic links, there still seems to be some confusion within the market about what the most effective link strategy is”. You can read the complete post at Building link-based popularity.

Also here is a quote the previous month from Adam Lasnik, Search Evangelist at Google, in Google Groups: “…reciprocal links have been around forever, and Google doesn’t frown on engaging in reciprocal linking in moderation. The key here is, indeed, moderation :). If, say, 90% of your backlinks are reciprocal, that’s probably not going to improve how our algorithms view your site. Or worse, if 90% of your backlinks are reciprocal and not likely to be of interest to your user. But exchanging links here and there — *especially* when done with clear editorial judgement (e.g., you’re not just accepting dozens of link exchanges willy-nilly) — that’s not the sort of thing Google looks down upon”

19 December 2007

Matt Cutts in a Pubcon question and answer session said “Trading links is a natural thing around the Web. Natural reciprocal links do happen but if 50 percent of your links are coming from link exchanges it begins to looks like you’re trying to build an artificial reputation and bump up the amount of links you don’t have”.

Information architecture is simply the practice of structuring information and is most often put to use when designing large websites. It is a term used to describe the organization of a web site and includes aspects such as navigation, functionality, content, information, and usability.

Designers of large and complex websites are particularly concerned with information architecture and the formal processes that it involves. In fact most of the larger web design companies have established information architecture departments. However small business website owners/designers can also benefit from some elements of the process and here is an illustrative example.
Read the rest of this entry »

For years usability studies and server log file analysis have tended to indicate that home page web designers have just a few seconds to create a favorable initial impression on the user. New evidence contained in a soon to be published paper suggests that a few seconds may be a gross over estimate.

Gitte Lindgaard and her colleagues from the Human Oriented Technology Lab (HOTLab) at Carleton University have conducted studies to ascertain how quickly people form an opinion about webpage visual appeal. The paper is to be published in the March-April 2006 issue of Behaviour & Information Technology.

Three studies were conducted in which subjects were presented with brief glimpses of previously classified home pages and asked to rate them for visual appeal. The results were highly correlated with assessments made over much longer periods of time and indicated that visual appeal can be assessed within 50 milliseconds. This is an astonishingly short period of time given that a normal human blink lasts 200–300 milliseconds.

Gitte Lindgaard and her colleagues have given the paper a rather apposite title “Attention web designers: You have 50 milliseconds to make a good first impression!”

In practice this means fast downloading home pages, limiting the graphics and providing information in the simplest way possible. If you are explaining, you are losing.