Web robots – often referred to as crawlers, bots, or spiders – are software programs that constantly travel the web, indexing the information found on millions and millions of websites every single day. Some sites, however, don’t wish to be indexed in search engines or accessed by these Web Robots. Now that you know what a Web Robot is and what it does, it’s important you know what can be done to limit their access to your site if you so desire. There may be a number of reasons for wanting to prevent bot access to a website page or specific directory. The most common reasons are related to security, privacy and duplicate content.

The Robots Exclusion Protocol, more commonly referred to as a /robots.txt file, provides webmasters with the ability to provide instructions on indexing the site to bots. The file, which must reside in the domain’s root directory, serves to limit the bots’ access to files within that domain’s root directory. There are often a large number of pages that make up an entire site, but many of those pages – like registration, login, 404 error, privacy policy and order confirmation pages – should not be indexed by search engines. The /robots.txt file also comes in particularly handy for webmasters with a wide network of sites with identical privacy policies, terms and conditions or e-commerce sites that have checkout pages, shopping carts, etc.

Addressing Duplicate Content with /robots.txt

The /robots.txt file can also help to eliminate duplicate content issues that arise with blogging software, such as WordPress. With WordPress – and all blogging software, for that matter – content from blog posts is published on the post URL itself, but copies of that content are also published on category pages, as well as tag and author archives. This inadvertently creates several pages of duplicate content. Since duplicate content can have a negative impact on a site’s ranking in the organic search results, the /robots.txt file can help to reduce the potential for duplicate content that can adversely affect the site’s search marketing strategy.

Understanding How To Use /robots.txt

In order to function properly, the /robots.txt file should be accessible at http://www.domain.com/robots.txt and reside in the domain’s root directory. The file itself should be created as a plain text document. Do NOT use Microsoft Word or another word processing program – the standard Notepad program that is installed with Windows or SimpleText/TextEdit with the Mac OS work best. The file name must be robots.txt and uploaded directly to the domain’s root directory. The commands within the file itself can be as simple or complex as your needs demand.

The standard, generic /robots.txt file – one that does not limit access to any of the information in your domain’s root directory – would be formatted like this:

User-agent: *
Disallow:

In order to block bot access to the domain’s root directory completely requires adding only one character to the standard or generic /robots.txt file and would look like this:

User-agent: *
Disallow: /

What if you want to limit bot access only to certain subdirectories or specific pages of the site? Not a problem. You would simply add each individual subdirectory or URL to the /robots.txt file as follows:

User-agent: *
Disallow: /checkout.asp
Disallow: /add_cart.asp
Disallow: /view_cart.asp
Disallow: /error.asp
Disallow: /shipquote.asp

The Robots.txt File Is Not Fool Proof

While the /robots.txt file does a good job of blocking a bot’s access to the domain’s root directory, it isn’t fool proof. Each individual page you do not want bots to index should also incorporate a properly formatted robots META tag. The standard robots META tag is configured like this:

<meta name=”robots” content=”index,follow” />

To help to prevent the bots from accessing individual URLs, the robots META tag in the header of the page should look like this:

<meta name=”robots” content=”noindex,nofollow” />

or

<meta name=”robots” content=”noindex,follow” />

The Bottom Line

A /robots.txt is a very useful tool and, unfortunately, an often overlooked and neglected aspect of web development. Now that you have a better understanding of what it is, what it does and how to use it, take some time to consider how your site may benefit from having a properly configured /robots.txt file. In the meantime, start checking out the /robots.txt files of the sites you visit to familiarize yourself with different configurations and uses for it.

In some instances a site will gain so many natural links that little else matters in the grand scheme of ranking.  Let’s take a look at a handful of summer blockbuster movie sites to see what kind of attention, if any, was paid to their on-site SEO efforts.  And, more importantly, a close look at where they rank and if a lack of attention to on-site SEO matters much in the world of blockbuster movie sites.

Star Trek

One of the most highly anticipated blockbuster movies of 2009 is certainly Star Trek.  Following the same school of thought as the always popular Star Wars series, the latest movie in the Star Trek saga rewinds to tell the story of the core characters in the original series before they joined the crew of the Enterprise.  This installment serves as the prequel to the original series.

As you would expect, the SERPs for a recent movie is a perfect example of the universal search Google is committed to providing for its users.  A general search for the term “Star Trek” generates a whopping 105 million results ranging from news results that aren’t necessarily applicable…like the three in this case, which clearly mention Star Trek, yet I feel far short of being particularly devoted to Star Trek.

Immediately following the Google News Results we find the Star Trek official website, dedicated to all things Star Trek.  The site not only promotes movies, like the 05/08 release of this film, but also serves the larger Trekker/Trekkie community – devoting sections of the site to the original series, the existing movies, The Next Generation or any other evolution of the original Gene Roddenberry creation.

After the official Star Trek sites come a couple of results from IMDB (Internet Movie Database), a trusted source of information about movies, television shows, actors, etc.  The next results are an integral part of Google’s universal search offering:  videos.  Not surprisingly, all three of the video results shown within the SERP for “Star Trek” link to trailers for the recent movie release.

And immediately following the videos, we finally stumble on to the official movie site, startrekmovie.com.  The Star Trek site is built entirely in Flash, which considering the visual demands of the techie-types that gravitate toward Sci-Fi flicks like Star Trek, shouldn’t come as any surprise.  It does make me wonder, though, if the site wouldn’t rank above the fold if it incorporated content the search engines could easily index coupled with Flash elements that appeal to more demanding users, as well.

Results for “Star Trek movie” and “Star Trek trailer” are much more promising for the summer blockbuster, with the official site listing well above the fold in both searches.  Remember, the more specific a search term the less competition there tends to be.  This is a perfect example, with over 150 million results for “Star Trek”, but just over 30 million for “Star Trek movie”.

TRANSFORMERS

If the previews for this movie are any indication, TRANSFORMERS is sure to be the pick of the litter of action films this summer.  While the movie isn’t scheduled to be released until June 24^th, the official movie site for “TRANSFORMERS:  Revenge of the Fallen” – www.transformersmovie.com – already ranks in the top position in Google for the general term “Transformers”.

It’s no surprise that the entire “TRANSFORMERS” site is dynamic, with very little content that can be indexed by the search engines.  Like the “Star Trek” movie site, that means the search engines must rely very heavily on the anchor text of links from outside sources to determine what terms are most applicable to the site.  Fortunately for summer blockbuster movies, receiving a great deal of attention from traditional media and trusted online media sources is worked into their marketing budgets.

Land Of The Lost

With a title like “Land of the Lost”, I wouldn’t expect this site to have a lot of trouble ranking well.  Like TRANSFORMERS, the movie has yet to be released and the public hasn’t had the chance to see if the film will live up to the hype.  That said, much like Star Trek, this movie has a bit of competition in the search results – itself.  A science fiction television series by the same name made its debut in 1974.  The IMDB result for the original series is, however, outranked by the movie’s official site:  www.landofthelost.net , which appears atop the SERPs for “Land of the Lost”.

Up

Perhaps the most surprising result in the SERPs comes for the uber-generic term “Up”.  I expected to at least have to include “movie” or “Disney” with the search in order find relative results.  I was wrong.  Upon inputting “Up” into the query, I was immediately greeted by the Disney Pixar summer film.  In fact, 4 out of the first 5 results were for the movie, Up.

Like the previous movie mentions, this site is also entirely done in Flash.  Further illustrating the impact and overall importance of a high number of incoming links from a wide variety of sources ranging from media websites to enthusiastic bloggers.  Rankings are a bit of a popularity contest and a link to a site plays very much like a vote for most popular.  The cache that comes with being a Disney film put Up at a link building advantage from the start.

Less Buzzworthy Movie Mentions

A few honorable mentions include:  Night at the Museum:  Battle of the Smithsonian, whose official movie site ranks 7^th, behind IMDB results, video results and Google news results for “Night at the Museum”.  Ice Age:  Dawn of the Dinosaurs ranks 1^st for “Ice Age”.

An interesting thing to note is that the soon to be released The Taking of Pelham 123 has an official site at catchthetrain.com, yet that URL appears NO WHERE within the search results for the term “Pelham 123″.  This certainly could serve as an argument in favor of including search terms in the URL, but with the complete absence of catchthetrain.com from the search results, I’d say there’s probably a bit more to it than that.

A more obvious point of difference between a site for a movie like TRANSFORMERS vs. The Taking of Pelham 123 lies in the number of incoming links.  TRANSFORMERS has over 100,000 links pointing to it, while The Taking of Pelham 123 has under 8,000 links.  That’s a huge discrepancy.

Playing By The Rules

Links are still Google’s currency of choice, as related to organic rankings.  That’s  no less true of a movie studio’s website than a local small business owner’s.  The most efficient way to improve a site’s ranking is to address on-site and off-site SEO issues.  If you’re a movie studio with a huge marketing budget, on the other hand, you can ignore on-site SEO entirely and still come out at or near the top of the search results because of the large number of links the site will acquire naturally.

Credit: BigStockPhoto.com

Believe it or not, new people join the search marketing world every day.  Frankly, that scares me a little bit.  It’s not the competition or the potential loss of business that scares me, but the sources

people are relying on to give them the “right” information.  That said, here are the top 6 most classic SEO articles that I suggest anyone new to the industry read.  And then read again.  Some things in SEO/SEM world haven’t changed much…and a refresher is always a good idea.

“Successful Site In 12 Months With Google Alone“

Written by Brett Tabke at WebmasterWorld over 7 years ago, the fundamental steps for getting started down the road toward success in Google are covered in this archived post.  While I don’t agree 100% with everything included here – like the idea of dismissing the use of keywords in the domain and the necessity of submitting the site to search engines manually – I love the emphasis that Brett puts on creating quality content regularly.

Several times within the post Brett mentions content, creating logical directories for content, keeping new content flowing, etc.  That is one thing that certainly hasn’t changed.  Content is still king and the sites the provide the most relevant and useful content always seem to work their way to the top of the SERPs.

The search engines may now be looking for more kinds of content (i.e. video) than back in 2002, it’s important to recognize that serving up the content that will be most helpful to users is still the goal of search engines.  Convincing the search engines that your content will be most helpful to search users is yours.

“SEO Is Easy? Let’s Look At The Hard 5 Percent“

A more recent post this time, by Todd Friesen, published in the 100% Organic section of Search Engine Land which helps to draw the line in the sand between some aspects of SEO that many believe can be done by anyone and the vastly important last 5% that if done incorrectly can have far-reaching and long-term consequences that can spell catastrophe for a site.

Todd highlights the following aspects of SEO as “The 5 Percent”:

• URL Structure (the fixing thereof)
• Advanced Linking
• Appropriate Cloaking (err…IP Delivery)
• Dynamic Template Modification
• Client Management (agency life)
• Redirection (site moves or redesign)

Particular emphasis is placed on the importance of managing clients’ expectations and having clear lines of communication.  He notes the particular challenges faced by both parties when the client feels they’ve received lackluster results, while the agency suffers the frustration of their changes not being implemented.

“Filthy Linking Rich And Getting Richer!“

Written by Mike Grehan back in October 2004, this post still holds true today.  With statements like:

“…And I’m afraid some of it may be pretty bleak reading if you have newly created, lowly indexed web pages and you’re desperately waiting for someone to link to them so that they stand a chance of ranking in a search engine with a static link based algorithm…”

And:

“…In fact, it has been proposed that “the rich get richer” effect drives the evolution of real networks. If one node has twice as many links as another node, then it is precisely twice as likely to receive a new link…”

Mike Grehan finds some important correlations between science, psychology and the underlying reasons that some sites build links more easily than others.  This post takes a strikingly scientific turn with references to Hungarian mathematician and genius, Paul Erdos and a study by sociologists Fredrick Liljeros and Christopher Edling of Stockholm University, the post itself possesses a great deal of insight into why the idea of “the rich get richer” does not just apply to power and wealth in a traditional sense, but in the virtual world of online currency – links.

This may be one of the more challenging posts you read this week, but it’s well worth the effort.

“101 Ways to Build Link Popularity“

Aaron Wall and Andy Hagans joined forces for this gem of a post highlighting the importance of link building back in 2006.  As illustrated by this quote, not that much has changed in the world of link building since then:

“Link Building… Time-intensive. Frustrating. Sometimes confusing. Yet Unavoidable. Because ultimately, it’s still the trump card for higher rankings…”

Obtaining high quality links from trusted sources is no easier today than back in 2006, and yet – no less important either.  While social networking and social voting sites like Digg and StumbleUpon have provided opportunities to expose content to a much larger audience, demands on the quality and type of content have increased significantly, as well.

“Spam Guide for Raters“

While I hesitate to recommend that anyone follow a SPAM guide, this article at Henk van Ess’s Search Bistro will help anyone identify SPAM techniques, regardless of intent to either abuse or report them.  The blog may not have been updated since late 2007, but this guide is as useful today for identifying spam-o-rific tactics – or, more constructively – understanding what is considered SPAM, than ever.

Topics covered include:

• Sneaky Redirects
• Hidden Text / Hidden Links
• Thin Affiliate Doorway Pages

“Hilltop in Plain English“

Jason Duke addressed the infamous “Hilltop” update carried out by Google in 2004 that seemed to throw a monkey wrench into the Link Olympics going on at that time.

The essence of “Hilltop” was a significant shift in attention from merely quantity of links to a site, to more quality links to a site from sites that are topically related or considered more authoritative.

Oh, How Little Has Changed…

What do all of these posts have in common?  They are all well beyond the time period in which they should be relevant and useful in the world of SEO.  Yet, they’re not.  They all include very important, basic and fundamental information that still applies as much today as it did when they were originally written.

There are several other posts that deserve a mention here, and probably hundreds of others I just don’t have room to mention.  Don’t discount a post’s relevance based solely on its age.

• “Tips for Attaining One Way Inbound Links“
• “SEO Friendly Affiliate Systems“
• “SEO 101 – The Timeless and Classic Hits“

Read them, test the theories when & where you can and determine for yourself what still applies and what doesn’t.  You’ll find that shockingly little has changed from an on-site SEO perspective and that links still rule the world.  What’s changed is where the links come from and understanding how to appeal to those that may link naturally to your site.

While there are plenty of paid tools out there that can be used to tackle a variety of SEO tasks, there are also a number of free tools that are useful, as well.  Many of the free tools have been developed by some of the best SEO minds in the industry and each has its own strengths and advantages.

Let’s start with some basics…these tools assess very basic aspects of on-site SEO and don’t require a lot of explanation. That said, the basics are still important and using the right tools to get a quick overview of a site’s on-site SEO is important.

Meta Tag Analyzer from SEOCentro – quick snapshot of a page’s META tag lengths, relevancy to page content, anchor text of links and other useful basics.

Keyword Density Analysis Tool from We Build Pages – while the importance of keyword density is hotly contested in SEO circles, it’s still a good idea to know the keyword density is of a page…if for no other reason than to know if you’re being “spammy”.

SEO Browser – see your site like the search engines see it…no layout, no graphics, just the down and dirty content that they can index.  Very useful.

Website Grader from HubSpot – a quick and very high level overview of a site’s overall search engine friendliness. It doesn’t provide much actionable information for SEO industry insiders, but for site owners it will provide a great snapshot and provide them with some valuable information regarding what goes into a search engine friendly website.

Now, on to some bigger and better tools…

Yahoo! Site Explorer

This is the tool used most often to research a site’s inlinks. The report is easy to run and easy to understand. In order for the report to accurately reflect the number of links back to site from outside sources, you must configure the result by choosing “Except from this domain” and “Only This URL” from the drop down menus.

Rank Checker from SEOBook.com

There are many instances when you want to know where a site and individual pages of a site rank for a long list of terms. Searching manually would take more time than you’re willing or able to spend. Thanks to the genius of Aaron Wall of SEO Book, all you need to arm yourself with is a Firefox plugin.

Aaron has created a video introduction to his Rank Checker tool – watch it. As I said, this is a Firefox plugin – therefore you must be using Mozilla Firefox in order to use it. If you don’t have Firefox, go download it.

SEO Toolbar from SEOBook.com

It seems as though I have some sort of connection to or bias in favor of SEO Book and Aaron Wall, but the fact is that he just builds some great tools – tools that I can’t imagine being without in carrying out my day-to-day tasks assessing and optimizing websites.

It is wildly useful and kills a hundred birds with one stone – pages indexed, domain age, Compete uniques, Diggs, Stumbles, you name it…if you’re going to try out only ONE TOOL as a result of reading this post, this is THE ONE. Aaron has also created a video introduction to the SEO Toolbar – watch it, too. Again, this is a Firefox plugin, so you will need to install Mozilla Firefox in order to use it.

Google AdWords: Keyword Tool

Why not get the information straight from the horse’s mouth whenever you can, right? This free keyword tool from Google gives you two keyword research options to choose from: 1) research by specific term; or 2) input your site’s URL and the tool will generate terms for you based on your site’s content. There are also a variety of filters available to help you fine tune the results provided by the tool.

SEO Book Keyword Suggestion Tool

If I had to choose just one keyword suggestion tool, this one would be it.  One reason I love it is because it separates the results for each term by search engine.  The SEOBook tool also provides data for Google Trends, Google Traffic Estimator, Yahoo! Suggest, Google Suggest and more. A great tool…almost hard to believe it’s free.

This is great information, especially for niche sites, because some have a wider audience of Google users, while others have a greater audience of Yahoo! users.  Since each search engine develops its own algorithm, the criteria for ranking well in each search engine is different. If your target audience is made up of primarily Yahoo! search users and you optimize primarily for Google, you’re hurting yourself.

Other Invaluable Firefox Plugins

Web Developer Toolbar – this will let you do everything from turning off images to disabling Java & cookies to validating CSS & HTML directly from the browser window.

SEO For Firefox – yep, you guessed it…another tool from SEO Book. One of this tool’s most useful functions, in my opinion, is that it allows you to highlight the NOFOLLOW links on a website page. And yes, there’s a video introduction for this SEO Book tool, too. Aaron covers all his bases.

RankQuest Toolbar – like the other toolbars, this one provides a wide variety of useful information, including an on-demand keyword density analyzer and “show META tags” function, plus much more.

One Last Standalone Tool

Xenu’s Link Sleuth – broken links are bad. I haven’t found a tool yet that provides more thorough or useful information regarding a site’s broken links than this tool. Not only does it report the existence of a broken link, but it also reports the pages on which the broken links are found – very useful in resolving the issue.

There is no question that the nature of the Information Age has changed the face of marketing forever.  Businesses big and small finally understand that having a website isn’t an option – it is necessity.  There isn’t a potentially successful business plan created today that doesn’t include a budget for building a website.  The vast majority of business owners have also begun to understand that simply having a website isn’t enough and that a sound Internet marketing plan is just as important as the existence of the website itself.

The landscape of Internet marketing, and search marketing in particular, is changing.  These days just optimizing your company’s website to perform well in search results isn’t sufficient.  Repetition is the key to success in the search results these days in much the say way it is in traditional marketing.  With respect to the search results, that means having multiple results on the page that correlate to your brand.  For instance, a search result for “athletic shoes” that includes multiple references to Nike is a search marketing success.

The way to accomplish that is by publishing content that references your brand or what you offer (i.e. the keywords used to find your products and/or services) on multiple sources, rather than relying solely on creating content for your website alone.  The fact is that the more sources upon which you create and publish unique content, the greater chance you have of monopolizing multiple results in the SERPs for your target keywords and successfully building your brand online.

Squidoo – Search Engine Friendly Content & 400,000+ Visitors Per Day

Perhaps one of the most well-known user-generated content sites is Squidoo.  Founded in 2005 by Seth Godin, best selling author and blogger, Squidoo boasts nearly 1 million user-generated “lenses” – the term Squidoo uses to reference to its user’s pages – and is among the 300 most popular sites in the United States.

With that popularity comes an enormous opportunity to connect with Squidoo users via your own personal lenses.  Not only that, Squidoo touts a search engine friendly format that increases the chances that someone doing a keyword search in search engines like Google and Yahoo! will find your lens in their search results.

Unlike many user-generated content and Social Media sites that use “NOFOLLOW” links, the links within your Squidoo lenses are “FOLLOW” links.  That means linking to pages of your website in a Squidoo lens will help to build link popularity.  If you’re going to create content for a user-generated site, getting the bonus of some added link popularity for your own site certainly helps to kill two birds with one stone.

eHow – 2 Million Pages & 16 Million Unique Visitors Monthly

The nature of eHow is a bit different than Squidoo, in that the entire site is essentially dedicated to “how-to” and tutorial type content.  That said, eHow is the most highly trafficked user-generated content site we will mention here.  In order to appeal to the eHow audience, it is important that your content follow a “how-to” approach.

For instance, if you sell headphones, creating an article for eHow entitled “Choosing The Best Headphones”.  If you sell cars, writing an article entitled “How To Improve Your Gas Mileage” would be applicable both to your industry and the format of eHow.  Unlike Squidoo, eHow does not provide users with the added bonus of “FOLLOW” links.  Any links you include within your articles will provide an avenue for users to find your website more easily, but it will not have a positive impact on your link building efforts.

HubPages – Search Friendly Infrastructure & Revenue Generation For Contributors

It’s important to note that the vast majority of user-generated content sites, including Squidoo and eHow, provide revenue generation opportunities for their contributors, but HubPages was the first to begin paying its contributors up front for articles they believed to be of particularly high quality – rather than paying them based on ads clicked by visitors to the articles alone.  They elected to do so in part because of the increase in exposure such well-crafted articles would bring to HubPages in the search results.

Why is HubPages so focused on quality articles?  It turns out that quality vs. quantity is the very motto of HubPages.  HubPages set out to overtake their closest competitor, Squidoo, in what – at the time – was considered someone radical and unconventional.  First, they removed all adult content from the site.  They also decided to force publishers who were using HubPages primarily to promote individual products and service to revise the articles to be more information in nature and reduce the excessive linking that had previously occurred.  Personal HubPages, duplicate content copied from other sources and pages that linked out to questionable sites (like known phishing sites, excessive pop-up ad sites and URLs that redirect were unpublished.

In the short term this reduced the overall traffic to HubPages enormously.  In the long run, though, the increase in the quality of the site content won out.  HubPages is now comparable, if not equal, to Squidoo in terms of traffic – despite the fact that HubPages continues to focus on quality information that does not include adult or promotional/marketing content that typically generates a lot of traffic.

Other User-Generated Content Sources

Squidoo, eHow and HubPages may be among the most popular user-generated content sites today, but they’re not the only games in town.  Other user-generated content sources to check out include:

Associated Content
Suite101.com
Bukisa
Today.com
Xomba

These user-generated content sites are among the most underutilized mediums for online reputation management and branding.  Sites like eHow, HubPages and Squidoo provide site owners with a unique opportunity to create content and publish it using their search engine friendly platforms.  Doing so not only gives you another information source that may be found in the search engines, but user-generated content sites also have an audience of millions of visitors on a monthly basis.

Remember – if you’re working to build a brand or maintain a company’s online reputation, the more sources you use to publish and promote content, the better.

Sitemaps vs. sitemaps – Yes, Virginia…there is a difference

Before getting into the details and purpose of XML Sitemaps, it’s important to understand the difference between a sitemap (lowercase) and a Sitemap (uppercase).  A sitemap is an HTML file, typically accessible to website visitors, which contains a link to each page of a site that visitors might find useful.  It is generally a single page composed only of text-based hyperlinks that can be used to find a specific page more easily – such as a “Contact Us” or specific product page – without being distracted by website design elements or page layout.  While search engines are likely to access an HTML sitemap during the course of crawling a site, this page is not constructed or intended for indexing purposes.

An XML Sitemap, on the other hand, is an XML file that is not intended for use by a website’s visitors, but by the search engines themselves.  The Sitemap itself is a UTF-8 encoded file that is created specifically to help search engines more efficiently index all of a website’s pages – particularly those that may not be linked to by outside sources or through the site’s text navigation itself.  Google follows the accepted Sitemap Protocol 0.9, as published by Sitemaps.org.  It’s also important to note that what we are discussing here are standard Sitemaps.  There are also Video, Mobile, News, Code Search and Geo Sitemaps that can be created for different types of content.

Is a Sitemap Necessary?

While the value of Sitemaps to SEO has been the subject of a great deal of discussion with opinions varying from “Sitemaps are essential” to “Sitemaps are useless”, the fact remains that an XML Sitemap is merely a file that provides search engines with information about a particular website and its pages.  While some may contend that Sitemaps are overrated, others take on a “maybe, maybe not” or “it can’t hurt” stand.

Google attempts to locate a Sitemap file at http://www.domain.com/sitemap.xml – however this Sitemap location can be changed within the Google Webmaster Tools control panel, should the Sitemap exist at an alternate URL.  In my opinion and experience, and particularly because Google searches for a Sitemap by default, there is no reason to not have one.  However, to answer the question, “is a Sitemap necessary?, the short answer is, “no – it is not necessary.”  Submitting a Sitemap to Google does not lead to higher rankings, nor is it a guarantee that a page will be indexed any more quickly than it would otherwise.

A Sitemap does, however, provide search engines with useful information that can help it crawl a site more efficiently, such as the last time the page was updated, how often the page typically changes and a specific page’s importance relative to the site’s other pages.  In an XML Sitemap, that information is displayed as follows:

<url>
<loc>http://www.domain.com/specific-page-address/</loc>
<lastmod>2009-02-19T17:51:04+00:00</lastmod>
<changefreq>always</changefreq>
<priority>0.5</priority>
</url>

Building a Sitemap

Some people shy away from creating and submitting a Sitemap because it requires a specific protocol that must be followed, as illustrated by Google’s Webmaster Tools Sitemap Protocol.  For some, having such specific requirements is intimidating and they prefer to not do it at all, rather than do it wrong.  If you’re that person, I have good news.  Thanks to some useful tools, creating a Sitemap file is easy.  In fact, there are a number of online tools that will do it for you – for free – provided that your site isn’t made up of tens of thousands of pages. 

• XML-Sitemaps.com – this tool will build a Sitemap for free for sites up to 500 pages.  If your site has more than 500 pages, they offer a standalone version of their software that will create a Sitemap with no limit on the number of pages for only $19.99 – and for an additional $10 they will even install the software for you.
• XMLSitemap.com – this site provides links to a number of free tools that will help you create XML Sitemaps that vary from web-based tools to PHP scripts.
• Google (XML) Sitemap Generator for WordPress – as is usually the case, WordPress users can rejoice!  There is a plugin that will create an XML Sitemap for you, and SO much more.  This plugin not only creates the XML file, it is highly configurable and can be set to include and exclude certain pages (such as categories, author and tag archives – which you may have a NOINDEX command on and wouldn’t want included in an XML Sitemap).  Not only that, it can be set to rebuild the XML file automatically upon any change to your WordPress site or blog, as well as notifying Google, Yahoo!, MSN and Ask.com automatically whenever the Sitemap is rebuilt.

Sitemap Generated – Now What?

Provided you’re not a WordPress user and don’t get to take advantage of the automation the above referenced plugin provides, once you’ve generated a Sitemap file in XML format, all that remains is to upload it to your domain’s root directory (if you don’t know how to go about doing this, you should probably leave this task to your Webmaster) and then submit it to Google.  Luckily, Google makes it pretty easy.  Here’s the step-by-step:

1. Log in to your Google Account
2. Follow the “Webmaster Tools” link
3. Click on the “Sitemaps” link to the left (if your Webmaster Tools account includes more than one domain, you will need to choose which domain first)
4. Input the file’s location – if the XML file is uploaded into the root directory, the file location will simply be sitemap.xml; if you placed the XML file in a folder, the file location will be folder/sitemap.xml

For more information regarding XML Sitemaps, visit Google’s About Sitemaps page and accompanying Sitemaps FAQs post at the Official Google Webmaster Blog.

As Twitter goes mainstream, more and more companies are jumping on the bandwagon and recognizing how useful it can be.  Twitter isn’t just for geeky computer types with nothing better to do with their time than talk to their friends 140 characters at a time.  Quite the opposite, really.  Twitter is quickly becoming a communication medium of choice between companies in a variety of industries and their customers.

The travel industry is no different.  With airlines like Virgin America and British Airways, as well as cruise lines like Carnival and Crystal Cruises already on board and using Twitter regularly to communicate everything from vacation specials to possible flight delays, the impact of Twitter can be felt throughout the travel industry.  While vacation specials and flight related information would be something you’d expect an airline to share via Twitter, the interaction doesn’t stop there.

Virgin America Really Gets It Right

Virgin America is using Twitter somewhat differently than other companies in the travel sector – they seem to be interested in using it to build a rapport with its target audience.  After all, isn’t that what Twitter is for?  What’s different about Virgin’s approach to Twitter?  It’s genuine.  You can clearly tell there is a human being at the other end of that Twitterstream and that’s a welcome change in today’s cold world of corporate policies and standard operating procedures that leave we human beings feeling as if all companies care about is our money.

What Virgin is trying to do is remind its customers that there are people behind the company – people that genuinely care about having fun, working together and creating a pleasant experience for Virgin’s customers every day.  It’s refreshing, to say the least.  Most of all, Virgin actually interacts with people via Twitter.  Who?  Huh?  What?  That’s right…real person-to-person interaction.  Here is a perfect example:

Virgin actively engages its Twitterverse regularly and with far more than just flight delays and vacation specials.  I applaud their efforts and hope that other companies take a close look at what they are doing from a public relations standpoint – and more importantly – I hope other companies follow suit.  Not only does Virgin ask their customer for feedback, they share those experiences openly in their own Twitterstream.  They ask what they’re doing well and what they could be doing better – and then expect to get answers.

Perhaps one of the most impressive aspects of Virgin’s Twitterstream are the images.  At first glance, it seems a bit odd to see so many images in a company’s tweets.  That is, until you realize the origin of the images – Virgin’s customers.  These are not high-dollar professional marketing photographs being used to psychologically lure consumers into believing that Virgin is awesome because of their imagery.  These are images taken by Virgin’s passengers.

The subjects of the photos vary from terminals and gates to scenery taken out the window during the flight to the aircrafts themselves, but the fact that Virgin encourages its travelers to interact with them in such a way is what is truly impressive – and may prove to be revolutionary.  Stop and think about it for a moment…you take a picture of the monitor in front of you on a Virgin flight, post it to Twitter letting your followers know you’re watching a college football game during your flight home.  Later you see that Virgin has retweeted that image.  Doesn’t that build rapport and make you feel as though Virgin is listening – that they may actually care about the experience they’re creating for their passengers?

Following Virgin’s Lead

If Virgin is monitoring something as seemingly benign and innocuous as the scenario above, imagine how responsive they would be in the event that one of their passengers had some sort of customer service related issue?  How do you suppose Virgin finds this information in the first place?  Chances are, they’re using Twitter Search – and so should you!  That may be one of many tools they’re using, but Twitter Search is certainly easy to use and allows you to search tweets in real time, so when someone’s tweet includes “Virgin America”, it will show up in the search results immediately – an invaluable online reputation monitoring tool, to say the least.

The lesson to be learned from Virgin America is this:  introduce a human factor back into your company’s interaction with customers.  What makes Virgin’s use of Twitter so impressive is that it isn’t full of corporate mission statements and canned phrases clearly written by marketing people in an attempt to manipulate the masses.  These are one-on-one interactions between whoever is charged with monitoring their Twitter account and actual Virgin customers.  It’s not merely corporate jargon and marketing-ease.  Virgin is clearly interested in building a rapport with their passengers and creating repeat business as a result of it.  What are you doing to build a rapport with your target audience?  Are you speaking their language?  More importantly, are you communicating with them using the mediums they’re using to communicate with each other about you?

photo credit: bribriTO

There is no quick and easy way to build links.  If something is quick and easy, it’s almost guaranteed to be worthless.  Building links is no different.  Before going any further, you should understand the difference between FOLLOW and NOFOLLOW links.  If you’re unsure, “What are DoFollow links?  What is NoFollow?  Remove the mystery” should help you understand.

Unless you are using a Firefox plugin – such as SearchStatus or SEO For Firefox, you would never know that many of the links you click on during your time online are NOFOLLOW links.  Just because a link is NOFOLLOW doesn’t mean it holds no value.  NOFOLLOW links from sources like StumbleUpon, Digg and Twitter still hold a great deal of value due to their potential to drive traffic.  They aren’t, however, useful for increasing a site’s inbound link count.

Directories – Easy to Get, but Low Value

Keep in mind that the easier a link is to get, the less value it will hold in the grand scheme of things.  What do I mean by that?  Online directories are a great example.  There are thousands of directories and a decent percentage of them offer free submission.  It can take several weeks for a link to appear as a result of a free submission, but they’re not hard links to obtain.

Many directories also offer a few paid options, typically ranging from a few bucks up to about $20 for a basic link.  You’re not paying for the link – you’re simply paying to jump to the front of the line and have your link added to the directory faster.  So, the moral of the story – if the link is easy to get, it’s on the low end of the totem pole of value.

Social Media – Great Traffic Sources, and Some Link Popularity Too

The popularity of social networking has led to the development of a seemingly endless number of sites with a social focus.  While many of these, like Digg and StumbleUpon, employ NOFOLLOW links, some – like Reddit – don’t use exclusively NOFOLLOW links.  That said, not only do links from these sources have the potential to drive visitors directly from the sites themselves, but links from them also count toward your site’s overall inbound link count.

Keep in mind that it is not only the sites themselves that can provide you with much needed inbound links.  The profiles you set up on them often include links to your website that have FOLLOW links.  BrightKite, Technorati, and Identi.ca profiles, for instance, all provide FOLLOW links.

Blogs – Become Part of Your Industry’s Online Community

As long as inbound links and PageRank play such a significant role in ranking, building links back to your site or to individual posts on your blog is essential.  Laying the ground work for future link building possibilities and enticing others to link to you of their own accord is all about building relationships.

Don’t underestimate the value to being active within the blogs and forums that cater to your industry or complimentary industries.  While many blog comment sections incorporate NOFOLLOW links by default, many do not.  Even the NOFOLLOW comment links hold value – particularly if you’re commenting on a highly trafficked blog – but only blogs that use FOLLOW links in the comments sections will increase your site’s number of inbound links.

There are plenty of blogs and forums that use FOLLOW links.  Monty’s Mega Marketing, JR’s Internet Marketing Strategies, Court’s Internet Marketing School, Life Learning Today, and Tucson SEO Solutions each provide lists of FOLLOW blogs and forums that you can use to get started.

Here’s a tip – don’t be a tool.  If you do nothing but drop links to your site without contributing something useful and worthwhile to the community as a whole, you’ll soon be dismissed as a spammer.  Yes, the comments you leave on a blog or forum using FOLLOW links may still help with link popularity, but wouldn’t it be nice if those comments actually encourage people to visit your site or blog, as well.

Remember, building links isn’t just about building links – it is about building relationships and exposing your site to as many potential visitors as possible, then providing those visitors with something of value upon their arrival.

Link Baiting – Enticing Other Sites to Link to You

These are the most difficult links to get.  Unlike directory listings, blog comments and social media profiles, you have no real control over these links.  You may be able to influence their decision in a variety of ways, including coming right out and asking for the link, but the decision is ultimately in someone else’s hands entirely.

Here are some tips that will help your keep a visitor’s attention and entice others to link to it:

1. Make a good first impression.  We’ve all heard that before, but the adage holds true online, too.  You’ve got a few seconds to make a good impression and that’s it.  Problems like broken links and images that don’t load quickly lead to frustration and reduce the chance of someone wanting to link to your site.
2. Give people a reason to come back.  If your site is made up of five static pages that tell about your business and your products, visitors have no reason to come back or link to you.  Provide something of value on a regular basis – like a useful blog post – and people are more likely to come back, as well as link to you.
3. Be YOU!  Whether you’re a one-person operation or there are 10,000 members on your team, the website should have a personality.  People are tired of the same old corporate line of bull, which is why Social Media has taken off like a rocket.  People want to deal with people – not corporate entities.  Give visitors a glimpse into who you are and they’re much more likely to find something of value worth linking to.
4. Don’t be afraid of controversy, but understand the difference between controversial and offensive.  You don’t want to shy away from controversial topics because controversy breeds traffic and encourages other sites to link to you.  You do, however, want to avoid alienating huge groups of people.  Tackling a controversial topic can backfire – approach it carefully and with a clear purpose.

Nothing is more important in today’s world of social networking and Web 2.0 than your online reputation.  Dipping a toe into that pool for the first time can be intimidating.  Millions of people use Social Media outlets, blogs, social networking and bookmarking sources on a daily basis – not only to discuss products, services and brands, but to share their experiences with others.  If you’re not actively monitoring and managing your online reputation, you’re falling behind countless individuals and companies that already are.

Knowing what is being said about you online is essential, whether from a personal or business standpoint.  More and more employers and prospective customers use the Internet to research a people and products by name, rather than simply relying on a resume, online profile or company website for information.

If you’re looking for a job, chances are anyone considering you for employment will Google you long before contacting you about a position.  If you don’t know what publicly accessible information exists, you are doing yourself a great disservice.  One of the best things you can do to manage your own reputation is start your own blog and make sure it ranks well when someone uses your name in a keyword search.

Free Reputation Monitoring Tools

Google offers a free service called “Google Alerts” (google.com/alerts) that will help you learn more about the conversations going on about you or your brand online.  While the information is useful, it is very basic and doesn’t provide a great deal of detail – nor does it include everything that may exist about you online.  That said, it should be first on your “to-do” list when working to monitor your reputation online.

Setting up a Google Alert couldn’t be easier – simply access your Google account and follow the “Alerts” link in the administration page.  If you’re managing your personal reputation, you will obviously want to set up an alert for your name (and for any variations of it that may be used when people mention you).

If you’re hoping to learn more about what the world is saying about your business, you will want to set up an alert for the name of your business – as well as your specific product names and/or the services that you offer.  Yahoo! offers a similar service, not surprisingly called “Yahoo! Alerts” (alerts.yahoo.com).

Creating Custom RSS Feeds

You can also monitor your online reputation on sources like Technorati, Yahoo! News, Google News and BlogPulse by setting up RSS feeds that will deliver results directly to your RSS reader of choice.  Technorati, for instance, allows you to set up “Watchlists” for selected terms.  These terms could be your name, your company, your brand, your products or anything else you want to keep tabs on.  You can then subscribe to your own watchlists via RSS.

BlogPulse offers a similar service, called “Search the Blogosphere”, which you can then subscribe to and receive updates via RSS.  That way you don’t need to take the time to log in and search manually on a daily or weekly basis – the results are delivered to your RSS reader automatically.  Yahoo! and Google News provide the same ability to set up RSS feeds.

Monitoring Social Media Sources

Using Social Media monitoring is also essential.  With the rise of Social Media, Internet users now have an unprecedented platform for sharing their experiences, thoughts and opinions with millions of other users whenever the urge strikes them.

We can communicate with each other in ways that were never possible before Social Media and networking sources like Twitter, StumbleUpon, Digg, Facebook and YouTube.  We are able to connect with one another on a real-time basis – sharing our experiences, making recommendations and exchanging information on an enormous scale.  I highly recommend using Twitter’s search feature (search.twitter.com) on a regular basis to keep tabs of real-time conversations going on about you, your company, your brand or your products and services.

Most popular Social Media sources use what are called “tags”.  When a user writes a review or a post or carries on a discussion, they include what are called tags that identifies what the information is in reference to.  My favorite tag search service is called Keotag and it tops my list because it provides access several to Social Media and networking sources on one easy to use platform.

Like Techorati and Blogpulse, Keotag allows you to subscribe to RSS feeds for the tags and sources you choose.  Blog Catalog also offers a new “Social Search” feature that you can use to search for mentions of your selected terms (your name, your brand, your products, etc.) across a variety of Social Media and networking outlets like Twitter, StumbleUpon and Facebook.

Paid Online Reputation Monitoring

It’s important to mention that in addition to the free services mentioned above, there are a variety of paid services you can subscribe to that will help you monitor your online reputation and keep tabs on where your name and/or brand are mentioned on the Internet.  Services, like Trackur and Reputation Defender, provide a great deal more detail, control and flexibility over monitoring your reputation.

It’s important to understand that monitoring your reputation is very different from managing your reputation.  Monitoring, meaning simply knowing what is being said, is a passive strategy.  It is equally important, if not more so, to take an active role in developing your online reputation.

Take an Active Approach to Your Online Reputation

There are seemingly innumerable resources out there that you can use to promote what you want people to know about yourself or your business.  Social networks are a great way to manage your reputation and increase your presence online and in the search results.  If someone were to search for you by name or for your company by brand, what information would they find?  You have more control over that than you might think.

Take LinkedIn, for instance.  LinkedIn is a networking tool for professionals.  Many believe that LinkedIn is little more than an online resume, but it is much more than that.  LinkedIn is a powerful tool for creating an online presence for yourself and your company by utilizing a variety of applications, creating a network of “connections” and obtaining recommendations from those you and your company have worked with over time.  LinkedIn also performs well in search results in major search engines, giving you and your company more exposure in search results.

There are a large number of ways to increase your presence in search results and actively manage your online reputation.  Keep in mind that the more Social Media and networking profiles you create, the more likely they are to show up in the search results for your name or your company name.  To look at it in a different way, if someone has said something negative about you online it may be important for you to try to bury that information further down in the search results so it isn’t among the first things to grab a search users attention.

Furthermore, the more results there are for your name or company name in the search results, the more reputable and reliable you or your company will seem to the person searching – and the less significant a negative comment will seem.  As Web 2.0 continues to evolve into Web 3.0, Web 4.0 and Web 999.0, it will be essential to have profiles on as many sources as possible, as you never know where someone might be searching for you or your company.

In addition to creating your own website and/or blog, you should create profiles and accounts using sources like MyBlogLog, Squidoo, Twitter, VisualCV, Google Profiles (which you set up through your Google account admin page), Digg, Fark, Mixx and many more.  If you’re not sure where to start, I recommend checking out Social Marker where you will find a list of some of the most widely used and popular social bookmarking and networking websites available today.  It certainly isn’t an all-inclusive list, but it is a great place to start for someone venturing into the realm of Social Media, networking and bookmarking for the very first time.

Imagine that you were about to embark on a cross country drive from New York to Los Angeles.  Now imagine that you have no GPS, no map and no idea where you’re going.  That is exactly what trying to optimize a site without doing keyword research is like.  Nothing is more important to your initial SEO strategy than keyword research. Many small business owners don’t even know where to begin, so the effort put into optimizing a site to perform well in organic search results is often wasted and the ultimate result is a great deal of frustration.

It would be pretty easy to rank in the number one position for “supercalifragilisticexpialidocious cupcakes”, but how many people actually use that term in a keyword search?  I’ll answer that question for you:  none.  Step one of any organic SEO strategy must be to identify the keyword terms that apply to your site and determine which of them generate the greatest number of searches.  Without keyword research there is no way to properly optimize the title, META tags, text content, etc.

Step 1:  Finding Popular Keyword Terms

Choosing and sticking to some good search engine marketing software is the key. There are a large number of keyword suggestion tools to choose from.  I have selected the three that follow for their ease of use…and it doesn’t hurt that they’re also free.

• Free Keyword Suggestion Tool from Keyword Discovery – this is the most simple and straight forward of the featured tools.  Rather than reporting the average number of searches from each search engine individually, it consolidates the number of searches from various search engines into one, easy to understand average number.  It’s a good tool to use to get an idea of the popularity of a particular search term in a very general sense.This tool also provides a list of terms based on the seed keyword you start with.  For example, if you input the term “ipod”, this tool will report the average number of searches done for that specific term, as well as the average number of searches for other terms that include the original seed keyword – like “ipod touch”, “ipod nano”, etc.

• SEO Book Keyword Suggestion Tool – this tool operates under the same premise as Keyword Discovery’s free tool, but it provides much more detailed information.  The SEO Book tool provides results from Google, Yahoo! and MSN separately, rather than combining the estimated searches into one all-inclusive number.  SEO Book also provides links to other useful research tools like Google Trends, Google Suggest and Google Insights.Like the Keyword Discovery tool, this tool provides a list of related terms that include the term input in your initial query.  While the amount of information provided by this tool can seem overwhelming initially, it is a good idea to familiarize yourself with this data – if not right away, then at some point in the future.  This is one of my favorite tools.

• Google Adwords Keyword Tool – perhaps the most noteworthy feature of this tool is that it actually comes from Google.  Essentially that means that you’re gleaning the data right from the horse’s mouth, so to speak.  Yes, you do have to have and be logged in to your Google Account.  If you don’t have a Google Account, sign up for one.Google’s keyword tool is particularly useful because, unlike the other two, it also provides a list of related terms that don’t necessarily include the actual seed keyword.  What does that mean exactly?  In short, it means you may input the term “ipod” as the seed keyword, but the Google Adwords tool will provide you with search volume data for related terms like “mp3 player” and “4th generation”.  This is a just a glimpse into how search engines have evolved over time and that organic search results are generated based not only on the specific term used in the query, but on semantically related terms.

Step 2:  Assessing Keyword Term Competition

Now that you’ve created a list of popular terms that apply to your site, the next step is determining how competitive those terms are.  Let’s be honest…”ipod” may be the most popular term that applies to your website, but that doesn’t mean it is a good term to target.  The competition for that term is ridiculously high.  That said, there are several ways to assess the competitiveness of a search term so you can choose realistic terms that apply to your website.

First, simply do a Google search using the term.  This is a very basic approach, but it can provide useful information and help you avoid wasting time doing more in-depth research into a term’s competition.  When you input “ipod” into a Google search the query returns about 240 million results.  Move on.  Focusing on a term with such enormous competition is not realistic for most websites.  Targeting a more specific term, like “refurbished ipod nano”, increases your likelihood of success substantially.

Once you’ve checked out the terms by doing a normal search, it’s time to take things a step further.  Rather than simply using the term itself in the search this time, input the term preceded by “allintitle:” (i.e. “allintitle:refurbished ipod nano” – without the quotation marks, of course).  This query will let you know how many website pages exist what that particular target term in the page title.

Similarly, doing a query for the same keyword term preceded by “allinanchor:” (i.e. “allinanchor:refurbished ipod nano” – again, without he quotation marks) will let you know the number of pages linked to with that particular anchor text.  Because the search engines use the anchor text of links to a page to help them to determine what a given page is about, anchor text is very important them.  Search engines also take the anchor text and number of links into account when ranking a page for that term, so knowing how many pages are linked to using that term is very useful.  As you would expect, the larger the number of results for the allintitle: and allinanchor: queries, the greater the competition is for that term.

Putting the Pieces Together

Once you’ve gathered useful competitive data on the list of applicable target terms for your site, the question becomes what to do next.  Rather than trying to optimize a page for an unrealistic term, like “ipod”, you can use the data you’ve collected via the allintitle: and allinanchor: queries to choose terms that are popular, but not ultra-competitive.  That is the key to developing a successful organic SEO strategy for your website – not only as it relates to on-page SEO factors titles, META tags and text content, but to link building strategies, as well.