An XML Sitemap is used to communicate to search engines the URLs that exist on a website and that are available for crawling. The communication that takes place between a website and the search engine is called ’sitemap protocol’ and it is typically in XML format. The sitemap protocol was written in such a way that it would allow for scalability so it can accommodate sites of any size. XML Sitemaps also allow websites to share information about each specific URL on a website; enabling the search engines to more accurately crawl each page on a website.

Information Communicated to Search Engines through an XML Sitemap:

1. When the URL was last updated
2. How often the URL changes
3. How important the URL is in relation to other URLs on the site

For more information please visit: http://en.wikipedia.org/wiki/Sitemap_Protocol

A Robots.txt file is simply a file that is placed on a website’s server with the intention of telling the search engine spiders not to crawl and index pages or sections of a website that would be otherwise publicly viewable.

A few suggested ways to use a Robots.txt file are:

1. To block the search engines from indexing an entire site.
2. Prevent specific sections of a website to be blocked from being indexed.
3. To communicate specific indexing instructions to individual search engines.

Does My Site Need a Robots.txt File?

With the emergence of so many search engines over the past decade, Robots.txt file have become standard protocol for search engine spiders to attempt to locate and view when they first arrive at a website. Search engine spiders are trained to start their crawl looking for the Robots.txt file; this allows the spiders to know which pages to index and which pages should be disallowed from being indexed. Even though your site may not have any pages that you would currently want blocked from being publicly viewable it is still a recommended best search engine optimization practice to include a Robots.txt file on the root folder of your site. It will act as an invitation with the search engines to successfully communicate the message to crawl all the pages on your site.

Some reasons why you may want to exclude spiders from your site:

1. The website is still in development and you do not want your unfinished work to be found by the public.
2. The information presented on your page only pertains to your specific audience once they have arrived at your site and is of no interest to an outside viewer
3. Certain common pages such as: Thank You pages, Privacy Policies, Etc. are prime candidates to be excluded from the search engine indexes because there is no benefit for a web visitor to enter the website onto them. 
4. You may want to exclude your site from specific search engines entirely; whether it be because you do not believe in that search engines procedures or it is a known bot that collects data you do not want shared (i.e.: email addresses)
5. If doorway pages are used on your site (we highly recommend not using them) you may want to exclude specific search engine spiders from viewing them as they will have a negative impact on your optimization efforts on search engines such as Google and Yahoo.

Creating a Robots.txt File

A Robots.txt file is simple to create. It can be created right in Notepad or any text editor. Once it is created it needs to be uploaded to the root folder of your website. Each entry in a Robots.txt file only contains two lines of text.

Layout for a Robots.txt file:

User-Agent: [Spider or Bot name]
Disallow: [Directory or File Name]

Examples of Robots.txt Files:

1. Exclude file or directory from a specific search engine spider:

User-Agent: Googlebot
Disallow: /section/examplefile.htm

2. Exclude a specific section of a website from every search engine spider and bot:

User-Agent: *
Disallow: /examplesection/

3. Allow spiders to index the entire site:

User-agent: *
Disallow:

4. Disallow spiders from indexing any part of the website:

User-agent: *
Disallow: /

Natural search engine optimization, also known as organic search engine optimization (SEO) is the process in which a website is analyzed and undergoes changes to aid in best communicating the target keywords to the search engines. Aspects of search engine optimization include changes to basic code, recommendations to the architecture, adjustments to content visible to the end user, and building link popularity.

The goal of search engine optimization (SEO) is to increase the amount of visitors to a website by ranking well on the search engine result pages. The higher a web site ranks in the results of a search engine, will increase the chances of users visiting the site. It is common practice for Internet users to not click through pages and pages of search results, so where a site ranks in a search is essential for directing more traffic toward the site.

Over 500 million people use search engines on a daily basis. The rising popularity of search engines is making it vital for any business to have a strong presence on the search engine result pages for keywords related to the specific industry. A well-optimized website can expect increased traffic, more leads, and additional sales.

There are many key factors as to why a website should be optimized. The major reason a website should be properly optimized is that over 90% of all online traffic stems from various search engines; such as: Google, Yahoo, MSN, etc…

Therefore having your businesses website accessible by the search engines is proven to be the best way to drive qualified traffic to your website. By properly marketing your businesses messaging through search engine optimization one can expect a boost in qualified web visitors.  

Over the past decade, search engines have evolved into the largest and most cost-effective platforms for advertisers; the ROI on search engine marketing is greater than television, radio, newspapers, or even direct mailings. The success of search engine optimization can be compared to how a phone book used to operate; the difference being that 500 million users are opening that phone book everyday. When a potential consumer is looking for information on a topic they are able to use the search engines to find hundreds even thousands of businesses offering what they are looking for. If a high ranking in the search engine result pages is obtained than this potential web traffic can be directed to your website. With so many regional, national, and international websites online today it is crucial that the right specialized search engine techniques are used to position your businesses website properly. This can mean the difference between opening up the flood gates to new sales or irrelevant web traffic that yields no positive benefit for your business.