SEO Expert...

Getting indexed	Preventing crawling	Increasing prominence	Legal precedents
The leading search engines, such as Google and Yahoo!, use crawlers to find pages for their algorithmic search results.	To avoid undesirable content in the search indexes, webmasters can instruct spiders not to crawl certain files or directories through the standard robots.	Cross linking between pages of the same website. Giving more links to main pages of the website, to increase PageRank used by search engines.	On October 17, 2002, SearchKing filed suit in the United States District Court, Western District of Oklahoma, against the search engine Google.

URL normalization
URL normalization (or URL canonicalization) is the process by which URLs are modified and standardized in a consistent manner.

The goal of the normalization process is to transform a URL into a normalized or canonical URL so it is possible to determine if two syntactically different URLs are equivalent.

Web crawler
A Web crawler is a computer program that browses the World Wide Web in a methodical, automated manner. Other terms for Web crawlers are ants, automatic indexers, bots, and worms or Web spider, Web robot, Web scutte.

This process is called Web crawling or spidering. Many sites, in particular search engines.

Vertical Search Engines

Web Search Engine
A web search engine is a tool designed to search for information on the World Wide Web. The search results are usually presented in a list and are commonly called hits.

The information may consist of web pages, images, information and other types of files.

Internet Media Type
An Internet media type, originally called a MIME type after MIME and sometimes a Content-type after the name of a header in several protocols whose value is such a type, is a two-part identifier for file formats on the Internet.

A media type is composed of at least two parts: a type, a subtype, and one or more optional parameters.

Character Encoding