Search engine optimization methods are techniques used by webmasters to get more visibility for their sites in search engine results pages.

Getting indexed

The leading search engines, such as Google and Yahoo!, use crawlers to find pages for their algorithmic search results. Pages that are linked from other search engine indexed pages do not need to be submitted because they are found automatically. Some search engines, notably Yahoo!, operate a paid submission service that guarantee crawling for either a set fee or cost per click. Such programs usually guarantee inclusion in the database, but do not guarantee specific ranking within the search results. Two major directories, the Yahoo Directory and the Open Directory Project both require manual submission and human editorial review. Google offers Google Webmaster Tools, for which an XML Sitemap feed can be created and submitted for free to ensure that all pages are found, especially pages that aren’t discoverable by automatically following links.

Search engine crawlers may look at a number of different factors when crawling a site. Not every page is indexed by the search engines. Distance of pages from the root directory of a site may also be a factor in whether or not pages get crawled.

Other methods

Variety of other methods are employed to get a webpage indexed and shown up in the results.

* Cross linking between pages of the same website. Giving more links to main pages of the website, to increase PageRank used by search engines. Linking from other websites, including link farming and comment spam.
* Keyword rich text in the webpage and key phrases, so as to match all search queries. Adding relevant keywords to a web page meta tags, including keyword stuffing.
* URL normalization for webpages with multiple urls, using “canonical” meta tag.

Preventing crawling

To avoid undesirable content in the search indexes, webmasters can instruct spiders not to crawl certain files or directories through the standard robots.txt file in the root directory of the domain. Additionally, a page can be explicitly excluded from a search engine’s database by using a meta tag specific to robots. When a search engine visits a site, the robots.txt located in the root directory is the first file crawled. The robots.txt file is then parsed, and will instruct the robot as to which pages are not to be crawled. As a search engine crawler may keep a cached copy of this file, it may on occasion crawl pages a webmaster does not wish crawled. Pages typically prevented from being crawled include login specific pages such as shopping carts and user-specific content such as search results from internal searches. In March 2007, Google warned webmasters that they should prevent indexing of internal search results because those pages are considered search spam.

Increasing prominence

A variety of other methods are employed to get a webpage shown up at the searchs results. These include:

* Cross linking between pages of the same website. Giving more links to main pages of the website, to increase PageRank used by search engines. Linking from other websites, including link farming and comment spam.
* Writing content that includes frequently searched keyword phrase, so as to be relevant to a wide variety of search queries. Adding relevant keywords to a web page meta tags, including keyword stuffing.
* URL normalization of web pages accessible via multiple urls, using the “canonical” meta tag.