Search engines use indexing robots, computer programs designed to scan the web and websites to record pages in their databases. Each search engine has its own robot, with its own crawling parameters (frequency, depth, etc.). For example, Google has its own indexing robot. Therefore, for a page to appear in search results, it must first be visited and indexed by the appropriate robot. This initial visit can result from a direct submission to the search engine or from the robot discovering the page through an external link.

To be indexed effectively, a page must be accessible and well-structured for indexing robots. The operation of these robots, also called crawlers, is explained in detail in a white paper on SEO published by Smile.