Web Crawling

An Internet bot (spider) that systematically browses the World Wide Web to index content and gather data for search engine databases.

Web crawling is the systematic discovery and collection of data across the World Wide Web, executed by automated programs known as spiders or bots. Major search engines deploy proprietary crawlers, such as Googlebot and Bingbot, which begin with a seed list of URLs and recursively follow hyperlinks to process billions of pages. The primary directive is to download page content and metadata for indexing, which forms the core foundation of a search engine's database. Crawlers adhere to a 'politeness policy,' checking the `robots.txt` file on each domain to respect site owner directives and manage server load, ensuring efficient data harvesting without overwhelming the target system. This process is non-negotiable for maintaining a current, searchable index of the web's vast information landscape.

https://en.wikipedia.org/wiki/Web_crawler