Translation guide
A web crawler is a program that automatically browses the web to index content. In Japanese, it is most commonly referred to by the borrowed English term, but native equivalents exist.
The most common way to refer to a web crawler in Japanese, used in both technical and general contexts.
The standard loanword from English, widely understood in IT contexts.
This site is indexed by web crawlers.
Explicitly 'web crawler', less common than the shorter form but unambiguous.
The web crawler detected a new page.
Sometimes used to refer to crawlers in the context of robots.txt, but can be ambiguous.
Control robot access with robots.txt.
Specifically a crawler used by search engines like Google.
Literally 'search engine crawler', clear and natural.
Search engine crawlers are visiting the site.
Short for 'robot', commonly used in SEO contexts.
A less common, more descriptive Japanese term for web crawler.
Literally 'patrol robot', sometimes used in technical documentation.
Patrol robots periodically visit websites.
Googlebot crawls pages.