Translation guide
A web crawler is a program that automatically browses the web to index content. In Japanese, it is most commonly referred to by the borrowed English term, but native equivalents exist.
The most common way to refer to a web crawler in Japanese, used in both technical and general contexts.
The standard loanword from English, widely understood in IT contexts.
このサイトはクローラーにインデックスされています。
This site is indexed by web crawlers.
Explicitly 'web crawler', less common than the shorter form but unambiguous.
ウェブクローラーが新しいページを検出しました。
The web crawler detected a new page.
Sometimes used to refer to crawlers in the context of robots.txt, but can be ambiguous.
robots.txtでロボットのアクセスを制御します。
Control robot access with robots.txt.
Specifically a crawler used by search engines like Google.
Literally 'search engine crawler', clear and natural.
検索エンジンのクローラーがサイトを巡回しています。
Search engine crawlers are visiting the site.
Short for 'robot', commonly used in SEO contexts.
Googleボットがページをクロールします。
Googlebot crawls pages.
A less common, more descriptive Japanese term for web crawler.
Literally 'patrol robot', sometimes used in technical documentation.
巡回ロボットが定期的にウェブサイトを訪れます。
Patrol robots periodically visit websites.