Skip to content
Clarvith

ClarvithBot

Clarvith Actors identify as ClarvithBot/1.0 (+https://clarvith.com/bot; public job-board lookup for search and reference; not for AI training). This page describes what the crawler fetches and how it behaves.

What it fetches

  • Public ATS HTTP APIs: Greenhouse, Lever, Ashby, and Workable job-board endpoints documented in the Actor README.
  • Company websites for discovery: homepage and, when allowed, careers paths such as /careers, /jobs, and /company/careers.
  • robots.txt for each host before other paths, per RFC 9309.

robots.txt (RFC 9309)

  • A 2xx robots.txt is obeyed (up to five redirects on that file).
  • 4xx on robots.txt means the host is allowed.
  • 5xx, timeout, or unreachable robots.txt blocks the host for that run.
  • 429 responses pause the host (60 seconds) and reduce rate; repeated 429 may disable a host.

Rate limits (as implemented)

Per-host request pacing in the shared HTTP client:

  • Lever and Workable API hosts: at most 1 request per second.
  • Greenhouse and Ashby API hosts: at most 3 requests per second.
  • Other hosts (company sites): default 1 request per second unless configured otherwise.

Concurrent company fetches respect these limits per host. Workable may be disabled for the remainder of a run after a second 429.

Opt out

To request blocking or discuss crawling, email support@clarvith.com with the hostname and your role. Standard robots.txt rules remain the primary control mechanism.