How to pull every job from a company's careers page without scraping HTML

Tidefetch — Mon, 09 Nov 2026 14:01:25 +0000

Most company careers pages aren't custom. They run on an applicant-tracking system (Greenhouse, Lever, Ashby, Workday, SmartRecruiters, Workable, Recruitee or Personio), and each one publishes a public JSON or XML feed so the page can show its jobs. You can call those feeds directly: no browser, no login, no proxies.

Find the board name

Open the careers page and click any job. The address tells you the system and the company's board name:

boards.greenhouse.io/stripe -> Greenhouse, "stripe"
jobs.lever.co/spotify -> Lever, "spotify"
jobs.ashbyhq.com/openai -> Ashby, "openai"
nvidia.wd5.myworkdayjobs.com/NVIDIAExternalCareerSite -> Workday

Greenhouse, Lever and Ashby: one request each

import requests

# Greenhouse: boards.greenhouse.io/stripe -> board "stripe"
r = requests.get("https://boards-api.greenhouse.io/v1/boards/stripe/jobs",
                 params={"content": "true"}, timeout=30)
r.raise_for_status()
for job in r.json()["jobs"]:
    print(job["title"], "|", job["location"]["name"], "|", job["absolute_url"])

# Lever: jobs.lever.co/spotify -> "spotify" (EU accounts use api.eu.lever.co)
for job in requests.get("https://api.lever.co/v0/postings/spotify",
                        params={"mode": "json"}, timeout=30).json():
    print(job["text"], "|", job["categories"].get("location"), "|", job["hostedUrl"])

# Ashby: jobs.ashbyhq.com/openai -> "openai"; includeCompensation adds pay ranges
data = requests.get("https://api.ashbyhq.com/posting-api/job-board/openai",
                    params={"includeCompensation": "true"}, timeout=30).json()
for job in data["jobs"]:
    pay = (job.get("compensation") or {}).get("scrapeableCompensationSalarySummary")
    print(job["title"], "|", job["location"], "|", pay)

Greenhouse's content field is HTML that has been escaped, so run html.unescape() before parsing it. An unknown board returns 404 on all three.

Workday: paged, 20 at a time

Take the host, tenant and site name from the careers address.

import time
import requests

host, tenant, site = "nvidia.wd5.myworkdayjobs.com", "nvidia", "NVIDIAExternalCareerSite"
api = f"https://{host}/wday/cxs/{tenant}/{site}"

jobs, offset = [], 0
while True:
    page = requests.post(f"{api}/jobs", json={"appliedFacets": {}, "limit": 20,
                                              "offset": offset, "searchText": "data engineer"},
                         timeout=30).json()
    rows = page.get("jobPostings", [])
    jobs += rows
    if len(rows) < 20:      # last page
        break
    offset += 20            # Workday rejects pages larger than 20
    time.sleep(0.5)         # be polite

# Full details (exact date, employment type, description) need one GET per job
info = requests.get(f"{api}{jobs[0]['externalPath']}", timeout=30).json()["jobPostingInfo"]
print(len(jobs), info["title"], info["startDate"], info["timeType"])

A limit above 20 returns HTTP 400. For big employers the total field can show 2,000 even though paging keeps working past that.

What I learned normalising thousands of postings

Pay is often missing, and where it exists it's messy. Ashby, Lever and Recruitee have structured pay fields; elsewhere it's in the description. In my runs, 80% of OpenAI's jobs had a range but only 2% of Stripe's.
"$20M-$50M in annual payment volume" looks like a salary range to a naive regex. Require a pay word (salary, compensation, pay range) nearby.
Trust the currency code over the symbol: "$150,500 - $225,700 CAD".
Employers mislabel periods. I've seen a $290,000 salary tagged "bi-weekly". But 25,000 Ukrainian hryvnia a month is a normal monthly wage, so only correct this for high-value currencies.
Give every job a stable key (system:company:id) and de-duplicate, both across runs and within one run.

If you'd rather not maintain it