A recruiter publishes a vacancy on Monday and expects candidates to find it through Google by the end of the week. Often that does not happen. The page is crawled weeks later, or not at all, and by the time it appears the role may already be filled.
TL;DR: Job listings are hard for Google because they are numerous, short-lived and often loaded with JavaScript. Three things cause most problems: listings buried deep in the site structure, content that only appears after scripts run, and expired vacancies that stay online. Fix them by linking listings from city and specialty hubs within three clicks of the homepage, making sure vacancy text is in the HTML, and removing expired jobs with the signals Google documents.
This article explains why career sites struggle with crawling, what a fast-changing listings site can teach them and how to check your own pages.
Why do job listing pages get crawled late or not at all?
Because job sites combine three things Google finds expensive: thousands of URLs, constant change and thin pages. A single employer can have hundreds of vacancies, each with its own URL, and a staffing brand can have tens of thousands, with pagination and filters multiplying the count.
Google’s crawl budget guide is written for this kind of site. It points to sites with 10,000 or more unique pages that change daily, and to sites with many URLs in the “Discovered – currently not indexed” status in Search Console. If your listings change daily, your site is in the group the guide describes.
What can a fast-changing listings site teach career sites about crawl depth?
That Googlebot crawls what is close to the homepage and neglects what is far. Pages sitting more than three clicks away, like the deepest vacancy pages on a large career site, get visited rarely, and a sitemap alone does not change that.
DOM.RIA, a real estate marketplace that adds thousands of listings a day, had street, metro and district directories more than three clicks from the homepage. Log analysis in JetOctopus, seo agency management software, showed Googlebot favoring city and regional pages, so the team added mini-sitemaps on city directories that linked to the under-crawled pages; in 15 days Googlebot visits to street pages grew from 525,000 to 1,398,000.
The same structure fits vacancies: hub pages by city, specialty or employer, each linking to its current listings, placed high in the site hierarchy. The DOM.RIA team also found that a footer sitemap did not help: in 15 days Googlebot visited only 16 of its sitemap pages. A list of URLs is not a substitute for links from pages the bot already trusts. For a recruitment brand, that means the city and specialty pages that candidates already visit deserve the strongest internal links.
Does JavaScript hide your vacancies from Google and AI crawlers?
It can, depending on the crawler. Googlebot renders JavaScript, though not instantly. AI crawlers are different: Vercel and MERJ reported in December 2024 that none of the major AI crawlers they tested render JavaScript. They fetch script files but do not run them, and Vercel validated its findings on two job board sites among others.
If a career site loads the job title, location and description through scripts after the page opens, an AI assistant’s crawler may fetch an empty shell. A simple test: view the page source, search for the job title and description, and compare it with what you see in the browser. If the text is missing from the source, your vacancy is invisible to crawlers that do not run scripts.
What should happen to expired job postings?
They should disappear or be clearly marked as expired. Google’s documentation for job posting structured data states that it does not allow expired job postings, and it gives three ways to handle them: set validThrough to a past date, remove the page so it returns a 404 or 410 status, or remove the JobPosting markup. If a job is filled early, Google advises removing the posting.
Pages that return a normal page for a removed job are also a crawling problem. Google’s crawl budget guide lists soft 404s and permanently removed pages that do not return 404 or 410 among the things that waste crawling. For a career site, a closed vacancy should either redirect to the closest live category or return a proper 404 or 410.
What technical details do individual job pages need?
Google’s documentation sets a few clear rules for job pages. The JobPosting structured data belongs on the page of the individual job, not on list or search result pages, and it has to match the job description a visitor sees in the browser. Job details must be viewable without logging in, so a vacancy behind a login wall cannot be used.
For discovery, Google’s documentation says it prefers the Indexing API for job URLs, because it prompts faster crawling, while still recommending a sitemap for full-site coverage. For a recruitment brand with short-lived vacancies this is worth discussing with developers: the speed of discovery directly affects how long a vacancy is visible while it is still open.
Finally, keep list pages useful. A paginated list of vacancies is a navigation page, not a place for structured data, and an empty city page after the last job in that city closes is a thin page. Decide in advance whether it redirects to a broader region or shows related roles.
A checklist for recruitment marketing teams
1. Count the vacancies in your sitemap and in Google’s index, and note the gap.
2. View the source of three vacancy pages and confirm the title, location and description are in the HTML.
3. Link every vacancy from a hub page by city, specialty or employer, and keep it within three clicks of the homepage.
4. Set up automatic handling for filled and expired jobs: validThrough, 404 or 410, or removal of the markup.
5. Check that the JobPosting markup sits on each vacancy’s own page and matches the visible text.
6. Review server logs once a month for the share of vacancies bots requested and the share of 404 responses.
Conclusion
Closing a vacancy is part of the job lifecycle, not an afterthought. A filled role that still ranks sends candidates to a dead end and can put the site at odds with Google’s rules for job posting markup. The cleanest approach is to tie the closing action in your applicant tracking system to the website, so that closing a job also sets the expiry signal or removes the page.
A vacancy page has a short life, so crawl speed is part of its value. Keep listings close to the homepage, put the text in the HTML, remove what has expired and check the logs. If your recruitment marketing is run by an agency, ask for these four checks in the next report. If it is run in-house, put them on the calendar: a monthly hour spent on structure, source, expiry and logs is cheaper than losing the first week of every vacancy.
FAQ
How quickly should expired job postings be removed from search?
As soon as the job is filled or closed. Google does not allow expired job postings and gives three options: a past validThrough date, a 404 or 410 response, or removing the JobPosting markup. Build the rule into your publishing system, so a closed job triggers it automatically rather than relying on someone to remember.
Do AI assistants read job listings that load with JavaScript?
Generally not. Vercel and MERJ found in December 2024 that none of the major AI crawlers they tested render JavaScript; Gemini is the exception because it uses Googlebot’s infrastructure. To check a page, compare the HTML source with the rendered page and confirm the job title and description are present in the source.
How many clicks from the homepage should a job listing be?
Aim for three at most. On DOM.RIA, directories sitting more than three clicks from the homepage were crawled rarely until mini-sitemaps on city pages linked to them. Build hubs by city and specialty, link each live vacancy from one, and keep the hubs close to the top of the site.
Do I need an XML sitemap for thousands of vacancies?
Yes, for coverage, but do not expect it to drive crawling. DOM.RIA’s footer sitemap led to only 16 sitemap pages visited by Googlebot in 15 days. Google recommends sitemaps for full-site coverage and says outdated ones waste crawling; for job URLs it prefers the Indexing API because it prompts faster crawling.
How can I tell whether Google actually crawls my vacancy pages?
Use two sources. Search Console’s Page indexing and Crawl Stats show part of the picture. Server logs show each request, so you can calculate how many of your live vacancies Googlebot requested in the last 30 days and how many requests ended in a 404. A low share points to a structure or linking problem.
