Coverage
Every source we read, when we last read it, and what it gave us. A site that blocks us is recorded and left alone.
- sources set up
- 1,113
- gave adverts on their last read
- 1,033
- marketplaces left alone
- 15
- public datasets joined
- 8
Every source
- 365 Car Sales21 of 21
- 5th Street Automotive2 of 2
- A1 Motors21 of 21
- AC Cars161 of 161
- Adelaide BMWnothing on this read
- Adelaide Chery12 of 12
- Adelaide City Kianothing on this read
- Adelaide Vehicle Centre21 of 21
- Adrian Brien Renault - New Inventorynothing on this read
- affordableautosnothing on this read
- Agostino GWM3 of 3
- Albany Toyota21 of 21
- Alice Springs Toyotanothing on this read
- allroundmotors21 of 21
- Alto Groupnothing on this read
- altousmotors140 of 140
- alwaysmovingmotors29 of 29
- AMR Used Carsnothing on this read
- Anthony Motorsnothing on this read
- anthonymotorsales21 of 21
- arrowcarsalesnothing on this read
- Ashmore Used Cars22 of 22
- Aspley Mazda, Aspley21 of 21
- assuredcars36 of 36
- atlascardealer33 of 33
- Australian Automotive Groupnothing on this read
- Australian Car Sales Group110 of 110
- Ausvehicle Sales21 of 21
- Auto Car Wholesalenothing on this read
- Auto Classic, Victoria Parknothing on this read
- Auto Cornernothing on this read
- Auto Credit Cars21 of 21
- Auto Home37 of 37
- Auto Jungle21 of 21
- Auto Marketnothing on this read
- Auto Sales @ Redcliffenothing on this read
- autobargaincenter68 of 68
- autocarcentre11 of 11
- autoloversaustralia44 of 44
- automallmtgravatt396 of 396
- Autopactnothing on this read
- Autopact Performance24 of 24
Merged into Autopact.
- Autosales Australia21 of 21
- autosellers20 of 20
- Autosports Groupnothing on this read
Not read now: surveyed and withdrawn 2026-08-12: answers HTTP 504 to every request, on two probes hours apart. A server fault rather t…
- Autotrader AU2,991 of 17,966
- autoxchangewa34 of 34
- awesomeautosprestige94 of 94
A row is one pass, not the yard’s size: an index pass every two hours reads the top of each yard, and a deep pass twice a day walks it. The dealer’s own page carries its live count.
Surveyed and left alone
Marketplaces that refuse an honest crawler are recorded with their reason and asked again every Monday, exactly as the crawler would ask. A block is never worked around.
- Audi Centre dealer networkOut of scope directly. Probed 2026-08-11: audicentresydney, audicentremelbourne, audicentrebrisbane and audicentredoncaster all answer HTTP 400 to robots.txt and to every inventory path — the server rejects an ordinary request. No circumvention attempted. Their stock is reachable legitimately through dealer:autosportsgroup, which owns several of them.Asked again 5 Oct: HTTP 403 to the honest user agent; still blocked
- carsales.com.auDataDome challenge on all listing fetches (403). robots.txt itself allows clean /cars/ paths for * with Crawl-delay 2, but WAF blocks in practice. Out of scope pending a licensed feed (survey 2026-08-10).Asked again 5 Oct: DataDome challenge (HTTP 403); still blocked
- DriveCloudflare managed challenge site-wide including robots.txt. Out of scope (survey 2026-08-10).Asked again 5 Oct: Cloudflare challenge (HTTP 403); still blocked
- easyauto123robots fully open, no WAF challenge observed (survey 2026-08-10). Extractor not yet built. Probe 2026-08-11: robots allows and the page returns HTTP 200, but /buy/used-cars is client-rendered (Next flight payload, zero ld+json, no make/price/vin keys in HTML) — no server-side stock for our extractors. Out of scope pending an official feed; not worth JS rendering. Re-probed 2026-08-11: inventory paths answer HTTP 404 while returning 180KB–740KB of markup — a soft 404 from a client-rendered app, so there is nothing in the HTML to parse. Not blocked, just not served server-side. Would need its JSON API identified, which is a separate piece of work. | 2026-08-15 sized: inventory page answers 200 and no existing extractor recognises it, and scanHost finds no listing URLs (easyauto123 publishes no usable sitemap; suttons has sitemap.xml but it lists no vehicles). Enabling needs a new extractor AND pagination-based discovery, not a config change. Do not re-probe expecting a quick win. | Re-probed 2026-09-05 for an extractor: the used-car grid is fetched by the page's own JavaScript through a Next.js server action (a POST to the page URL keyed by an action id inside the bundle), not through any public URL; /api/inventory/agg answers only aggregations (2,059 cars, every stock number) and /api/inventory/stats only per-car view counts; the sitemap robots.txt names answers 404; no detail URL shape could be found for a stock number. Replaying the app's internal action protocol is not a public read path, so no extractor was built. Left out of scope.Asked again 5 Oct: answers HTTP 200 to the honest user agent, 540 KB of markup; whether it is in scope is a decision for a person, not this script
- Grays (auctions)surveyed 2026-08-11: robots.txt disallows the vehicle category. Respected — out of scope.Asked again 5 Oct: answers HTTP 200 to the honest user agent, 439 KB of markup; whether it is in scope is a decision for a person, not this script
- Gumtree AUPeakhour WAF 403s everything including robots.txt. Out of scope (survey 2026-08-10).Asked again 5 Oct: HTTP 403 to the honest user agent; still blocked
- Hyundai Australiasurveyed 2026-08-11: robots.txt permits crawling, but the site publishes no used-vehicle inventory page — new-model brochures only.Asked again 5 Oct: answers HTTP 200 to the honest user agent, 119 KB of markup; whether it is in scope is a decision for a person, not this script
- Lexus Australia (national used locator)surveyed 2026-08-11: robots.txt disallows the used-vehicle search path. Respected — out of scope.Asked again 5 Oct: answers HTTP 200 to the honest user agent, 761 KB of markup; whether it is in scope is a decision for a person, not this script
- Lexus dealer networkOut of scope. Probed 2026-08-11: every dealer site tested — Sydney City, Parramatta, Brisbane, Perth — serves robots.txt at 200 but returns a Cloudflare "Just a moment" challenge (403) on its inventory pages. lexus.com.au's own sitemap.xml does the same. Recorded and left alone; no circumvention attempted.Asked again 5 Oct: answers HTTP 200 to the honest user agent, 761 KB of markup; whether it is in scope is a decision for a person, not this script
- Manheim Australia (auctions)surveyed 2026-08-11: returns HTTP 403 to the honest UA on robots.txt and every inventory path. Not circumvented — out of scope.Asked again 5 Oct: HTTP 403 to the honest user agent; still blocked
- MG Motor Australiasurveyed 2026-08-11: robots.txt permits crawling, but the site publishes no used-vehicle inventory page — new-model brochures only.Asked again 5 Oct: answers HTTP 200 to the honest user agent, 353 KB of markup; whether it is in scope is a decision for a person, not this script
- Peter Warren Automotiverobots Allow: /, no challenge observed (survey 2026-08-10). Extractor not yet built. Superseded 2026-08-11: confirmed as a Dealer Studio site and re-seeded as dealer:peterwarren, which is crawlable. | 2026-08-15: duplicate of dealer:peterwarren, same host, which is enabled on dealer-studio with 1,689 live listings. Marked merged so it is not re-seeded as a separate yard.
- Subaru Australia (national used locator)surveyed 2026-08-11: robots.txt disallows the used-vehicle search path. Respected — out of scope.Asked again 5 Oct: answers HTTP 200 to the honest user agent, 350 KB of markup; whether it is in scope is a decision for a person, not this script
- Suttonsrobots fully open (survey 2026-08-10). Extractor not yet built. Re-probed 2026-08-11: inventory paths answer HTTP 404 while returning 180KB–740KB of markup — a soft 404 from a client-rendered app, so there is nothing in the HTML to parse. Not blocked, just not served server-side. Would need its JSON API identified, which is a separate piece of work. | 2026-08-15 sized: inventory page answers 200 and no existing extractor recognises it, and scanHost finds no listing URLs (easyauto123 publishes no usable sitemap; suttons has sitemap.xml but it lists no vehicles). Enabling needs a new extractor AND pagination-based discovery, not a config change. Do not re-probe expecting a quick win. | Re-probed 2026-09-05 for an extractor: /inventory/cars is a shell; the cards are painted in by a POST to /inventory/cars/filter carrying an Elasticsearch query the page builds, and every shape of that request tried from the honest user agent answered HTTP 500 (seven variants, with and without the page's headers). The sitemap lists no vehicles. No public GET path serves the stock, so no extractor was built. Left out of scope.Asked again 5 Oct: answers HTTP 200 to the honest user agent, 938 KB of markup; whether it is in scope is a decision for a person, not this script
- Toyota Australia (national used locator)surveyed 2026-08-11: robots.txt served, but every inventory path returns a bot-detection interstitial. Not circumvented — out of scope.Asked again 5 Oct: answers HTTP 200 to the honest user agent, 395 KB of markup; whether it is in scope is a decision for a person, not this script
Public datasets
- ABS ASGS Ed 3 Postal Areas 2021 (centroids)joined
- ABS regional statistics (POA)joined
- ANCAP safety ratingsjoined
- BITRE Road Vehicles Australiajoined
- FCAI new vehicle sales (topline)joined
- FuelWatch WAjoined
- Green Vehicle Guidejoined
- Australian public holidaysrecorded, not yet joined
- RBA cash rate + ABS CPIjoined
- Vehicle recalls (vehiclerecalls.gov.au)recorded, not yet joined