Spotlot

Coverage

Every source we read, when we last read it, and what it gave us. A site that blocks us is recorded and left alone.

Live ·

sources set up
1,113
gave adverts on their last read
1,033
marketplaces left alone
15
public datasets joined
8

Every source

100 of 1,113 · adverts read of adverts found on the last pass

Show 50 more

A row is one pass, not the yard’s size: an index pass every two hours reads the top of each yard, and a deep pass twice a day walks it. The dealer’s own page carries its live count.

Surveyed and left alone

Marketplaces that refuse an honest crawler are recorded with their reason and asked again every Monday, exactly as the crawler would ask. A block is never worked around.

  • Audi Centre dealer networkOut of scope directly. Probed 2026-08-11: audicentresydney, audicentremelbourne, audicentrebrisbane and audicentredoncaster all answer HTTP 400 to robots.txt and to every inventory path — the server rejects an ordinary request. No circumvention attempted. Their stock is reachable legitimately through dealer:autosportsgroup, which owns several of them.Asked again 5 Oct: HTTP 403 to the honest user agent; still blocked
  • carsales.com.auDataDome challenge on all listing fetches (403). robots.txt itself allows clean /cars/ paths for * with Crawl-delay 2, but WAF blocks in practice. Out of scope pending a licensed feed (survey 2026-08-10).Asked again 5 Oct: DataDome challenge (HTTP 403); still blocked
  • DriveCloudflare managed challenge site-wide including robots.txt. Out of scope (survey 2026-08-10).Asked again 5 Oct: Cloudflare challenge (HTTP 403); still blocked
  • easyauto123robots fully open, no WAF challenge observed (survey 2026-08-10). Extractor not yet built. Probe 2026-08-11: robots allows and the page returns HTTP 200, but /buy/used-cars is client-rendered (Next flight payload, zero ld+json, no make/price/vin keys in HTML) — no server-side stock for our extractors. Out of scope pending an official feed; not worth JS rendering. Re-probed 2026-08-11: inventory paths answer HTTP 404 while returning 180KB–740KB of markup — a soft 404 from a client-rendered app, so there is nothing in the HTML to parse. Not blocked, just not served server-side. Would need its JSON API identified, which is a separate piece of work. | 2026-08-15 sized: inventory page answers 200 and no existing extractor recognises it, and scanHost finds no listing URLs (easyauto123 publishes no usable sitemap; suttons has sitemap.xml but it lists no vehicles). Enabling needs a new extractor AND pagination-based discovery, not a config change. Do not re-probe expecting a quick win. | Re-probed 2026-09-05 for an extractor: the used-car grid is fetched by the page's own JavaScript through a Next.js server action (a POST to the page URL keyed by an action id inside the bundle), not through any public URL; /api/inventory/agg answers only aggregations (2,059 cars, every stock number) and /api/inventory/stats only per-car view counts; the sitemap robots.txt names answers 404; no detail URL shape could be found for a stock number. Replaying the app's internal action protocol is not a public read path, so no extractor was built. Left out of scope.Asked again 5 Oct: answers HTTP 200 to the honest user agent, 540 KB of markup; whether it is in scope is a decision for a person, not this script
  • Grays (auctions)surveyed 2026-08-11: robots.txt disallows the vehicle category. Respected — out of scope.Asked again 5 Oct: answers HTTP 200 to the honest user agent, 439 KB of markup; whether it is in scope is a decision for a person, not this script
  • Gumtree AUPeakhour WAF 403s everything including robots.txt. Out of scope (survey 2026-08-10).Asked again 5 Oct: HTTP 403 to the honest user agent; still blocked
  • Hyundai Australiasurveyed 2026-08-11: robots.txt permits crawling, but the site publishes no used-vehicle inventory page — new-model brochures only.Asked again 5 Oct: answers HTTP 200 to the honest user agent, 119 KB of markup; whether it is in scope is a decision for a person, not this script
  • Lexus Australia (national used locator)surveyed 2026-08-11: robots.txt disallows the used-vehicle search path. Respected — out of scope.Asked again 5 Oct: answers HTTP 200 to the honest user agent, 761 KB of markup; whether it is in scope is a decision for a person, not this script
  • Lexus dealer networkOut of scope. Probed 2026-08-11: every dealer site tested — Sydney City, Parramatta, Brisbane, Perth — serves robots.txt at 200 but returns a Cloudflare "Just a moment" challenge (403) on its inventory pages. lexus.com.au's own sitemap.xml does the same. Recorded and left alone; no circumvention attempted.Asked again 5 Oct: answers HTTP 200 to the honest user agent, 761 KB of markup; whether it is in scope is a decision for a person, not this script
  • Manheim Australia (auctions)surveyed 2026-08-11: returns HTTP 403 to the honest UA on robots.txt and every inventory path. Not circumvented — out of scope.Asked again 5 Oct: HTTP 403 to the honest user agent; still blocked
  • MG Motor Australiasurveyed 2026-08-11: robots.txt permits crawling, but the site publishes no used-vehicle inventory page — new-model brochures only.Asked again 5 Oct: answers HTTP 200 to the honest user agent, 353 KB of markup; whether it is in scope is a decision for a person, not this script
  • Peter Warren Automotiverobots Allow: /, no challenge observed (survey 2026-08-10). Extractor not yet built. Superseded 2026-08-11: confirmed as a Dealer Studio site and re-seeded as dealer:peterwarren, which is crawlable. | 2026-08-15: duplicate of dealer:peterwarren, same host, which is enabled on dealer-studio with 1,689 live listings. Marked merged so it is not re-seeded as a separate yard.
  • Subaru Australia (national used locator)surveyed 2026-08-11: robots.txt disallows the used-vehicle search path. Respected — out of scope.Asked again 5 Oct: answers HTTP 200 to the honest user agent, 350 KB of markup; whether it is in scope is a decision for a person, not this script
  • Suttonsrobots fully open (survey 2026-08-10). Extractor not yet built. Re-probed 2026-08-11: inventory paths answer HTTP 404 while returning 180KB–740KB of markup — a soft 404 from a client-rendered app, so there is nothing in the HTML to parse. Not blocked, just not served server-side. Would need its JSON API identified, which is a separate piece of work. | 2026-08-15 sized: inventory page answers 200 and no existing extractor recognises it, and scanHost finds no listing URLs (easyauto123 publishes no usable sitemap; suttons has sitemap.xml but it lists no vehicles). Enabling needs a new extractor AND pagination-based discovery, not a config change. Do not re-probe expecting a quick win. | Re-probed 2026-09-05 for an extractor: /inventory/cars is a shell; the cards are painted in by a POST to /inventory/cars/filter carrying an Elasticsearch query the page builds, and every shape of that request tried from the honest user agent answered HTTP 500 (seven variants, with and without the page's headers). The sitemap lists no vehicles. No public GET path serves the stock, so no extractor was built. Left out of scope.Asked again 5 Oct: answers HTTP 200 to the honest user agent, 938 KB of markup; whether it is in scope is a decision for a person, not this script
  • Toyota Australia (national used locator)surveyed 2026-08-11: robots.txt served, but every inventory path returns a bot-detection interstitial. Not circumvented — out of scope.Asked again 5 Oct: answers HTTP 200 to the honest user agent, 395 KB of markup; whether it is in scope is a decision for a person, not this script

Public datasets

  • ABS ASGS Ed 3 Postal Areas 2021 (centroids)joined
  • ABS regional statistics (POA)joined
  • ANCAP safety ratingsjoined
  • BITRE Road Vehicles Australiajoined
  • FCAI new vehicle sales (topline)joined
  • FuelWatch WAjoined
  • Green Vehicle Guidejoined
  • Australian public holidaysrecorded, not yet joined
  • RBA cash rate + ABS CPIjoined
  • Vehicle recalls (vehiclerecalls.gov.au)recorded, not yet joined