BaliProp

The problem

Finding a long-term rental in Bali means checking a dozen agency sites with inconsistent formats, duplicate listings, and no shared filters. Prices appear in three currencies and two rental periods. The same villa is listed by four agencies at four prices. I built this to solve my own search and kept going.

What I built

BaliProp is a React app over a Python scraping pipeline and PostgreSQL. It serves 13,246 listings merged from ten sources, with a clustered map, price and size filters derived live from the data, currency and monthly/yearly toggles, saved and archived lists, and PDF export of a shortlist for agent outreach.

The scraping pipeline

Ten scrapers feed the database, each handling one source's quirks: Inertia JSON APIs, server-rendered HTML, WordPress REST, and static curated rows. Every scraper calls store.upsert_source(...) into a listings table keyed by (source, listing_id) with first_seen and last_seen tracking.

The failure behaviour is the important part. A failed run keeps the previous snapshot, so a source going down or changing its markup never empties the app. A daily update.py orchestrates scrape, image download, merge, and publish into a single-row app_data payload that the Express API serves in one request.

Merging conservatively

The same villa relisted by two agencies gets merged on two signals only:

• Perceptual photo hash, catching the same photograph through any CDN or crop
• Exact coordinate coincidence, within 15 to 40 metres

Never on title or price. The asymmetry is deliberate: showing a duplicate is a minor annoyance, hiding a distinct villa is a real failure. Merged cards then reconcile to the best available value, taking the cheapest price while showing the spread, preferring exact coordinates over approximate ones, filling each field from the richest source, and unioning the photo galleries. Every original listing URL is preserved as an "also on" link, so nothing a source contributed is lost.

The app layer

React hooks derive every slider bound from the data itself rather than hardcoded ranges, so a new source with different price bands does not require a code change. Filters cover area, bedrooms, bathrooms, build size, land size, rental term and type, plus free-text search. Favourites and archive persist locally.

Production runs on Firebase Hosting with thumbnails on Cloud Storage. Adding a new source means one new scraper module, not a schema migration.