BaliProp
Rent and buy listings from ten Bali property sources, merged onto one searchable map
13,246listings in the latest data pull
The problem
Finding a long-term rental in Bali means checking a dozen agency sites with inconsistent formats, duplicate listings, and no shared filters. Prices appear in three currencies and two rental periods. The same villa is listed by four agencies at four prices. I built this to solve my own search and kept going.
What I built
BaliProp is a React app over a Python scraping pipeline and PostgreSQL. It serves 13,246 listings merged from ten sources, with a clustered map, price and size filters derived live from the data, currency and monthly/yearly toggles, saved and archived lists, and PDF export of a shortlist for agent outreach.
The scraping pipeline
Ten scrapers feed one database, each built around one source's quirks: clean data feeds, messy HTML pages, WordPress sites and a few hand-picked listings. Each one tracks when a listing first appeared and when it was last seen.
The failure behaviour is the important part. A failed run keeps the previous snapshot, so a source going down or changing its markup never empties the app. A daily update.py orchestrates scrape, image download, merge, and publish into a single-row app_data payload that the Express API serves in one request.
Merging conservatively
The same villa relisted by two agencies gets merged on two signals only:
- Photo fingerprinting, recognising the same villa photo even after a site crops or resizes it
- Exact coordinate coincidence, within 15 to 40 metres
Never on title or price. The asymmetry is deliberate: showing a duplicate is a minor annoyance, hiding a distinct villa is a real failure. Merged cards then reconcile to the best available value, taking the cheapest price while showing the spread, preferring exact coordinates over approximate ones, filling each field from the richest source, and unioning the photo galleries. Every original listing URL is preserved as an "also on" link, so nothing a source contributed is lost.
The app layer
Price and size sliders adjust to whatever the data contains, so adding a new source with different ranges does not mean rebuilding the filters. You can filter by area, bedrooms, bathrooms, build size, land size, rental term and type, or search in plain text. Favourites and archived listings stay on your device.
Production runs on Firebase Hosting with thumbnails on Cloud Storage. Adding a new source means one new scraper module, not a schema migration.
