Boligscanner — System Architecture
An automated pipeline for screening rental property investments in Norwegian student cities
Boligscanner watches for FINN.no property alerts, scrapes and enriches every listing they point to, runs it through a financial model, and only interrupts me on Slack when a listing actually clears my investment bar. Built solo to replace manually refreshing FINN searches every day — this page is the end-to-end system, written for recruiters and developers who want to see how the pieces actually fit together.
What actually lands on my phone
A real Slack alert from the running system. When a listing clears an investment rule — here, positive cashflow — Boligbot posts the price, capital required, break-even rent, yield, interest-rate tolerance, distance to the nearest university, and the key condition-report findings pulled from the sales prospectus.
- TriggerMatched rule: positive cashflow
- Financial modelCapital required, break-even rent, yield, net cashflow, rate-hike tolerance
- Geo enrichmentNearest university and driving time
- Document analysisCondition grades and risk keywords from the salgsoppgave PDF
One listing, end to end
Watch the system process a single fictional Kristiansand listing in (simulated) real time — from the FINN alert landing in Gmail to the Slack notification. All values are example data.
(Animated walkthrough — requires JavaScript.) In short: a FINN alert email arrives, the listing URL is extracted and deduplicated, the listing is scraped field by field, price and costs feed the financial model while the address is geocoded and routed to the nearest university, the prospectus PDF is downloaded, everything is upserted into SQLite, synced one-way to the dashboard, and — only if it clears the investment rules — posted to Slack. The static diagram below covers the same flow.
finn.no/realestate/homes/000000
A FINN saved-search alert lands. The scanner notices unread mail and opens it automatically.
finn.no/realestate/homes/000000
The URL is lifted out of the email and checked against every link from previous runs before scraping.
The listing is parsed one field at a time, then each value is routed to the module that needs it.
Price and shared costs run through the annuity model; bedroom count drives the 24 000 kr rent estimate. Pure functions — numbers in, numbers out.
The address becomes a coordinate, then a routing call computes the real driving route to the nearest university.
42 pages · condition report, board minutes · saved
The prospectus is downloaded at scrape time, then scanned for condition-report flags worth knowing before a viewing.
Scraped and derived values assemble into one record. A re-scrape lights up and updates this same row.
New and changed rows move one way into the dashboard's own database.
Each rule is checked in turn. Only a listing that clears every one triggers a Slack message.
Five layers, one daily run
Gmail alert in, scored listing and Slack ping out. Each layer has one job and hands off to the next.
Data Ingestion
Gmail APIGmail inbox
Polls Gmail read-only for new FINN saved-search alerts since the last run — cheap to run often, never reprocesses old mail.
URL extraction
Parses each alert's HTML body for finn.no/realestate/... links, then drops any already processed in a prior run.
Scraping & Extraction
requests + BeautifulSoupListing scrape
Fetches each new listing page and pulls price, size, rooms, ownership type, felleskost, build year, and image — with fallbacks for FINN's varying markup.
Sales documents
Locates the sales prospectus either directly from the FINN listing or, if unavailable, automatically follows the broker's website to retrieve it there. The PDF is then downloaded and parsed for condition-report insights that aren't visible on the listing itself.
Enrichment & Calculations
OpenRouteService + pure PythonGeocoding
Converts the address to lat/lng. Failed lookups are retried and logged rather than dropped — every spatial metric depends on this.
Schools & drive time
Finds the nearest schools/universities and computes real driving distance and time — not straight-line — via a routing call.
Financial scoring
Pure calculation layer: yield, cashflow, capital required, loan payment, ROI, break-even rent, and a 0–100 composite score.
Storage
SQLiteNormalize & upsert
Types and number formats are normalized, missing fields made explicit, then upserted keyed on listing ID/URL — re-scraping updates, never duplicates.
Dashboard & Notifications
Flask + Slack WebhooksDashboard sync
New and changed rows are copied into the dashboard's own SQLite copy, upserted on the same key. Read-only — nothing writes back to the scanner.
Slack alert
Listings that clear configured rules (min yield, max price, positive cashflow) trigger a webhook message — the channel stays signal, not noise.
Technical details worth knowing
The parts that make the difference between a script and a system that runs unattended.
Duplicate prevention
Links are de-duplicated against every run before scraping, and storage upserts on listing ID/URL — the same listing never creates a second row.
Sold-listing detection
Listings that stop appearing on re-scrape are marked sold rather than deleted, so price history and time-on-market survive into the Sold view.
Geocoding & drive time
Every address becomes a coordinate, then a real driving time to nearby schools/universities via routing — not a straight-line estimate.
Investment scoring
Yield, cashflow, ROI, break-even rent, and a 0–100 composite score, recomputed live in the dashboard when you tweak interest, equity, or term.
Filtering rules
Configurable minimum yield, maximum price, and positive-cashflow thresholds decide what's worth a Slack interruption versus a quiet scrape.
Document retrieval
Salgsoppgave PDFs are grabbed automatically at scrape time, since revisiting a listing manually later often means it's already sold.
Dashboard synchronization
A one-way sync keyed on listing ID/URL keeps the read-only dashboard current without ever writing back to the scanner's database.
Fallback-tolerant parsing
Each scraped field has a fallback path, since FINN's markup differs across apartments, houses, and plots.
Step by step
Expand any stage for the library, service, and reasoning behind it.
1Inbox monitoring (Gmail API)
A scheduled job authenticates to Gmail with a read-only OAuth2 scope and polls for new messages matching FINN.no's saved-search alert format. Only messages received since the last successful run are fetched, keeping the job cheap and naturally idempotent — re-running it never reprocesses old mail.
2Filtering & link extraction
Each candidate message is checked to confirm it's a genuine FINN alert rather than unrelated mail from the same address. The HTML body is parsed with BeautifulSoup to pull every listing URL out of it, then de-duplicated against links already processed in previous runs.
3Structured scraping of each listing
For every new link, the scanner fetches the FINN listing page and parses out price, address, size (m²), bedroom count, ownership type (eier / andel / aksje), shared monthly costs (felleskost), build year, and the primary image. Each field has a fallback path, since FINN's markup differs across listing types.
4Geocoding the address
The scraped address string is converted into a latitude/longitude pair. Every downstream spatial calculation depends on this coordinate, so failed lookups are retried and logged rather than silently dropped — a bad geocode would quietly poison every metric built on top of it.
5Nearby schools & driving distance
Using the geocoded coordinates, the scanner finds the nearest schools/universities from a reference list and calls a routing service for real driving distance and time — not straight-line — from the property to each one. This is what the dashboard's "university distance" column and filter are built on.
6Financial modeling
A pure calculation layer — no network or database calls — takes price, estimated rent, and assumptions (interest rate, equity %, loan term) and produces gross yield, monthly cashflow after mortgage, capital required, loan payment, ROI, break-even rent, and a 0–100 composite score. The dashboard's own metrics.py recomputes the same formulas live when you change assumptions in the UI.
7Sales-document retrieval
When a listing has an associated prospectus (salgsoppgave), the scanner locates the document link and downloads the PDF — full condition reports, board minutes, and building history the FINN page itself doesn't carry.
8Normalization & SQLite storage
Every scraped and derived field passes through normalization — consistent types, Norwegian-formatted numbers parsed to floats, missing fields made explicit None — before being upserted into SQLite, keyed by listing ID/URL so re-scraping updates in place instead of duplicating.
9Dashboard synchronization
The dashboard is a separate deployment with its own copy of the database, so a sync step copies new and changed rows from the scanner's SQLite file into the dashboard's, upserting on the same natural key used in normalization. Nothing here writes back — the dashboard only ever reads.
10Slack notifications on rule match
Not every scraped listing is worth an interruption. After metrics are computed, each listing is checked against configured investment rules (minimum yield, maximum price, positive cashflow, etc.). Only listings that clear the bar trigger a Slack incoming webhook message.
11Autonomous scheduled execution
Steps 1–10 run inside a Docker container as a background service, with an internal scheduler triggering a full scan once a day. Containerizing means it isn't tied to a laptop being on, and the exact runtime is reproducible on any host it's deployed to.
Technology stack
What's used where, and why.
Python
End to end
One language across scraping, geo calculations, financial modeling, and the dashboard server.
Flask
Dashboard server
Server-rendered HTML, no JS build step — right-sized for a small, read-mostly, single-user dashboard.
SQLite
Storage, both sides
One writer, a couple of readers, ~1,500 rows — a single file, zero server process to run or patch.
Gmail API
Ingestion trigger
FINN has no public listings API; reusing its own saved-search alerts avoids crawling search-result pages.
BeautifulSoup
HTML parsing
Used on both email bodies and listing pages — forgiving of inconsistent real-world markup.
OpenRouteService
Geocoding & routing
Geocoding and driving-distance routing behind one API, free tier sized for a once-a-day scanner.
Slack Webhooks
Notifications
One HTTP POST to an incoming webhook — no bot, no OAuth app, no new channel to check.
Docker
Runtime & scheduling
Packages the Python version and dependencies so the scanner runs unattended, on schedule, anywhere.
GitHub
Version control
Two repos by design — the dashboard is shareable and secret-free; the scanner keeps API keys private.
Architecture decisions
The choices most likely to get questioned, and why they hold up.
Why email-driven ingestion instead of scraping FINN directly?
FINN has no public listings API, and hitting its search-result pages directly on a schedule is more fragile than it looks — pagination, markup, and anti-scraping surface all shift over time. Reusing FINN's own saved-search email alerts turns "find new listings" into "read new mail I already receive": one page fetched per link the alert already gave the scanner, no crawling required.
Why was SQLite sufficient?
The dataset is small — around 1,500 tracked listings, a single writer (the scanner), and a couple of readers (the dashboard, ad-hoc queries). That's squarely inside SQLite's comfort zone: no server process to run, secure, or pay for, and the entire database is one file, trivial to back up or open directly. A client-server database would add real operational surface for no benefit at this scale.
Why Docker?
The scanner has to run unattended, once a day, without a laptop being open. Docker bundles the Python version and system dependencies into one image, so the environment that works in development is exactly what runs on schedule — movable to a different host or restarted cleanly without reinstalling anything by hand.
How is this designed to be modular and extensible?
Each stage — ingestion, scraping, geocoding, metrics, storage, notification — is a separate module with a narrow interface. Financial modeling is pure functions with no I/O: numbers in, numbers out (the dashboard's own metrics.py follows the identical pattern, and app.py/database.py/metrics.py stay strictly separated). That means the listing source could move off Gmail, or Slack could be swapped for email or Discord, without touching the modules on either side of the change.