- ๐ต๏ธโโ๏ธ Scrapes housing data from online sources (fx. boligportal)
- ๐จโ๐ง Data pipelines built using Puppeteer and TypeScript
- ๐ง Analysis done in Dash, Plotly and Python to find houses that are especially cheap for a given size / location
To start the pipeline and collect data run:
yarn start
This will append the collected housing data to the out.nljson.dump
file.
To visualise this data run:
yarn graph
and navigate to http://127.0.0.1:8050/
to view and interact with the data.
- We collect data using Puppeteer.js to navigate to different webpages and scrape data from them, and then continue to the next page.
- This data is turned into a stream which is then processed to clean up each record - making it both more easily readable and more easily consumable.
- The stream of JSON records is then piped to
out.nljson.dump
and each record is separated by newlines (both source and destination behaviours can be modified insidesrc/index.ts
) - We then use pandas dataframes to further clean the data (mostly eliminating data not needed for the visualisation, and removing duplicate records). Then we use Plotly to display this data in a quick/simple way - followed by using Dash to add interactivity (i.e. creating links to the original posting)