mrmartineau / cloudflare-worker-scraper

Page Metadata Scraper with Cloudflare workers

Hi there 👋

My name is Zander

I make things for the web.

Note

When I'm not getting paid to work, I work on one of my many side-projects like Otter, my Code Notes project, Journal or my various Raycast extensions.

• 🚀 I'm currently working for Dare & I've worked for a wide variety of startups, agencies and companies like Fathom, Utopia Music, Babylon Health, Digital Futures, Heights, Curve and FairFX amongst others..
• 🌱 Recently, I've been working with A.I. on my Journal & Otter, Next.js 14 on Otter, Astro for my new website, Eleventy (on my Code Notes project), Remix (for Utopia Music), SvelteKit (for my journalling app) and PostgreSQL (and auth with Supabase) and a whole lot more
• 💬 Ask me about design systems or what I think of React
• 📫 How to reach me: Mastodon or email
• 😄 Pronouns: he/him/his
• ⚡ Fun fact: I used to be an assistant cameraman and worked on many TV shows, documentaries and one movie

Interests and hobbies

• 🏌️‍♂️ Golf
• 🚵‍♂️ Mountain biking
• Making 🍕 Pizza & 🍖🔥 BBQ
• 🍿 Movies
• 🥁 Drumming
• 💖 Salted Caramel

Tip

Feel free to say hello, fork my code or read my blog or checkout my blog/portfolio at zander.wtf

cloudflare-worker-scraper's Issues

Improve link type checker

The link type checker is fairly "unintelligent" at the moment, it looks at the url/path and infers based on that. For example:

• https://www.youtube.com/watch?v=ctEksNz7tqg would be video
• something.jpg would be image

All the rules are in the types.ts file, but I'm sure they can be improved.

Does anyone have a better solution?

JSON LD on Youtube videos needs improving

The 1st two JSON LD selectors have been specifically added to get more accurate data for YouTube videos, but it seems HTMLRewriter is not able to parse them. I think this might be due to the JSON LD script being within a web component..

The selectors are:

[
  {
    selector:
      '#content #microformat script[type="application/ld+json"]',
  },
  {
    selector:
      'ytd-player-microformat-renderer script[type="application/ld+json"]',
  },
  {
    selector: 'script[type="application/ld+json"]',
  },
]

Are there better set of selectors that can be used to get this information?

[enhancement] Add option to delay & add proxy rotation

This is the first time in a long time I saw code being structred and written so well, that it even makes sense just by looking at it. Simply awesome how you managed to do that & its helping me a lot to plan & structure my own stuff.

On a side note: you might like to check the awesomeness from https://github.com/AutomaApp/automa & how they automate stuff in the browser. Not so much for coding but for possible workaorunds & fallback scenarios.

[Suggestion to take your code provided even further]
The following is a braindump of features that would make your scraper even more awesome. Consider it being wishful thinking and not a complaint about missing stuff.

1. Allow for finding sitemap.xml and convert to json

2. Allow using json path to filter results from previous step to e.g. only follow ./products/*

3. Scrape the metadata from previous step ('products' in this example)

   3.1 Additional feature enhancement to this: allow following a specific URL from within that product page to scrape additional data (e.g. if the parent prodcut page holds a 'manufactorer' tag or class, follow that link to provide additional meta data by following the related manufactorer link provided to get e.g. the SKU or whatever from that page too in a single attempt)

4. Add an option to set a threshold to randomly delay requests
   Context: Reduce being spotted as robot + Avoid IP adress ban for too many requests from same IP adress origin within a timeframe

5. Alternatively to previous step or even as addition: SloMo mode (e.g. set a timeframe in which the crawler should operate (e.g. parse list over a time period of 12 hours)

6. Allow IP adress rotation (no idea how this can be achieved) or to create/specify local proxy

7. Allow to set / update / delete cookie prior parsing (e.g. to log in or fake fresh user or ....

Knowing ou likely got other stuff to do, I still hope you might consider the one or the other or at least think about way to allow others creating "modules" that plug-in to what you shared already.