Comments (8)
Ah, got it now. Actually, someone brought up a similar request in #19 (and I think elsewhere, but I can't find it now).
I'll look at doing two things:
- Backfill users on your lists
- fetch replies to posts on your lists
from fedifetcher.
This is good to know, esp. point 3. Thanks for explaining that to me. I am the only user on my self-hosted server and your script makes my experience so much better–no blocking in robots.txt. My favorite feature is backfilling users. It's indispensable. I've used it since you first released it. Thanks for the tool and your continued work to make it better.
from fedifetcher.
Let me know how you get on with this as well, especially if you use lists extensively. As I don't personally use them at all, I find it hard to predict exactly how well it'll work.
If need be it shouldn't be hard to limit it to a specific list of lists to process.
from fedifetcher.
I will let you know. I am going to upgrade to this version and do the first run manually later today.
from fedifetcher.
Thanks for the suggestion @csolisr
I must admit I don't think I quite understand what you are trying to achieve here.
Say, I got the id of a user, e.g. yourself, this endpoint will return the list of lists that I have added you to.
What would I want to back fill here?
from fedifetcher.
Sorry for the delay, what I meant here is to do the following:
- For a given user, check all of the lists
- For each list, check all the accounts it includes
- Backfill each account listed on each of those lists
from fedifetcher.
First, I like this. Unless I am misunderstanding what this does, this could be "dangerous" for me, personally. There are a few lists I would like to backfill, but I assign everyone to a list based on primary subject matter so, it would backfill like crazy. If there were a way to configure which list ID to fetch... that would be great.
I don't know how many people use lists, but I know back in the early days of all the influx of Twitter users after Musk bought it, lists were being pushed as a way to curate our non-algorithmic experience.
Say you have 500 users in different lists, what action does it do every time it runs? Backfill and fetch those 500 users each time?
from fedifetcher.
Say you have 500 users in different lists, what action does it do every time it runs? Backfill and fetch those 500 users each time?
No: the script remembers users it has backfilled, and won't re-backfill them. Especially with the backoff implemented in #124, I would judge the risk very low. What it would do is:
- Get a list of the user's lists (actually a max of 99 - an arbitrary hardcoded limit, which may change)
- For each of those 99 lists:
- Get a list of the last 100 (or whatever the user has configured) posts in this list, and fetch new context for each post in that list according to the rules specified in #123.
- Get a list of the last 10 (or whatever the user has configured) accounts added to the list, filter those it has already processed in a previous run, and for the remaining fetch the latest posts.
As such, after the first run the impact should be negligible.
If you don't want users on your server to be able to use this, then you could always block the FediFetcher
user agent from the /api/v1/lists
endpoint, using robots.txt
, nginx or your firewall: that would prevent FediFetcher from getting past step 1.
from fedifetcher.
Related Issues (20)
- Add option not to include timestamp in logs HOT 7
- FediFetcher fails on checking followings with 401 HOT 4
- [Question] Is it necessary for Akkoma? HOT 2
- 7.0.5 breaks logging when launched as a cronjob HOT 4
- blocks and run FediFetcher as Admin HOT 5
- Implement more detailed UA HOT 3
- ‘Rate-Limit’ FediFetcher HOT 13
- Robots.txt parsing might not be 100% correct HOT 5
- Robots.txt is fetched for every request made. HOT 1
- 7.1.2 Crashes due to an error in copy-pasted code HOT 1
- from-lists causes TypeError HOT 3
- Add custom extension to robots.txt cache HOT 1
- Job failed: 'dict' object is not callable HOT 6
- Backfill profiles while fetching list context
- Check out any other software that might be worth supporting HOT 21
- FediFetcher crashes when using multiple tokens (admin:read:accounts scope) HOT 4
- Handling of threads.net HOT 1
- Should I set "Application Website" or "Redirect URI" in the Mastodon app when using GitHub actions? HOT 1
- getAllRepliesToKnownPosts: All jobs have failed HOT 4
- Seems to be fetching lots of replies not connected to posts in my home feed HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from fedifetcher.