Comments (3)
It seems the reddit api uses before
and after
options that take a coded Id in listings, e.g. for comments the Id of the comment is prefixed with t1
, such as t1_e3ykxbc
.
To do this I'd probably need to save a metadata file locally that stores the Ids of the latest data, then subsequent runs of the tool would first read from the metadata file if it exists and use the Ids to fetch the latest content.
Also would need a new CLI option to enable this feature, options might be:
-p, --persist
-c, --continue
-c, --cron
-c, --config <filename>
-l, --latest
Any ideas?
from orca.
I like either "cron" or "latest". You could also use "-r, --resume".
from orca.
I went with an --only-latest
flag for this feature, its available now.
Orca now generates an orca.config.js
file in the CWD.
- when data is downloaded the latest entry for each data type is recorded, and the data is written to the file system
- if using the
--only-latest
flag the last entry is used when fetching data from the Reddit API, the latest data will overwrite the files previously written to the file system with only the latest data. - if not using the
--only-latest
all data is downloaded and the data files are overwritten on the file system
So now just run the below command in a cron job to get only the latest data since the last download.
npx @mortond/orca --access-token=70162531-eWBggyup_FAKE_Usdf1cz7u-G9pM_dhrVf3g --only-latest
Make sure to copy or process the data files written to the file system before running Orca multiple times.
from orca.
Related Issues (7)
- No comment data before ~2018 HOT 6
- Backup a list of my current subscriptions
- Allow refresh token use for long running jobs HOT 1
- Set default output format to Markdown, with options for .txt, .yml, .json, etc HOT 1
- Is it possible to organize the destination files based on the subreddit in question? HOT 1
- For saved content, is it possible to fetch the media rather than just the post link? HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from orca.