jarulsamy / twitter-archive

A CLI tool to download media from bookmarked tweets.

Twitter-Archive

A CLI Python application to download all media (and hopefully more) from bookmarked tweets (for now). Eventually I hope to make this a general archive utility for Twitter, allowing users to download/archive all kinds of tweets.

Originally, before the V2 Twitter API, this app used Selenium to try and scrape the contents of a users bookmarks page. Now, since the release of the V2 API, the application has been rewritten. This new version is much faster and more robust.

---

Installation and Setup

Installation

Twitter-Archive can be installed with pip

$ pip install twitter-archive

Alternatively, you can clone this repository and install from the repository instead of from PyPi.

$ git clone https://github.com/jarulsamy/Twitter-Archive
$ cd Twitter-Archive
$ pip install .

To properly authenticate with the Twitter API, you will have to create a developer application. This will provide you with a client ID and client secret.

Twitter Developer App Setup

Refer to these docs to setup your Twitter developer account and project.

Authentication and Usage

There are several options for passing the client ID and client secret to the application. Only one of the following is required.

Option 1: Environment Variables

Set the relevant environment variables as so:

$ export TWITTER_ARCHIVE_CLIENT_ID="YOUR_CLIENT_ID_HERE"
$ export TWITTER_ARCHIVE_CLIENT_SECRET="YOUR_CLIENT_SECRET_HERE"

Now you can use the application until you restart your shell.

Option 2: Dotenv Variables

Alternatively to environment variables, you can save your tokens in a .env file in your current working directory. This file is automatically read and loaded by twitter-archive at runtime to load the necessary variables. An example .env file would look like this:

TWITTER_ARCHIVE_CLIENT_ID=YOUR_CLIENT_ID_HERE
TWITTER_ARCHIVE_CLIENT_SECRET=YOUR_CLIENT_SECRET_HERE

Option 3: CLI Flags

The tokens can also be passed in as CLI flags, but this is generally discouraged as most shells keep a history of commands entered, and this obviously risks leaking your keys. For example:

$ twitter-archive --client-id="YOUR_CLIENT_ID_HERE" --client-secret="YOUR_CLIENT_SECRET_HERE"

Usage

You can then invoke the app with:

$ twitter-archive

By default, the app will print a URL to prompt the user to authorize the application with Twitters official APIs. Once you navigate to that link and login with Twitter, the app will fetch a manifest of all the bookmarked tweets and begin saving any photos/videos to disk.

You can view the built-in CLI help menu for more info:

$ twitter-archive --help
Usage: twitter-archive [--client-id ID] [--client-secret ID] [--headless] [--no-clobber] [--num-download-threads N] [--quiet]
                       [-o FILE] [-i FILE | -m FILE] [-v] [--version] [--help]

A CLI Tool to archive tweets v0.0.7

Options:
  --client-id ID        Specify the client ID. (default: None)
  --client-secret ID    Specify the client ID. (default: None)
  --headless            Don't use interactive authentication. (default: False)
  --no-clobber          Don't redownload/overwrite existing media. (default: False)
  --num-download-threads N
                        Number of threads to use while downloading media. (default: 8)
  --quiet               Disable download progress bars (default: False)
  -o FILE, --media-output FILE
                        Path to output downloaded media. (default: media)
  -i FILE, --manifest-input FILE
                        Use an existing manifest and download all media. (default: None)
  -m FILE, --manifest-output FILE
                        Path to output bookmark manifest. (default: bookmark-manifest.json)
  -v, --verbose
  --version             show program's version number and exit
  --help                Show this help message ane exit.

Acknowledgment

The Twitter developer team did an excellent job on the new APIs. The new APIs are substantially more intuitive and allow us to interact with many more features of Twitter. While it did take two years, the openness, transparency, and attention to feedback is much appreciated!

The relevant forum post is available here.

twitter-archive's Issues

Server not working

Hi,

I have followed all the commands and installed all the requirements, but after I run the server, I get the following error:

Could you please help on resolving the error, the server was running on 8080

I also get the twitter APP side error like this

Any hep would be appreciated

Setup automatic documentation building.

Add options for disabling/configuring the download thread pool.

Setup automatic builds on push and deployment.

Handle rename from 'twitter-bookmark-downloader' to 'Twitter-Archive'

Add dry run CLI flag.

Get some demo gifs or screenshots.

Add more tests for the internals.

• OAuthRequestHandler
• Consider mocking the entire authentication flow. For now that seems a little overkill.
• get_bookmarks()
  • Mock the API calls, and ensure the resulting manifest is in the expected format.
• download_tweets()
  • Given a manifest, mock HTTP endpoints, and ensure each of them is given a request.

twitter-archive: command not found

I'm quite new to all these things so I'm not sure what I did wrong. I have python 3.9 installed and everything in the requirement.txt. and I followed all the steps in the readme. But when I try to invoke the app it tells me that the command is not found.

Add no-clobber CLI flag.

Twitter Something Went Wrong

I couldn't figure out what exactly I did wrong. I couldn't figure out how to grant the scopes from the readme because I don't know where to find them, and I don't know what URL to put for the "website URL" in the user authentication settings.

not downloading

i can't seem to get this to work. any ideas as to why? when i run the script, i get like to images and that's it.

Add quiet CLI flag.

How to authenticate on Windows

From what I can tell, all the commands are for Linux, and I've been trying for a while now to figure out how I can authenticate and actually use the damn thing with no luck. If the commands could be in Windows format as well then I'd appreciate it, or if somebody who got it working on Windows explain how the hell I can do it.

"TypeError: 'NoneType' object is not iterable"

Don't know any coding stuff so this could very well be easily fixable, but I get this whenever I try to run the application:

For context, I use Windows 11, Powershell and Firefox. It has worked in the past, but I always have to reinstall the whole thing to get it working. I've also tried using "py -m twitter-archive" and everything but none of them work, so I've resorted to just using the .exe, as can be seen on the screenshot.

I also usually put the client ID and secret as commands right after, but currently get the same error message no matter what.
Again, apologies if this is an easily fixable and obvious issue.

Twitter developer app instructions are ambiguous.

Currently the instructions for creating a new twitter developer account/project/app are somewhat vague and difficult to follow. Updating these instructions (maybe to their own dedicated file) would make using this app much easier.