darvester / darvester

Powerful Discord OSINT toolkit written in Python and Dart/Flutter

Darvester GEN2

🖥️ PoC Discord user and guild information harvester

Darvester aims to provide safe Discord OSINT harvesting, abiding by sane rate limiting and providing automated processing - now written in Dart/Flutter

Repo Notice

Currently, there is no activity and much of this code is likely outdated. The main task at hand is making nyxx-self feature complete, comparable to the discord-py.self library. Aside from that, the ported harvester loop in the gen2 branch is incomplete and is waiting for nyxx-self. Although there is much to work on, the frontend supports importing an SQLite database populated with the Python branch.

✨ Features

• Rate-limit/soft ban avoidance
• Automated processing
• Flexible configuration
• Utilization of the Git version control system to provide chronological data
• Detailed logging
• and more

💽 Data logged for each user

• Profile created date, and first seen date
• Username and discriminator
• User ID (or Snowflake)
• Bio/about me
• Connected accounts (reddit, YouTube, Facebook, etc.)
• Public Discord flags (Discord Staff, Early Bot Developer, Certified Mod, etc.)
• Avatar URL
• Status/Activity ("Playing", "Listening to", etc.)
• Nitro tier

💾 Data logged for each guild

• Name
• Icon URL
• Owner name and ID
• Splash URL
• Member count
• Description
• Features (thread length, community, etc.)
• Nitro tier

⚠️ Disclaimer

Using this tool, you agree not to hold the contributors and developers accountable for any damages that may occur. This tool violates Discord terms of service and may result in your access to Discord services terminated.

What does Darvester do?

Darvester is meant to be the all-in-one solution for open-source intelligence on the Discord platform. With the recent GEN2 releases, Darvester now provides an easy-to-use frontend UI along with new features such as multiple harvesting instances, or Isolates, for individual tokens, a refreshed Material 3 UI, and an all-in-one packaged app - no more Python dependencies and virtual environments!

Darvester is built on a custom fork of nyxx named nyxx-self, maintained by the author of Darvester

darvester's Issues

[gen1] RuntimeError on interrupt

Describe the bug

Initializing 100%|███████████████████████████████████████████████████████████████████████████| 4/4 [00:30<00:00, 0.13/s]
Guilds   5%|███▎                                                                    |  1/22 [00:28<09:39, 0.04 guilds/s]Traceback (most recent call last):
  File "C:\Python310\lib\asyncio\proactor_events.py", line 116, in __del__
    self.close()
  File "C:\Python310\lib\asyncio\proactor_events.py", line 108, in close
    self._loop.call_soon(self._call_connection_lost, None)
  File "C:\Python310\lib\asyncio\base_events.py", line 750, in call_soon
    self._check_closed()
  File "C:\Python310\lib\asyncio\base_events.py", line 515, in _check_closed
    raise RuntimeError('Event loop is closed')
RuntimeError: Event loop is closed
Exception ignored in: <function _ProactorBasePipeTransport.__del__ at 0x000002673EE5E560>
Traceback (most recent call last):
  File "C:\Python310\lib\asyncio\proactor_events.py", line 116, in __del__
    self.close()
  File "C:\Python310\lib\asyncio\proactor_events.py", line 108, in close
    self._loop.call_soon(self._call_connection_lost, None)
  File "C:\Python310\lib\asyncio\base_events.py", line 750, in call_soon
    self._check_closed()
  File "C:\Python310\lib\asyncio\base_events.py", line 515, in _check_closed
    raise RuntimeError('Event loop is closed')
RuntimeError: Event loop is closed
Exception ignored in: <function _ProactorBasePipeTransport.__del__ at 0x000002673EE5E560>
Traceback (most recent call last):
  File "C:\Python310\lib\asyncio\proactor_events.py", line 116, in __del__
  File "C:\Python310\lib\asyncio\proactor_events.py", line 108, in close
  File "C:\Python310\lib\asyncio\base_events.py", line 750, in call_soon
  File "C:\Python310\lib\asyncio\base_events.py", line 515, in _check_closed
RuntimeError: Event loop is closed

To Reproduce
Ctrl+C while running in harvest loop, not on delay

Desktop (please complete the following information):

• OS: [e.g. iOS] Windows 10
• Version [e.g. 22] Python 3.10

Additional context
Add any other context about the problem here.

Handle users found in multiple guilds (mutual_guilds)

• From get_guild do joined_at for member

Log user "about me" (may require selfcord modifications)

SQL updating is not consistent

When a row already exists, only the data gets updated and not the added rows

[REQUEST] Harvest user's banner

I may be senile, but I don't see it in the schema

Improve API requesting to minimize ban or Cloudflare block

I will improve you redme.md file

Is your feature request related to a problem? Please describe.
I will improve you redme.md file

Describe the solution you'd like
More clear redme.md file. help users to understand the project better

Describe alternatives you've considered
By adding the design element and emoji

Additional context
NA

[Enhancement] Use .env file instead of cfg.py for better security

Use instead of cfg.py the .env file for better security. I can work on it if you desired.

[Enhancement] Scrape member trough Preview functionality

Enable scraping trough the "Explore Public Servers" tab, when clicked on a server there is a preview of the server and the members list is still visible so it can be scraped without risking to much for multiple server joins. Maybe it can automatically scrape all server with a specific keyword

Thread BotStatus

Thread BotStatus because it takes a long time to update the presence, and it's blocking code

TODO Thread

List of tasks to improve project

• #3
• #5
• #6
• #7
• #8
• Implement caching (part of #8)
• #12
• #14
• #17
• #20
• #22
• #30

Planned

• #13
• #16
• #21
• #31

Will be updated

Revamp the log messages

They are extremely messy

Under Tracks, visit Github links are not working

Describe the bug
Under Tracks, visit Github links are not working

To Reproduce
Steps to reproduce the behavior:

1. Go to tracks card
2. Click on Visit Github

Expected behavior
Should redirect to corresponding link

Screenshots
If applicable, add screenshots to help explain your problem.

Desktop (please complete the following information):

• OS: WIndows
• Browser Brave
• Version 1.44

[GEN2] Address code signing for releases (#47)

https://www.electron.build/code-signing

[BUG] IDs in JSON surpass integer limits

Describe the bug
When parsing the database JSON in JS, JS native JSON parser does not convert Snowflake IDs properly to BigInt/BigNumber's

Suggestion: Store them as strings, if this isn't detrimental to the prior databases

Menu system - listen for events while doing operations

- Implement a multithreaded application that handles commands and background operations at the same time, like Bettercap (image above)

Add support for version control

Implement a Git-like system. When a users info changes after compared against the database, commit it

[GEN2] Implement Settings page

feat: Add ability to ignore guilds as names alongside ID's

Is your feature request related to a problem? Please describe.
N/A

Describe the solution you'd like

guild: discord.Guild
for ignored_guild in ignored_guilds:
	if guild.name.lower() in str(ignored_guild).lower() or guild.id == ignored_guild:
		continue

i hate this project

discord api sucks

[GEN1] Individual columns are not populating correctly

Columns such as owner, mutual_guilds, or other columns that contain deserialized data are not populated. This excludes the main data column and other columns that hold simple strings or numbers. Lack of testing with the recent deserializer refactor is probably the blame here

Support more database output protocols

Such as MySQL and PostgreSQL

[GEN2] Implement the Drift ORM

Currently, Darvester is using Sqflite. We may be able to minimize the boilerplate code and even allow for more efficient and secure code by using an ORM.

Drift ORM:

• https://pub.dev/packages/drift
• https://drift.simonbinder.eu/docs/getting-started/

Add threading to allow logging with multiple user tokens

[GEN2] Implement Drift's Isolate API

https://drift.simonbinder.eu/docs/advanced-features/isolates/

The code works on the main isolate and will display Guilds and Users. When spawning a harvester isolate, the code will throw the exception above.

This is the believed cause of this error:

[ERROR:flutter/runtime/dart_isolate.cc(1098)] Unhandled exception:
Bad state: The BackgroundIsolateBinaryMessenger.instance value is invalid until BackgroundIsolateBinaryMessenger.ensureInitialized is executed.
#0      BackgroundIsolateBinaryMessenger.instance (package:flutter/src/services/_background_isolate_binary_messenger_io.dart:27:7)
#1      _findBinaryMessenger (package:flutter/src/services/platform_channel.dart:135:42)
#2      MethodChannel.binaryMessenger (package:flutter/src/services/platform_channel.dart:273:56)
#3      MethodChannel._invokeMethod (package:flutter/src/services/platform_channel.dart:308:15)
#4      MethodChannel.invokeMethod (package:flutter/src/services/platform_channel.dart:491:12)
#5      MethodChannel.invokeMapMethod (package:flutter/src/services/platform_channel.dart:518:49)
#6      MethodChannelSharedPreferencesStore.getAll (package:shared_preferences_platform_interface/method_channel_shared_preferences.dart:60:28)
#7      SharedPreferences._getSharedPreferencesMap (package:shared_preferences/shared_preferences.dart:229:38)
#8      SharedPreferences.getInstance (package:shared_preferences/shared_preferences.dart:67:19)
#9      Preferences.getString (package:darvester/util.dart:104:63)
#10     _openConnection.<anonymous closure> (package:darvester/database.dart:113:54)
#11     new Future.sync (dart:async/future.dart:302:31)
#12     LazyDatabase._awaitOpened (package:drift/src/utils/lazy_database.dart:47:14)
#13     LazyDatabase.ensureOpen (package:drift/src/utils/lazy_database.dart:61:12)
#14     DatabaseConnectionUser.doWhenOpened (package:drift/src/runtime/api/connection_user.dart:162:21)
#15     CustomSelectStatement._executeRaw (package:drift/src/runtime/query_builder/statements/select/custom_select.dart:51:16)
#16     CustomSelectStatement.get (package:drift/src/runtime/query_builder/statements/select/custom_select.dart:37:12)
#17     Selectable.getSingle (package:drift/src/runtime/query_builder/statements/query.dart:230:19)
#18     DarvesterDatabase.getTableCount (package:darvester/database.dart:91:82)
#19     DarvesterDatabase.upsertGuild (package:darvester/database.dart:103:16)
#20     Harvester.loop (package:darvester/darvester/core.dart:169:10)
<asynchronous suspension>

(BAN-B608) Risk of possible SQL injection vector through string-based query construction

Description

An SQL injection attack consists of insertion or “injection” of a SQL query via the input data given to an application. It is a very common attack vector. Unless care is taken to sanitize and control the input data when building such SQL statement strings, an injection attack becomes possible.

Occurrences

There are 8 occurrences of this issue in the repository.

See all occurrences on DeepSource → deepsource.io/gh/V3ntus/darvester/issue/BAN-B608/occurrences/

[GEN2] [BUG] Cross-platform issues

Improper paths

Describe the bug

• Attempting to start the core through the process manager fails
• pip environments are not universal (for example Windows binaries within the environment are in .env\Scrips\ and not .env\bin\

[GEN2] Electron-based first run flow

• Paged system
  _{Checked means done. Commits will be linked here when able}
• Check for supported Python version
• Setup python venv environment
• Install python requirements for both the API and harvester
• Implement a process manager (pm2 maybe, as it contains a NodeJS API)

[GEN1] `owner` is not valid JSON after sanitization

The sanitization function (_generate_values) in sqlutil.py does not output valid JSON. The function replaces double quotes with single quotes which will not deserialize

Refactor code into separate modules

[CI/CD] Configure flutter builds and artifacts uploads

[GEN2] OTA Updates

Handling OTA updates adds to the ease-of-use philosophy of this project:
https://www.electronjs.org/docs/latest/tutorial/updates

Add new method for token discord

Hello,

I have write new method to get the Discord Token, because with the old one I never succeeded, but with this one I did, so both are different, so I think it's good to put them on the wiki

## - Acquiring your Disord user token, second method

1. Go to [discord.com](https://discord.com/) with **Google Chrome** or **Firefox**

2. Open Developer Tools Panel (Windows: *Ctrl + Shift + i* - Mac: *Cmd + Option + *); Go to Network Tab

3. Reload you're cache ; Windows: *Ctrl + F5* - Mac: *Cmd + Maj + R*

4. Search **/api** in the *Filter* toolbox; click on **library**

5. Search the **Headers Tab**; Now, search **authorization**.

6. Right + clic on the *token*, copy the value, it's you're token !

7. Copy and define it in `cfg.py` in the root of this project

![Example for Discord Token with Google Chrome](https://i.ibb.co/xDSW1gr/Sans-titre-1.jpg)

Log user status/presence

(#20) Add data columns alongside JSON `data` column

[BAN] 429 Ban Report - Overnight usage

Date and time

No response

How long was the tool open for?

Overnight - 12 hours?

Were you using the harvesting account while the script was active? If so, what were you doing?

n/a

Log output:

No response

Version

main

Additional info:

No response

[GEN2] Remove `darvester-api` in favor of NJS `sqlite3` bindings

To reduce overhead and codebase complexity, darvester-api can be removed in favor of implementing database actions via node.js sqlite3 package

This also brings the question of API removal. Serving an API can be implemented in the future with Express which will keep API server functionality for use with developer applications

(PY-D0003) Missing module/function docstring

Description

The module/function docstring is missing. If you want to ignore this, you can configure this in the .deepsource.toml file. Please refer to docs for available options.

Occurrences

There are 28 occurrences of this issue in the repository.

See all occurrences on DeepSource → deepsource.io/gh/V3ntus/darvester/issue/PY-D0003/occurrences/

[GEN1] refactor the error handler on gateway login

Currently, the error handler only returns a 429 since at the time, that was the most common API problem. Now, this is improper error handling and should now print proper gateway error codes and message

[GEN2] [BUG] Manager status not updated properly

Describe the bug
Manager processes statuses are not being updated after start.

To Reproduce
Steps to reproduce the behavior:

1. Open Manager
2. Start a process
3. Status stays offline but is clearly working in PM2

darvester/src/components/Manager.jsx

Lines 50 to 72 in a76f0ce

	window.electronAPI.onGetStatus(
	(_event, status) => {
	if (status.error) {
	console.log(status.error);
	return
	}
	status.message.filter((proc) => {
	return ["davester-api", "darvester-core"].includes(proc.name);
	}).forEach(element => {
	if (element.name === "darvester-core") {
	setCoreStatus({
	changing: ["stopping", "launching"].includes(element.pm2_env.status),
	status: parseStatus(status)
	});
	} else if (element.name === "darvester-api") {
	setapiStatus({
	changing: ["stopping", "launching"].includes(element.pm2_env.status),
	status: parseStatus(status)
	});
	}
	});
	}
	);

[GEN2] Enhance user experience in mobile devices

Describe the bug
When trying to test the page on small screens I get a general error in the content

To Reproduce
Steps to reproduce the behavior:

1. Go to 'http://darvester.gladiusso.com/'
2. Click on 'dev tools'
3. slide the bar
4. See error

Expected behavior
A clear and concise description of what you expected to happen.

Screenshots

Desktop (please complete the following information):

• OS: [Windows 10]
• Browser [e.g. chrome]
• Version [e.g. 22]

Smartphone (please complete the following information):

• Device: [e.g. iPhone6]
• OS: [e.g. iOS8.1]
• Browser [e.g. stock browser, safari]
• Version [e.g. 22]

Additional context
Add any other context about the problem here.

Reorganize codebase

Refactoring and reorganization of code is needed. Codebase is very convoluted

[GEN2] Darvester Core - Configuration Wizard

Guide the user in configuring Darvester Core. (token, etc.)

Will probably be stored as JSON (?) then parsed in the Python

Move to async loops instead of threads?

Darvester 2.0 - Paradigm

Darvester 2.0, the next iteration of Darvester

Darvester 2.0 will feature a more streamlined and bundled approach. For every release, the package should contain everything needed to harvest, view, and alter harvested data easily with minimal effort and configuration.

Languages

Python

darvester-api is already written in Python FastAPI.

darvester, again, is already written in Python and is considered almost completely feature complete and stable.

React.js

darvester-api-frontend already written, still in development, yet the core functionality (plus some) is complete.

Program layout

Darvester 2.0 will have three threads. The harvester loop, the API backend, and the frontend.

1. The API

^{Technically the second item, but it's important enough to be described first.}

The API handles all inserts, accesses, updates, and others. Anything we need from the database or need to do to the database is handled with the API.

With this large responsibility though, authentication is needed. Assuming FastAPI is used: FastAPI Security Intro

2. The Harvester/Core

The harvester loop will implement the core functionality of the original Darvester iteration. Database class and methods, UI, and alike will be stripped out, with the addition of API request methods.

3. The Frontend

Web and console will be utilized for the frontend.

Web will essentially utilize the already written darvester-api-frontend, written in React.js. For every release, we will have a build of the nodejs frontend ready to be served.

Console will most likely use the same implementation the current version of Darvester has.

Pipeline / CI/CD

How will we bundle the Python apps into a cross-platform release?

I can see both the API and Darvester written in Rust someday. That's a long ways away though. For now, serve releases with prebuilt web frontend builds.

One build system to rule them all?

Monorepo - An interesting tool/page for repositories containing multiple projects with close relationships.

Electron? Users may launch an Electron application that launches the frontend for first-time config. Electron as GUI for Python backend which utilizes IPC

[GEN2] Implement live console stream within the Manager tab

For both API and core, implement a console view that prints stdout from each process

Create database structure

Efficiently store user and guild info in individual tables

• Member: Include most data from these attributes

• bot (boolean)
• connected_accounts (str) API ref (Maybe this will be a separate table)
• created_at (string, else integer if epoch)
• display_name (string) (maybe replace for mention?)
• guild (int if it is a guild ID else string)
• joined_at (string, else int)
• mutual_guilds (string, else int)
• public_flags (string) (assuming list of flags also here)
• roles (string) (?)

• Guild:

• created_at (string, else int)
• description (string)
• features (string)
• id (int)
• member_count (int)
• owner (string) (?)
• owner_id (int)
• shard_id (int) (? is this useful at all?)

TODO:

• Link each data key to proper term in API docs
• Alter proposed structure to account for data objects returned by discord.py

[GEN2-FLUTTER] [MACOS?] Accessing external database file throws exception unless not in sandbox

Because of Mac OS application entitlements, the Flutter port will throw an exception when trying to access the database file for the first time (ex. starting the application, then going straight to Guilds).

It was observed that after resetting the database path, opening the database file locator dialog, then setting it again would fix the issue.

Later, after a gruesome amount of time spent into research, it was narrowed down to the Mac OS entitlement com.apple.security.app-sandbox restricting file access unless it was user-selected. Setting this to false fixes the issue.

[Enhancement] Use lurking instead of get_guild to prevent captchas

Use

client.join_guild(guildid,lurking=True)

instead of

client.get_guild(guildid)

for the guild that are get by the Explore Public Server, so there is no need to join the server and captcha don't appear(seems that lurking is still bugged for now, or i can't make it work for scraping member but this is a good feature to implement)