Node graphs, OSINT data mining, and plugins. Connect unstructured and public data for transformative insights
License: GNU Affero General Public License v3.0
I'm currently working on the OSINTBuddy project in my free time.
Learn more about me on my personal site/blog studium.dev or join the discord or lemmy community to get involved with the osintbuddy project.
Self-Taught
Life-Long Learner
Fan of Linux & Libre Software
Is your feature request related to a problem? Please describe.
When running docker-compose up program returns an error and does not upload the image
Describe the solution you'd like
Additional context
I made a copy of .env.example to .env and even so the image did not upload
The console output:
osintbuddy-backend-1 | Traceback (most recent call last):
osintbuddy-backend-1 | File "/app/app/backend_pre_start.py", line 6, in <module>
osintbuddy-backend-1 | from app.db.session import SessionLocal
osintbuddy-backend-1 | File "/app/app/db/session.py", line 5, in <module>
osintbuddy-backend-1 | from app.core.config import settings
osintbuddy-backend-1 | File "/app/app/core/config.py", line 76, in <module>
osintbuddy-backend-1 | settings = Settings()
osintbuddy-backend-1 | ^^^^^^^^^^
osintbuddy-backend-1 | File "pydantic/env_settings.py", line 39, in pydantic.env_settings.BaseSettings.__init__
osintbuddy-backend-1 | File "pydantic/main.py", line 341, in pydantic.main.BaseModel.__init__
osintbuddy-backend-1 | pydantic.error_wrappers.ValidationError: 1 validation error for Settings
osintbuddy-backend-1 | SENTRY_DSN
osintbuddy-backend-1 | object of type 'NoneType' has no len() (type=type_error)
osintbuddy-backend-1 | INFO [alembic.runtime.migration] Context impl PostgresqlImpl.
osintbuddy-backend-1 | INFO [alembic.runtime.migration] Will assume transactional DDL.
osintbuddy-backend-1 | Traceback (most recent call last):
osintbuddy-backend-1 | File "/app/./app/initial_data.py", line 3, in <module>
osintbuddy-backend-1 | from app.db.init_db import init_db
osintbuddy-backend-1 | File "/app/app/db/init_db.py", line 3, in <module>
osintbuddy-backend-1 | from app.core.config import settings
osintbuddy-backend-1 | File "/app/app/core/config.py", line 76, in <module>
osintbuddy-backend-1 | settings = Settings()
osintbuddy-backend-1 | ^^^^^^^^^^
osintbuddy-backend-1 | File "pydantic/env_settings.py", line 39, in pydantic.env_settings.BaseSettings.__init__
osintbuddy-backend-1 | File "pydantic/main.py", line 341, in pydantic.main.BaseModel.__init__
osintbuddy-backend-1 | pydantic.error_wrappers.ValidationError: 1 validation error for Settings
osintbuddy-backend-1 | SENTRY_DSN
osintbuddy-backend-1 | object of type 'NoneType' has no len() (type=type_error)
osintbuddy-backend-1 exited with code 1
Describe the bug
Consistency
To Reproduce
Steps to reproduce the behavior:
Expected behavior
Installation to work, and instructions to be consistent in all available locations.
Screenshots
Desktop (please complete the following information):
Additional context
Latest version doesn't install and instructions are incorrect on main website
Projects don't seem to be saved.
Expected behavior
Projects should be saved either automatically or by clicking on a button.
Bug Description:
The database container fails on its initial launch, which causes a chain of issues resulting in osintbuddy authentication to fail since Casdoor relies on a working database.
How to Reproduce:
./launch start./launcher logs dbExpected behavior:
Postgres should launch and the queries should execute without error.
Screenshots & Logs:
https://paste.c-net.org/PraisesWarriors
The log output is large. A snippet of the error causing the Postgres database to fail during its launch sequence:
2024-04-16 09:14:24.538 UTC [35] FATAL: zero-length delimited identifier at or near """" at character 12
2024-04-16 09:14:24.538 UTC [35] STATEMENT: ALTER USER ""postgres"" WITH PASSWORD E'"password"';
Environment:
Additional context:
...
Problem:
Syntax errors are not captured at save time, instead they happen at plugin execution time, at which point the error only shows up in the logs.
Feature description:
ast.parse on the python text should be enough)Hello guys !
I'm on KALI LINUX, and following the installation process, this command "./launcher start" return
┌──(me㉿s404)-[~/Tools/osintbuddy]
└─$ ./launcher start
unknown shorthand flag: 'f' in -f
See 'docker --help'.
Usage: docker [OPTIONS] COMMAND
A self-sufficient runtime for containers
Options:
--config string Location of client config files (default "/home/me/.docker")
-c, --context string Name of the context to use to connect to the daemon (overrides DOCKER_HOST env var and default context set with "docker context use")
-D, --debug Enable debug mode
-H, --host list Daemon socket(s) to connect to
-l, --log-level string Set the logging level ("debug"|"info"|"warn"|"error"|"fatal") (default "info")
--tls Use TLS; implied by --tlsverify
--tlscacert string Trust certs signed only by this CA (default "/home/me/.docker/ca.pem")
--tlscert string Path to TLS certificate file (default "/home/me/.docker/cert.pem")
--tlskey string Path to TLS key file (default "/home/me/.docker/key.pem")
--tlsverify Use TLS and verify the remote
-v, --version Print version information and quit
Management Commands:
builder Manage builds
completion Generate the autocompletion script for the specified shell
config Manage Docker configs
container Manage containers
context Manage contexts
....
....
....
[OSINTBuddy: start] We ran into an error starting OSINTBuddy.
And following this doc https://osintbuddy.com/docs/installation, where to find the env.example file ?
Describe the bug
In the application, there seems to be no way to set the FastAPI BASE_URL to anything other than localhost:8000.
To Reproduce
Set .env to desired values:
BASE_URL=http://100.108.63.94:8000
Expected behavior
The baseURL should become http://100.108.63.94:8000.
Screenshots
Desktop (please complete the following information):
Additional context
We should consider adding logic to
Alternatively, we could just add REACT_APP_BASE_URL and NODE_ENV directly into the .env file. I added them, duplicated the values, executed "docker-compose build", and then "docker-compose up -d", and the baseURL became properly set.
Let me know your thoughts.
TODO:
Describe the bug
<backend>/api/v1/nodes/refresh endpoint returns an empty plugin list when backend code is not hosted at /app
To Reproduce
Steps to reproduce the behavior:
osintbuddy somewhere in your hostcd /path/to/osintbuddy/backend/backend/app; ~/.virtualenvs/osintbuddy/bin/uvicorn app.main:app --port 5000 --header server:app$ curl http://localhost:5000/api/v1/nodes/refresh
{"status":"success","plugins":[]}
Expected behavior
I expect .plugins not to be an empty list
Screenshots
n/a
Desktop (please complete the following information):
Additional context
n/a
This is powerfull tool, but it needed support for social networking.
Back in the day I did something not quite similar using dumb python scripts, for example to pull date from VK website for analysis, but you know, given that this tool have great potential to make truly open-source osint software, it should be incorporated.
From my share I would try to cook something up related to Russian-speaking SN web sites, see you soon.
Describe the bug
Install fails on other machines
To Reproduce
Steps to reproduce the behavior:
Expected behavior
Osintbuddy to install correctly
Screenshots
See comments
Desktop (please complete the following information):
Additional context
Yarn config in .env incorrect
Describe the bug
Missing LICENSE file
To Reproduce
N/A
Expected behavior
Find a license (MIT, Apache, GPL,... ) definition at https://github.com/jerlendds/osintbuddy/blob/main/LICENSE
Screenshots
N/A
Desktop (please complete the following information):
N/A
Additional context
A license is required for some individuals or companies to decide whether they can use and eventually contribute to the project
Currently to add a transform and/or to add a new node type you must switch between coding up the node on the frontend then coding up an endpoint for a transform. This could be greatly simplified by being able to extend the application at runtime and would make OSINTBuddy incredibly powerful and versatile
Bug description
The docker worker is not running and exiting with code 127. The error shown is /usr/bin/env: ‘bash\r’: No such file or directory
This seems to only happen on Windows environments.
To Reproduce
Expected behavior
All scripts should run without the trailing \r character and the image should be running after the stack is started successfully.
Screenshots
Desktop:
Additional context
I made sure .env uses LF instead of CRLF as it is the only file I modified from the main branch.
Previously start-reload.sh was also not found but I fixed that by setting git's autocrlf setting to false and then re-cloning the repository.
I changed wsl.conf to have the following configuration:
[interop]
appendWindowsPath = false
But the error persists.
Describe the bug
Osintbuddy main port conflict with macOS AirPlay.
To Reproduce
Steps to reproduce the behavior:
Attempt to launch osintbuddy on port 5000 (default)
Expected behavior
Not to conflict with ports OS uses by default
Desktop (please complete the following information):
Additional context
macOS users need to turn of AirPlay receiver to use osintbuddy
The CSE Search list is very long, making it hard for me to tell which ones might work or I could be selecting known broken searches... Either way, I have tested this with 10 randomly selected searches from the list, using the same query and page request. All result in a frontend error, "There was an error fetching CSE results. Please try again."
This list is not a request to fix those searches, I selected them at random.
In addition to that error, the console shows:
osintbuddy-backend-1 | plugin instance: <app.plugins.cse.CSESearchPlugin object at 0x7f82da07ee10>
osintbuddy-backend-1 | resp: <Response [200 OK]>
osintbuddy-backend-1 | INFO: 172.18.0.1:38180 - "GET /api/v1/nodes/transforms?node_type=CSE%20Search HTTP/1.1" 200 OK
osintbuddy-microservice-1 | Crawling results for: 001794496531944888666:iyxger-cwug{"time":"2023-06-13T12:04:23.529339569Z","id":"","remote_ip":"172.18.0.2","host":"microservice:1323","method":"GET","uri":"/google-cse?query=Elon%20Musk&pages=1&id=001794496531944888666%3Aiyxger-cwug","user_agent":"python-httpx/0.24.0","status":200,"error":"","latency":3000179116,"latency_human":"3.000179116s","bytes_in":0,"bytes_out":5}
osintbuddy-microservice-1 | 2023/06/13 12:04:23 Something went wrong: Bad Request
It appears the initial request/response is successful, but microservices fails to handle the data? or the data is returned malformed?
osintbuddy-janus-1 | exec /usr/local/bin/docker-entrypoint.sh: no such file or directory
Connection on localhost:3000 interface
- Creating a new project, it works, perfect
Connection to external interface (ens33): 192.168.1.100:3000
- Backend is reported as not running
- Not working
Test solutions:
- Modify DOMAIN in .env file (192.168.1.100)
- Change backend port in docker-compose.yml (5000:5000)
None of these solutions solved the bug
Screenshots
Desktop (please complete the following information):
Describe the bug
Cannot run backend's docker-compose due to failures building backend's Dockerfile
To Reproduce
Steps to reproduce the behavior:
$ cd backend
$ docker-compose build backend
Building backend
[+] Building 1.5s (11/13)
=> [internal] load build definition from backend.Dockerfile 0.0s
=> => transferring dockerfile: 761B 0.0s
=> [internal] load .dockerignore 0.0s
=> => transferring context: 2B 0.0s
=> [internal] load metadata for docker.io/library/python:3.11.3-slim-bullseye 1.2s
=> [1/9] FROM docker.io/library/python:3.11.3-slim-bullseye@sha256:551c9529e77896518ac5693d7e 0.0s
=> [internal] load build context 0.0s
=> => transferring context: 62.64kB 0.0s
=> CACHED [2/9] WORKDIR /app/ 0.0s
=> CACHED [3/9] RUN apt-get -y update && apt-get -y install nmap wget gnupg 0.0s
=> CACHED [4/9] RUN wget -q -O - https://dl.google.com/linux/linux_signing_key.pub | apt-key 0.0s
=> CANCELED [5/9] COPY requirements.txt /app/requirements.txt 0.0s
=> CACHED [6/9] RUN pip install --no-cache-dir --upgrade pip && pip install --no-cache-di 0.0s
=> ERROR [7/9] COPY osintbuddy-plugins /osintbuddy-plugins/ 0.0s
------
> [7/9] COPY osintbuddy-plugins /osintbuddy-plugins/:
------
failed to compute cache key: "/osintbuddy-plugins" not found: not found
ERROR: Service 'backend' failed to build : Build failed
Expected behavior
docker-compose build finishes without errors
Screenshots
N/A
Desktop (please complete the following information):
Additional context
Bug caused by missing dependency:
Hello,
I am trying to install osintbuddy on AWS (t3.2xlarge arch: amd64),
git clone --recurse-submodules https://github.com/jerlendds/osintbuddy.git
cd osintbuddy
cp .env.example .env
docker compose up
[ui build-stage 6/6] RUN yarn build:
0.666 yarn run v1.22.19
0.712 $ craco build
2.944 Creating an optimized production build...
23.00 Failed to compile.
23.00
23.00 Error: [BABEL] /app/src/index.tsx: React Refresh Babel transform should only be enabled in development environment. Instead, the environment is: "production". If you want to override this check, pass {skipEnvCheck: true} as plugin options. (While processing: "/app/node_modules/react-refresh/babel.js")
23.00 at Generator.next ()
23.00 at Generator.next ()
23.00 at Generator.next ()
23.00 at Generator.next ()
23.00 at cachedFunction.next ()
23.00 at loadPluginDescriptor.next ()
23.00 at loadPluginDescriptors.next ()
23.00 at Generator.next ()
23.00 at loadFullConfig.next ()
23.00 at transform.next ()
23.00
23.00
23.07 info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.
23.07 error Command failed with exit code 1.
failed to solve: process "/bin/sh -c yarn build" did not complete successfully: exit code: 1
Do you have an idea how to solve this problem ?
Describe the bug
A clear and concise description of what the bug is.
To Reproduce
Steps to reproduce the behavior:
Expected behavior
The input should not change without the user editing it
Screenshots
n/a
Desktop (please complete the following information):
Additional context
n/a
Problem:
It's common for the compose to partially fail:
When this happens, the interface boots, and user clicks "new graph".
Graph creation then fails (since janus is dead), but not before leaving over some rows in the database.
The user then has a non-functional "graph" name that, if deleted, will leave behind some other resources (solr index namespace). This will cause further errors when re-creating a graph for the same name, even after janus was fixed.
This causes friction between admins and users, since there's no clear indication for the reason of the failure (except for going through the docker logs and looking at stack traces).
Feature description:
The app would benefit from some multiple level of sanity checks:
depends_on: X: service_healthy to make sure the webserver does not start without first having working janus, solr, pg, s3, redis, etc etcAdditional context:
The sooner it crashes the sooner we can fix it
It would be great to have a shodan.io plugin available.
The API key could be added on a special settings page in the Plugin section in the Settings menu.
nslookup works fine for reverse lookups IF someone has configured a reverse zone file, which they don't always do so going to be using a third party service: https://api.hackertarget.com/reverseiplookup/?q=
Bug Description:
casdoor unable to login to database
How to Reproduce:
./launcher bootstrap
./launcher start
./launcher logs db
Screenshots & Logs:
ob_dev_db | 2024-05-06 06:00:36.093 UTC [1] LOG: starting PostgreSQL 15.4 (Debian 15.4-2.pgdg120+1) on x86_64-pc-linux-gnu, compiled by gcc (Debian 12.2.0-14) 12.2.0, 64-bit
ob_dev_db | 2024-05-06 06:00:36.094 UTC [1] LOG: listening on IPv4 address "0.0.0.0", port 5432
ob_dev_db | 2024-05-06 06:00:36.094 UTC [1] LOG: listening on IPv6 address "::", port 5432
ob_dev_db | 2024-05-06 06:00:36.107 UTC [1] LOG: listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
ob_dev_db | 2024-05-06 06:00:36.125 UTC [65] LOG: database system was shut down at 2024-05-06 06:00:35 UTC
ob_dev_db | 2024-05-06 06:00:36.135 UTC [1] LOG: database system is ready to accept connections
ob_dev_db | 2024-05-06 06:00:36.166 UTC [69] FATAL: password authentication failed for user "casdoor"
ob_dev_db | 2024-05-06 06:00:36.166 UTC [69] DETAIL: Role "casdoor" does not exist.
ob_dev_db | Connection matched pg_hba.conf line 100: "host all all all scram-sha-256"
ob_dev_db | 2024-05-06 06:00:37.847 UTC [72] FATAL: database "casdoor" does not exist
Environment:
Additional context:
...
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
Django
The Web framework for perfectionists with deadlines.
Laravel
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
Microsoft
Open source projects and samples from Microsoft.
Google
Google ❤️ Open Source for everyone.
Alibaba
Alibaba Open Source for everyone
D3
Data-Driven Documents codes.
Tencent
China tencent open source team.