Over the weekend I ran into an issue where the VPS I use didn't have the horsepower to complete a task in a reasonable ammount of time so I came up with the idea of remote workers/agents (not sure what to call them yet) that can be installed on other servers and will fetch tasks from reconness, complete them and return the results.
I can create the remote workers/agents but to really get this working there needs to be changes to the core of reconness. Essentially a "orchestrator" needs to be created that will distribute tasks & process the results. Below I'll explain what I think is required to get a basic proof of concept up and running, please feel free to make suggestions or criticisms as this is the first time doing any sort of distributed computing.
Worker Registration
When an worker is first installed it needs to register with reconness, so that reconness knows they exist & can assign tasks to them. As a POC they will send a POST request (with a valid JWT) with an worker ID (random string) to reconness.
POST: { Agent: "abc", Status: "New" }
- An API endpoint needs to be created to accept and store these ID's
- Bonus, they should also be able to de-register themselves e.g when uninstalling the worker.
Worker Status
Workers will "check in" with reconness periodically & provide their status (free or busy),
Example:
POST: {agent: "abc", status: "free"}
POST: {agent: "abc", status: "busy", task: "123"}
Workers that don't provide an update within 15 minutes should be considered dead & deleted. Workers that are free can have tasks assigned to them, those that are busy shouldn't.
- An endpoint needs to be created for workers to report their status (or maybe use the same as above)
- A "garbage collection" script needs to be run on the saved workers so that dead one's are removed
Worker Tasks
When running an agent inside reconnness (e.g nmap), you should have the option (e.g a checkbox) to send it to a worker. Reconness should then assign this task to a worker by updating an API endpoint with the task to run e.g /api/worker/tasks
Example:
/api/workers/tasks
{ taskID: "123", agent: "abc", task: "nmap -T4 yahoo.com", status: "pending"}
Agents will check this endpoint for tasks & accept them by sending a post request saying they accept task 123
Example:
POST: { taskID: "123", agent: "abc", action: "accept"}
Reonness would then update the endpoint and change the status to wip or etc
Example:
/api/workers/tasks
{ taskID: "123", agent: "abc", task: "nmap -T4 yahoo.com", status: "wip"}
When workers complete a task they would send a post request saying its complete, with the results. Reconness would then need to run the agent script on the results to save the data. The worker would also update their status to free.
Example:
POST: { taskID: "123", agent: "abc", action: "Complete", results: "nmap cli output..."}
I think this covers the basics of what would be needed for a working poc, also note the JSON above are just examples so feel free to use what you think is best.