eshaan7 / flask-shell2http

Execute shell commands via HTTP server (via flask's endpoints).

Flask-Shell2HTTP

A minimalist Flask extension that serves as a RESTful/HTTP wrapper for python's subprocess API.

• Convert any command-line tool into a REST API service.
• Execute pre-defined shell commands asynchronously and securely via flask's endpoints with dynamic arguments, file upload, callback function capabilities.
• Designed for binary to binary/HTTP communication, development, prototyping, remote control and more.

Use Cases

• Set a script that runs on a succesful POST request to an endpoint of your choice. See Example code.
• Map a base command to an endpoint and pass dynamic arguments to it. See Example code.
• Can also process multiple uploaded files in one command. See Example code.
• This is useful for internal docker-to-docker communications if you have different binaries distributed in micro-containers. See real-life example.
• You can define a callback function/ use signals to listen for process completion. See Example code.
  • Maybe want to pass some additional context to the callback function ?
  • Maybe intercept on completion and update the result ? See Example code
• You can also apply View Decorators to the exposed endpoint. See Example code.

Note: This extension is primarily meant for executing long-running shell commands/scripts (like nmap, code-analysis' tools) in background from an HTTP request and getting the result at a later time.

Documentation

• Read the Quickstart from the documentation to get started!
• I also highly recommend the Examples section.
• CHANGELOG.

Quick Start

Dependencies

• Python: >=v3.6
• Flask
• Flask-Executor

Installation

$ pip install flask flask_shell2http

Example Program

Create a file called app.py.

from flask import Flask
from flask_executor import Executor
from flask_shell2http import Shell2HTTP

# Flask application instance
app = Flask(__name__)

executor = Executor(app)
shell2http = Shell2HTTP(app=app, executor=executor, base_url_prefix="/commands/")

def my_callback_fn(context, future):
  # optional user-defined callback function
  print(context, future.result())

shell2http.register_command(endpoint="saythis", command_name="echo", callback_fn=my_callback_fn, decorators=[])

Run the application server with, $ flask run -p 4000.

With <10 lines of code, we succesfully mapped the shell command echo to the endpoint /commands/saythis.

Making HTTP calls

This section demonstrates how we can now call/ execute commands over HTTP that we just mapped in the example above.

$ curl -X POST -H 'Content-Type: application/json' -d '{"args": ["Hello", "World!"]}' http://localhost:4000/commands/saythis

or using python's requests module,

# You can also add a timeout if you want, default value is 3600 seconds
data = {"args": ["Hello", "World!"], "timeout": 60}
resp = requests.post("http://localhost:4000/commands/saythis", json=data)
print("Result:", resp.json())

Note: You can see the JSON schema for the POST request here.

returns JSON,

{
  "key": "ddbe0a94",
  "result_url": "http://localhost:4000/commands/saythis?key=ddbe0a94&wait=false",
  "status": "running"
}

Then using this key you can query for the result or just by going to the result_url,

$ curl http://localhost:4000/commands/saythis?key=ddbe0a94&wait=true # wait=true so we do not have to poll

Returns result in JSON,

{
  "report": "Hello World!\n",
  "key": "ddbe0a94",
  "start_time": 1593019807.7754705,
  "end_time": 1593019807.782958,
  "process_time": 0.00748753547668457,
  "returncode": 0,
  "error": null
}

Inspiration

This was initially made to integrate various command-line tools easily with Intel Owl, which I am working on as part of Google Summer of Code.

The name was inspired by the awesome folks over at msoap/shell2http.

flask-shell2http's Issues

Fails if ran with multiple gunicorn workers

I figured out that the issue has to do with having multiple Gunicorn workers handling requests. The report only exists for the same worker that handled the original request. I am not sure how to solve it yet, but there you have it.

Originally posted by @dale-wahl in #24 (comment)

[test case] command that raises exception with returncode of -1

An important test case is missing.
See codecov diff: https://codecov.io/gh/Eshaan7/flask-shell2http/src/master/flask_shell2http/classes.py#L146

future_key already exists

Hello, I'm using flask-shell2http for communication and launching commands across docker containers. I'm trying to run a very simple script on one container from another container just to get the capability working. When I first ran the command from one container, I didn't have my permissions set on the script that was located on the other container but a key was still given to that command. Once I fixed the permissions on the script and tried to run the command again, it tried to give the same key value and I get an error in return saying future_key already exists

How can I fix this? Thank you and I am finding this application extremely useful.

Better typing and mypy in testing

Permission denied

Question why am i getting a permission denied error on one of your example scripts for running a post to a python file
no need to give any example other than it dont work lol

Add some kind of basic auth

I added some basic key checking but I wanted to know if there was a better way to add it, or if you want me to open a PR which adds some kind of hook to allow a custom function to be run on either the get or post to validate the args and in doing so allow for some kind of key / auth check?

# base_entrypoint.py
# change import to local version
from .api import Shell2HttpAPI

# api.py
# auth function
def check_stack_api_key(challenge_key: str) -> bool:
    key = os.environ.get("STACK_API_KEY", None)  # Get key from environment
    if key is None:
        return False
    if challenge_key == key:
        return True
    return False

class Shell2HttpAPI(MethodView):
    def get(self):
        ...

        # call the auth check
        stack_api_key = request.args.get("STACK_API_KEY", None)
        if not check_stack_api_key(challenge_key=stack_api_key):
            raise Exception("STACK_API_KEY doesn't match.")

    def post(self):
        ...
        # call the auth check
        json_input = request.get_json()
        if not check_stack_api_key(
            challenge_key=json_input.get("STACK_API_KEY", None)
        ):
            raise Exception("STACK_API_KEY doesn't match.")

Force Option and Result URL

It would be great to make the calls idempotent to some degree so that if you call it twice you can still follow the result without having to manually reconstruct the correct result url.

The error JSON could have the correct result_url so you don't need to construct it.

Failed to get result {'error': 'future_key 2ea86600 already exists'}

There could be an option during call to ignore if theres an existing entry and overwrite it?

BTW awesome lib, this was exactly what I was looking for. 🤗

use python format function

Hello @eshaan7 thanks for this project I really like it 👍
but I've got a simple question about the command variables, it's possible to use format string in python to select the right place of the variable?
for example

>>> command = "curl {url} --timeout {timeout}".format(url="http://google.com/",timeout=20)
"curl http://google.com/ --timeout 20"

Convert Shell2HTTP class to inherit from Flask.Blueprint

Currently, flask-shell2http works just like any other flask extension; you create an object from a class to which you pass the flask application instance. We could follow a different approach making it solely available as a pluggable Flask.Blueprint.

Some initial ideas:

• The Shell2HTTP class could derive from the Flask.Blueprint class; that way we can drop the init_app method, the register_command can be a wrapper over the Flask.Blueprint.add_url_rule method (it already is, but it works on app.add_url_rule). Then, the enduser would do something like,

from flask_shell2http import Shell2HTTP
# default blueprint instantiation
shell2http_bp = Shell2HTTP('awesome', __name__, url_prefix='/commands')
# extra method by flask-shell2http
shell2http_bp.register_command(endpoint="echo", command_name="echo", ...)

and then in the application factory,

app = Flask(__name__)
app.register_blueprint(shell2http_bp)

This would open doors for many other features since Flask.Flask and Flask.Blueprint derive from the same abstract base class and flask blueprints are very powerful.

Seeing intermittent future_key errors

2023-04-25 23:12:39 ERROR:flask_shell2http:future_key ebbb407f already exists 2023-04-25 23:12:39 ERROR:flask_shell2http:No report exists for key: 'ebbb407f'.

These are interspersed with working calls.

I turned off wait=true and switched to polling. This reduced the problem but didn't eliminate it.

Are there any docs on how to endure the key doesn't already exist? I don't believe I'm managing the keys externally to shell2http.

Thanks so much.

Question

Is it possible to handle command that shall work on files provide be the caller? If so how?

More examples/ use-cases

This extension is quite configurable and different users may choose to use it differently. For this, we should add more examples under examples/. These can be something new or ones already listed in the README.md.

Few ideas:

• An example showing encryption/decryption of the response using decorators.
• An example showing how flask-shell2http can be used for post-exploitation scenarios during a pentest (or as a backdoor). So perhaps an example integration with the https://github.com/operatorequals/covertutils or operatorequals/httpimport#1).

Create an ENUM for statuses

create an ENUM class for all statuses types

`wait` query param in `GET`

Adding your own route while using shell2http

Hey,

I am currently trying to use a vanilla flask route along with routes defined by shell2http, however, when I access the vanilla flask route, I get a 404. Any idea on how I can do this?

Nested JSON

I have an issue where the output of my cli tool is JSON but then the nested JSON inside the report field is all messed up and it seems I can't properly json.loads the main result. I wonder if there might be a way to detect json and include it properly rather than escaped inside as a substring?

`future.done()` and `callback_fn` race condition

When using &wait=true, the callback_fn cannot be used to modify the result because it is called after the future completes.

This happens if one calls future.done() to wait before the callback fn is executed, it would not work as intended.

Implement Logging

Should implement logging in the package. The logging configuration should be customizable by the user.
From Flask's docs: https://flask.palletsprojects.com/en/1.1.x/logging/#other-libraries

Possibility to interrupt command

Is it possible to interrupt command (like CTRL+C)?

For example, I started Locust (load testing framework) for one file, now I want to stop it and start one more test with other params.

[Testing] need to add unittests

Tests can be derived from the examples.

Enable ci/tests for python 3.10

A way to get binary output

First of all, thank you for this project! It was very easy to set up a couple of endpoints.

Now, I am trying to create another endpoint for a command which returns binary output and I'm getting an error:

"'utf-8' codec can't decode byte 0xff in position 0: invalid start byte"

Is there a way to configure an endpoint to return binary data, for example, in base64?

Generation of result key

I ran a few tests for an API I built to run a certain command line program, however I noticed that the keys returned are always for a given route, no matter what the parameters are, is this a feature?

[New Feature] intercepting arguments before command execution

Discussed in #56

^{Originally posted by tomvanderputte November 6, 2023}
Is it possible to intercept and change the dynamic arguments before the command is triggered?
The reason is: the arguments point to input/output files in a certain location. The application that sends the POST requests is not aware of this location (nor do I want it to be), so it sends only the filename.

So I want to edit the arguments/parameters to predfix the sent arguments with the correct path. How could I achieve this?

Enable test cases for flask 2.0

Cannot Import 'safe_join' from 'flask.helpers'

There is an ImportError while running 'flask run'

ImportError: cannot import name 'safe_join' from 'flask.helpers' (/env/lib/python3.9/site-packages/flask/helpers.py)

safe_join was removed from flask.helpers in flask version 2.1.0 released on 28-03-2022

From flask Version 2.1.0 CHANGES.rst

safe_join is removed, use werkzeug.utils.safe_join instead.

No report exists for key...

I am currently moving a Flask app to Gunicorn and am having an issue where I cannot ever retrieve a report even though it appears that the command will eventually be successfully run. I will send a post to an API endpoint and will receive a successful response that includes the key and result_url, but when I make a get request to that url, I only receive the "No report exists for key..." response. At first I thought it might be a delay, but even with repeated get requests I am not able to receive reports.

I have been looking through your documentation to get a better idea of how to access reports and understand how long before they expire/disappear. I had noticed previously that if I keep making a request for a specific key, it will eventually return No report exists for key even if one had previously existed. I am wondering if you have run into similar issues and if you could point me in the right direction to try to track down these reports or log them somewhere more permanent. I also tried playing with the Executor a bit, but am not entirely sure where to start with that.

Thank you for the help.

Testing suite

As listed in Flask's official docs, we should add a testing suite to this project.

See Point number 5 here.