hxsecurity / dongtai-agent-python

Python Agent is a Python application probe of DongTai IAST, which collects method invocation data during runtime of Python application by dynamic hooks.

DongTai-agent-python

• 中文版本(Chinese version)

Project Introduction

DongTai-agent-python is DongTai IAST's data acquisition tool for Python applications. In a Python application, the required data is collected through patching methods and functions, sent to the DongTai OpenAPI service, and then the cloud engine processes the data to determine if there are security vulnerabilities.

DongTai-agent-python

• dongtai_agent_python/api/ Report the collected data to the DongTai OpenAPI service.
• dongtai_agent_python/assess/ Hook python methods according to the cloud strategy.
• dongtai_agent_python/assess_ext/ Hook cpython underlying methods according to cloud strategy.
• dongtai_agent_python/cli/ Control the hot update of the agent version.
• dongtai_agent_python/context/ Request context and context tracker.
• dongtai_agent_python/middleware/ Used to access different python frameworks, currently supports Django and Flask, both of which are introduced in the form of middleware.
• dongtai_agent_python/policy/ Strategy rules and tainted data processing.
• dongtai_agent_python/setting/ Agent configuration.
• dongtai_agent_python/config.json For configuration DongTai OpenAPI Url, Token, Project Name.

Application Scenarios

• DevOps
• Security test the application before it goes online
• Third-party Component Management
• Code audit
• 0day digging

Requirements

• Python: >=3.6
• CPython
• Compiling Dependencies (Agent version >= 1.1.4)
  • gcc (Linux/macOS)
  • make (Linux/macOS)
  • cmake: >= 3.6
  • Visual Studio (Windows)
  • bash (Alpine Linux)
  • libc-dev (Alpine Linux)
  • linux-headers (Alpine Linux)
• Web Framework
  • Django: 3.0-3.2, 4.0
  • Flask: 1.0-1.2, 2.0
• Python packages
  • psutil: >= 5.8.0
  • requests: >= 2.25.1
  • pip: >= 19.2.3

Quick Start

Please refer to the Quick Start.

Quick Development

1. Fork the DongTai-agent-python , clone your fork:

   git clone https://github.com/<your-username>/DongTai-agent-python
   

2. Write code to your needs.

3. Modify the configuration file dongtai_agent_python/config.json

   • iast.server.token: "3d6bb430bc3e0b20dcc2d00000000000000a"
   • iast.server.url: "https://iast-test.huoxian.cn/openapi"
   • project.name: "DemoProjectName"

   URL and token from the hole state IAST-web page (eg: https://iast-test.huoxian.cn/deploy) > python-agent deployment page,Obtained from the shell command of downloading agent，Replace the url domain name and token respectively

4. The project is packaged and executed in the root directory of the agent project

   python3 setup.py sdist

5. Install the agent
   After packaging, the dist directory will be generated, and the installation package will be found in the dist directory, Put the dongtai_agent_python.tar.gz installation package on the machine where the Web-server is located，Execute pip installation

   pip3 install ./dongtai-python-agent.tar.gz 

Project access Agent

Agent Configuration

Environment Variables

• DEBUG mode: DEBUG=1
• Auto Create Project: AUTO_CREATE_PROJECT=1
• Project Name: PROJECT_NAME=Demo
• Project Version: PROJECT_VERSION=v1.0
• Agent Name: ENGINE_NAME=test-flask
• Log Path: LOG_PATH=/tmp/dongtai-agent-python.log

You can also configure the value in dongtai_agent_python/config.json

• debug
• project.name
• project.version
• engine.name
• log.log_path

Note: The priority of the system environment variable is higher than the configuration file

Django

1. Enter the main directory of the app
2. Open the app/settings.py file and find the line of MIDDLEWARE
3. Insert below the line dongtai_agent_python.middlewares.django_middleware.FireMiddleware
4. Restart app

Flask

1. Modify the entry file of the project (such as app.py) and add the following content

   app = Flask(__name__)
   
   from dongtai_agent_python.middlewares.flask_middleware import AgentMiddleware
   app.wsgi_app = AgentMiddleware(app.wsgi_app, app)
   
   if __name__ == '__main__':
       app.run()

2. Restart app

dongtai-agent-python's Issues

Django: 3.0/3.1 AttributeError: 'HttpResponse' object has no attribute 'headers'

 Traceback (most recent call last):
  File "/opt/hostedtoolcache/Python/3.6.15/x64/lib/python3.6/site-packages/django/core/handlers/exception.py", line 47, in inner
    response = get_response(request)
  File "/opt/hostedtoolcache/Python/3.6.15/x64/lib/python3.6/site-packages/django/utils/deprecation.py", line 116, in __call__
    response = self.process_response(request, response)
  File "/opt/hostedtoolcache/Python/3.6.15/x64/lib/python3.6/site-packages/dongtai_agent_python/middlewares/django_middleware.py", line 91, in process_response
    resp_header = dict(response.headers)
AttributeError: 'HttpResponse' object has no attribute 'headers'

111

使用正常的app.run()方式启动
File "/opt/nsfocus/368_env/lib/python3.6/site-packages/flask/app.py", line 2450, in wsgi_app
response = self.handle_exception(e)
File "/opt/nsfocus/368_env/lib/python3.6/site-packages/flask/app.py", line 1867, in handle_exception
reraise(exc_type, exc_value, tb)
File "/opt/nsfocus/368_env/lib/python3.6/site-packages/flask/_compat.py", line 39, in reraise
raise value
File "/opt/nsfocus/368_env/lib/python3.6/site-packages/flask/app.py", line 2446, in wsgi_app
ctx.push()
File "/opt/nsfocus/368_env/lib/python3.6/site-packages/flask/ctx.py", line 390, in push
self.session = session_interface.open_session(self.app, self.request)
File "/opt/nsfocus/368_env/lib/python3.6/site-packages/flask/sessions.py", line 343, in open_session
val = request.cookies.get(app.session_cookie_name)
AttributeError: 'str' object has no attribute 'get'
使用gunicorn启动报错信息上面加下面
Traceback (most recent call last):
File "/opt/nsfocus/368_env/lib64/python3.6/site-packages/gunicorn/workers/sync.py", line 135, in handle
self.handle_request(listener, req, client, addr)
File "/opt/nsfocus/368_env/lib64/python3.6/site-packages/gunicorn/workers/sync.py", line 180, in handle_request
if isinstance(respiter, environ['wsgi.file_wrapper']):
KeyError: 'wsgi.file_wrapper'
想请问一下这种情况怎么排查Agent问题，原来的flask版本为2.0.0 降低到1.1.4 还是不行，请大佬指教一下

Set pypi and system package mirrors in docker

Currently pypi and system package mirrors are hardcoded and slow in Github Actions, We can add ARGs to Dockerfile for different environments

[Bug]python agent 一直处于暂停状态, 请求接口超时:

Preflight Checklist

• I agree to follow the Code of Conduct that this project adheres to.
• I have searched the issue tracker for an issue that matches the one I want to file, without success.
• I am not looking for support or already pursued the available support channels without success.

Version

最新版本

Installation Type

Official Docker Compose

Service Name

DongTai-OpenAPI, DongTai-agent-python

Describe the details of the bug and the steps to reproduce it

安装好agent后启动flask项目，python agent一直处于暂停状态，log显示请求接口超时

`

[2022-09-19 10:32:16,679] INFO [base_middleware] python agent init, version: 1.4.0
[2022-09-19 10:32:16,694] INFO [system_info] psutil import success
[2022-09-19 10:32:16,694] INFO [system_info] get network success
[2022-09-19 10:32:36,722] ERROR [openapi] report failed: HTTPConnectionPool(host='172.20.148.212', port=8002): Max retries exceeded with url: /openapi/api/v1/agent/register (Caused by ConnectTimeoutError(<urllib3.connection.HTTPConnection object at 0x7f4cfa817f28>, 'Connection to 172.20.148.212 timed out. (connect timeout=20)'))
Traceback (most recent call last):
  File "/root/testproject/venv/lib/python3.6/site-packages/urllib3/connection.py", line 175, in _new_conn
    (self._dns_host, self.port), self.timeout, **extra_kw
  File "/root/testproject/venv/lib/python3.6/site-packages/urllib3/util/connection.py", line 95, in create_connection
    raise err
  File "/root/testproject/venv/lib/python3.6/site-packages/urllib3/util/connection.py", line 85, in create_connection
    sock.connect(sa)
socket.timeout: timed out

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/root/testproject/venv/lib/python3.6/site-packages/urllib3/connectionpool.py", line 710, in urlopen
    chunked=chunked,
  File "/root/testproject/venv/lib/python3.6/site-packages/urllib3/connectionpool.py", line 398, in _make_request
    conn.request(method, url, **httplib_request_kw)
  File "/root/testproject/venv/lib/python3.6/site-packages/urllib3/connection.py", line 239, in request
    super(HTTPConnection, self).request(method, url, body=body, headers=headers)
  File "/usr/lib64/python3.6/http/client.py", line 1254, in request
    self._send_request(method, url, body, headers, encode_chunked)
  File "/usr/lib64/python3.6/http/client.py", line 1300, in _send_request
    self.endheaders(body, encode_chunked=encode_chunked)
  File "/usr/lib64/python3.6/http/client.py", line 1249, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "/usr/lib64/python3.6/http/client.py", line 1036, in _send_output
    self.send(msg)
  File "/usr/lib64/python3.6/http/client.py", line 974, in send
    self.connect()
  File "/root/testproject/venv/lib/python3.6/site-packages/urllib3/connection.py", line 205, in connect
    conn = self._new_conn()
  File "/root/testproject/venv/lib/python3.6/site-packages/urllib3/connection.py", line 182, in _new_conn
    % (self.host, self.timeout),
urllib3.exceptions.ConnectTimeoutError: (<urllib3.connection.HTTPConnection object at 0x7f4cfa817f28>, 'Connection to 172.20.148.212 timed out. (connect timeout=20)')

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/root/testproject/venv/lib/python3.6/site-packages/requests/adapters.py", line 450, in send
    timeout=timeout
  File "/root/testproject/venv/lib/python3.6/site-packages/urllib3/connectionpool.py", line 788, in urlopen
    method, url, error=e, _pool=self, _stacktrace=sys.exc_info()[2]
  File "/root/testproject/venv/lib/python3.6/site-packages/urllib3/util/retry.py", line 592, in increment
    raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPConnectionPool(host='172.20.148.212', port=8002): Max retries exceeded with url: /openapi/api/v1/agent/register (Caused by ConnectTimeoutError(<urllib3.connection.HTTPConnection object at 0x7f4cfa817f28>, 'Connection to 172.20.148.212 timed out. (connect timeout=20)'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/root/testproject/venv/lib64/python3.6/site-packages/dongtai_agent_python/api/openapi.py", line 83, in report
    res = requests.post(api_url, data=body_data, timeout=20, headers=self.headers)
  File "/root/testproject/venv/lib/python3.6/site-packages/requests/api.py", line 117, in post
    return request('post', url, data=data, json=json, **kwargs)
  File "/root/testproject/venv/lib/python3.6/site-packages/requests/api.py", line 61, in request
    return session.request(method=method, url=url, **kwargs)
  File "/root/testproject/venv/lib/python3.6/site-packages/requests/sessions.py", line 529, in request
    resp = self.send(prep, **send_kwargs)
  File "/root/testproject/venv/lib/python3.6/site-packages/requests/sessions.py", line 645, in send
    r = adapter.send(request, **kwargs)
  File "/root/testproject/venv/lib/python3.6/site-packages/requests/adapters.py", line 507, in send
    raise ConnectTimeout(e, request=request)
requests.exceptions.ConnectTimeout: HTTPConnectionPool(host='172.20.148.212', port=8002): Max retries exceeded with url: /openapi/api/v1/agent/register (Caused by ConnectTimeoutError(<urllib3.connection.HTTPConnection object at 0x7f4cfa817f28>, 'Connection to 172.20.148.212 timed out. (connect timeout=20)'))

[2022-09-19 10:32:36,722] ERROR [base_middleware] python agent register error
[2022-09-19 10:32:37,595] INFO [openapi] startup time: 20876.171827316284ms report failed
[2022-09-19 10:32:37,596] INFO [base_middleware] python agent hook open

`

Additional Information

No response

Logs

No response

[Bug]: python-agent运行时的bug

Preflight Checklist

• I agree to follow the Code of Conduct that this project adheres to.
• I have searched the issue tracker for an issue that matches the one I want to file, without success.
• I am not looking for support or already pursued the available support channels without success.

Version

1.1.2

Installation Type

Official Docker Compose

Service Name

DongTai-agent-python

flask，版本为1.1.2

Describe the details of the bug and the steps to reproduce it

当在代码中使用eval("self.aData...")的时候InstallFcnHook无法识别aData（其中aData是变量），会报错：AttributeError: 'InstallFcnHook' object has no attribute 'aData'

Additional Information

No response

Logs

No response

setup.cfg can not contains non-ascii character on windows

Traceback (most recent call last):
  File "C:\hostedtoolcache\windows\Python\3.6.8\x64\lib\runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "C:\hostedtoolcache\windows\Python\3.6.8\x64\lib\runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "C:\Users\runneradmin\AppData\Roaming\Python\Python36\site-packages\flask\__main__.py", line 14, in <module>
    main(as_module=True)
  File "C:\Users\runneradmin\AppData\Roaming\Python\Python36\site-packages\flask\cli.py", line 906, in main
    cli.main(args=args, prog_name=name)
  File "C:\Users\runneradmin\AppData\Roaming\Python\Python36\site-packages\flask\cli.py", line 569, in main
    return super(FlaskGroup, self).main(*args, **kwargs)
  File "C:\Users\runneradmin\AppData\Roaming\Python\Python36\site-packages\click\core.py", line 1053, in main
    rv = self.invoke(ctx)
  File "C:\Users\runneradmin\AppData\Roaming\Python\Python36\site-packages\click\core.py", line 1659, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "C:\Users\runneradmin\AppData\Roaming\Python\Python36\site-packages\click\core.py", line 1395, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "C:\Users\runneradmin\AppData\Roaming\Python\Python36\site-packages\click\core.py", line 754, in invoke
    return __callback(*args, **kwargs)
  File "C:\Users\runneradmin\AppData\Roaming\Python\Python36\site-packages\click\decorators.py", line 84, in new_func
    return ctx.invoke(f, obj, *args, **kwargs)
  File "C:\Users\runneradmin\AppData\Roaming\Python\Python36\site-packages\click\core.py", line 754, in invoke
    return __callback(*args, **kwargs)
  File "C:\Users\runneradmin\AppData\Roaming\Python\Python36\site-packages\flask\cli.py", line 779, in run_command
    app = DispatchingApp(info.load_app, use_eager_loading=eager_loading)
  File "C:\Users\runneradmin\AppData\Roaming\Python\Python36\site-packages\flask\cli.py", line 300, in __init__
    self._load_unlocked()
  File "C:\Users\runneradmin\AppData\Roaming\Python\Python36\site-packages\flask\cli.py", line 324, in _load_unlocked
    self._app = rv = self.loader()
  File "C:\Users\runneradmin\AppData\Roaming\Python\Python36\site-packages\flask\cli.py", line 386, in load_app
    raise_if_not_found=False)
  File "C:\Users\runneradmin\AppData\Roaming\Python\Python36\site-packages\flask\cli.py", line 236, in locate_app
    __import__(module_name)
  File "D:\a\dongtai-agent-python-test-actions\dongtai-agent-python-test-actions\testFlask\app.py", line 6, in <module>
    app.wsgi_app = AgentMiddleware(app.wsgi_app, app)
  File "C:\Users\runneradmin\AppData\Roaming\Python\Python36\site-packages\dongtai_agent_python\middlewares\flask_middleware.py", line 35, in __init__
    register_resp = self.agent_upload.agent_register(cur_middle)
  File "C:\Users\runneradmin\AppData\Roaming\Python\Python36\site-packages\dongtai_agent_python\report\upload_data.py", line 245, in agent_register
    Resp = self.base_report(url, register_data)
  File "C:\Users\runneradmin\AppData\Roaming\Python\Python36\site-packages\dongtai_agent_python\report\upload_data.py", line 158, in base_report
    print(body)
  File "C:\hostedtoolcache\windows\Python\3.6.8\x64\lib\encodings\cp1252.py", line 19, in encode
    return codecs.charmap_encode(input,self.errors,encoding_table)[0]
UnicodeEncodeError: 'charmap' codec can't encode character '\u65e0' in position 367: character maps to <undefined>

see: pypa/pip#8931

有个bug想请教一下大佬

使用正常的app.run()方式启动
File "/path/lib/python3.6/site-packages/flask/app.py", line 2450, in wsgi_app
response = self.handle_exception(e)
File "/path/lib/python3.6/site-packages/flask/app.py", line 1867, in handle_exception
reraise(exc_type, exc_value, tb)
File "/path/lib/python3.6/site-packages/flask/_compat.py", line 39, in reraise
raise value
File "/path/lib/python3.6/site-packages/flask/app.py", line 2446, in wsgi_app
ctx.push()
File "/path/lib/python3.6/site-packages/flask/ctx.py", line 390, in push
self.session = session_interface.open_session(self.app, self.request)
File "/path/lib/python3.6/site-packages/flask/sessions.py", line 343, in open_session
val = request.cookies.get(app.session_cookie_name)
AttributeError: 'str' object has no attribute 'get'
使用gunicorn启动报错信息上面加下面
Traceback (most recent call last):
File "/path/lib64/python3.6/site-packages/gunicorn/workers/sync.py", line 135, in handle
self.handle_request(listener, req, client, addr)
File "/path/lib64/python3.6/site-packages/gunicorn/workers/sync.py", line 180, in handle_request
if isinstance(respiter, environ['wsgi.file_wrapper']):
KeyError: 'wsgi.file_wrapper'
想请问一下这种情况怎么排查Agent问题，原来的flask版本为2.0.0 降低到1.1.4 还是不行，请大佬指教一下
我想学习一下Agent的开发，请大佬删除一下上一条我不小心提交且已经关闭的issues，不好意思，麻烦您了