sophos / sophos-central-api-connector

Leverage Sophos Central API

Sophos Central API Connector

Python library to utilise many of the features in Sophos Central API across multiple or single tenants

• Documentation: Sophos Central API
• Documentation: Sophos Central API Connector

---

Table of contents:

• Sophos Central API Connector

  • Features
  • Quick start
    • Important!
  • Prerequisites
  • Install
  • Authentication
    • Static Credentials
    • AWS Secrets Manager
  • Basic Examples
    • Help
    • Tenants List
    • Inventory
    • Alerts/Event Information
    • Local Site
  • Output Options
  • Troubleshooting
  • Structure

  ---

Features

All features can be run against single or multiple tenants

• Gather tenant system inventory
  • Output to stdout, json, Splunk
• Gather alerts
  • Alert polling
  • Output to stdout, json, Splunk
• Local Sites
  • Clean up Global exclusions
    • Compare exclusions to SophosLabs Intelix
  • Generate report
• IOC Hunting - Utilising Live Discover or XDR DataLake
  • MISP Attribute hunting (eventId, tags)
  • RAW JSON input
  • Saved search

---

Quick start

Want to test as quickly as possible? Follow the below quick start steps to begin looking at your Sophos Central data!

1. Install latest version of Python 3

2. Create a folder e.g "sophos_test"

3. Open a command prompt/terminal

4. Create a Python Virtual Environment:

   python -m venv <folder_name>
   

5. Activate the Python Virtual Environment:

   <path_to_folder>\Scripts\activate
   

6. Install the Sophos Central API Connector (this will also install the requirements):

   pip install sophos-central-api-connector
   

7. Once it has finished installing browse to:

   cd <path_to_folder>\Lib\site-packages\sophos_central_api_connector
   

8. Run the following command to view help to begin:

   python sophos_central_main.py --help
   

9. Add your Sophos Central API id and secret to the sophos_config.ini under the folder: \Lib\site-packages\sophos_central_api_connector\config

   Important!

   We would recommend that the static entry is only used for testing purposes and the token is stored and accessed securely. Please reference the authentication section

---

Prerequisites

In order to use the package you will require a valid API key from your Sophos Central tenant. To obtain a valid API key please reference the documentation here

---

Install

pip install --user sophos_central_api_connector

---

Authentication

There are two options for authentication, the setting used here will be used for all areas of authentication. As mentioned under the configuration section we recommend using the AWS Secrets Manager for storing these credentials. Only use the static credentials for testing purposes.

Static Credentials

To specify using the static credentials which are in the *config.ini files you can use the following: python3 sophos_central_main.py --auth static

AWS Secrets Manager

To specify using the AWS settings which are in the *config.ini files to retrieve the secrets and token you can use the following: python3 sophos_central_main.py --auth aws

---

Basic Examples

Help

To get information on the CLI commands when using the sophos_central_main.py run:

python sophos_central_main.py --help

Tenants List

To get a list of tenants:

python sophos_central_main.py --auth <auth_option> --get tenants

Inventory

To get inventory data:

python sophos_central_main.py --auth <auth_option> --get inventory --output <output_option>

Alerts/Event Information

To get alert data:

python sophos_central_main.py --auth <auth_option> --get alerts --days <integer: 1-90> --output <output_option>

Local Site

To get a list of local site data:

python sophos_central_main.py --auth <auth_option> --get local-sites --output <output_option>

---

Output Options

There are four output options available for the inventory, simply add one of the following after --output:

• stdout: Print the information to the console.
• json: Save the output of the request to a json file
• splunk: This will send the data to Splunk with no changes made. This will apply the settings made in the transform files.
• splunk_trans: Using this output will apply the information set in the splunk_config.ini for the host, source and sourcetype. This will overrun the settings in the transform files in Splunk but not the Index that the data should be sent to.

---

Troubleshooting

All logging is done via the python logging library. Valid logging levels are:

• INFO
• DEBUG
• CRITICAL
• WARNING
• ERROR

For basic feedback set the logging level to INFO

---

Structure

Below is the structure after installing through pip:

sophos_central_api_connector
|   .gitignore
|   LICENSE
|   MANIFEST.in
|   README.md
|   requirements.txt
|   setup.py
|___docs
|       alerts.md
|       intelix.md
|       intelix_configuration.md
|       inventory.md
|       ioc_hunter.md
|       local_sites.md
|       misp_configuration.md
|       sophos_configuration.md
|       splunk_configuration.md
|___queries
|       |___live_discover_queries
|               ld_ioc_hunter.sql
|       |___xdr_queries
|               xdr_ioc_hunter.sql
|___sophos_central_api_connector
|       ioc_hunter.py
|       sophos_central_api_live_discover.py
|       sophos_central_api_auth.py
|       sophos_central_api_awssecrets.py
|       sophos_central_api_connector_utils.py
|       sophos_central_api_delete_data.py
|       sophos_central_api_get_data.py
|       sophos_central_api_intelix.py
|       sophos_central_api_output.py
|       sophos_central_api_polling.py
|       sophos_central_api_tenants.py
|       sophos_central_api_hec_splunk.py
|       sophos_central_main.py
|       get_admins.py
|       get_roles.py
|       get_firewall_groups.py
|       get_firewalls.py
|___config
|       intelix_config.ini
|       misp_config.ini
|       sophos_central_api_config.py
|       sophos_config.ini
|       splunk_config.ini

Below is the structure with all the files that are created through different mechanisms:

sophos_central_api_connector
|   .gitignore
|   LICENSE
|   MANIFEST.in
|   README.md
|   requirements.txt
|   setup.py
|___sophos_central_api_connector
|   |___docs
|   |       alerts.md
|   |       intelix.md
|   |       intelix_configuration.md
|   |       inventory.md
|   |       ioc_hunter.md
|   |       local_sites.md
|   |       misp_configuration.md
|   |       sophos_configuration.md
|   |       splunk_configuration.md
|___queries
|       |___live_discover_queries
|               ld_ioc_hunter.sql
|       |___xdr_queries
|               xdr_ioc_hunter.sql
|       ioc_hunter.py
|       sophos_central_api_live_discover.py
|       sophos_central_api_auth.py
|       sophos_central_api_awssecrets.py
|       sophos_central_api_connector_utils.py
|       sophos_central_api_delete_data.py
|       sophos_central_api_get_data.py
|       sophos_central_api_intelix.py
|       sophos_central_api_output.py
|       sophos_central_api_polling.py
|       sophos_central_api_tenants.py
|       sophos_central_api_hec_splunk.py
|       sophos_central_main.py
|       get_admins.py
|       get_roles.py
|       get_firewall_groups.py
|       get_firewalls.py
|___config
|       intelix_config.ini
|       misp_config.ini
|       sophos_central_api_config.py
|       sophos_config.ini
|       splunk_config.ini
|___logs
|       failed_events.json
|___output
|   |___get_alerts
|   |       <tenant_name>_<tenant_id>.json
|   |       ...
|   |___get_inventory
|   |       <tenant_name>_<tenant_id>.json
|   |       ...
|   |___get_local_sites
|   |       <tenant_name>_<tenant_id>.json
|   |       ...
|   |___admin_data
|   |       <tenant_name>_<tenant_id>.json
|   |       ...	
|   |___roles_data
|   |       <tenant_name>_<tenant_id>.json
|   |       ...
|   |_firewall_groups
|   |       <tenant_name>_<tenant_id>.json
|   |       ...
|   |_firewall_inventory
|   |       <tenant_name>_<tenant_id>.json
|   |       ...
|   |___intelix
|       |___delete_local_sites
|           <date>_<time>_deletion_details.json
|           <date>_<time>_deletion_report.json
|           ...
|       <date>_<time>_intelix_results.json
|       <date>_<time>_results_combined.json
|       <tenant_id>_<date>_<time>_<risk_level>_dry_run_report.json
|       ...
|   |___query_results
|       <xdr_datalake/live-discover>_query_list.json
|       <xdr_datalake/live-discover>_search_data_<timestamp>.json
|       <xdr_datalake/live-discover>_result_data_<timestamp>.json
|       live-discover_endpoint_data_<timestamp>.json
|___polling
|       poll_config.json
|       alert_ids.json
|       temp_alert_ids.json

sophos-central-api-connector's Issues

Link to documentation is broken

https://github.com/sophos-cybersecurity/sophos_central_api_connector/tree/master/sophos_central_api_connector/docs

Generates a 404 error

KeyError: 'apiHost'

I appear to be getting a very similar error when attempting to perform a number of operations, I'm slightly able to bypass it by commenting line 74 in sophos_central_api_tenants.py but it doesn't work for all use cases, only a few.

File "C:\Users\krsecurity\Sophos\lib\site-packages\sophos_central_api_connector\sophos_central_api_tenants.py", line 74, in get_next_page
"page_url": item['apiHost']
KeyError: 'apiHost'

I get the above error when simply trying to run 'python sophos_central_main.py -a static -g alerts --days 1'.

I get similar errors when basically running any of the scripts, including --get tenants etc.

Any help at all?

Thanks

Latest release (0.1.4) has faulty dependencies (fixed in `master`)

Hi, thanks for the great set of scripts, it's been really helpful.

I encountered a problem when trying to install this package using poetry add sophos-central-api-connector. It output the following:

Because no versions of requests match >2.25.1
 and requests (2.25.1) depends on idna (>=2.5,<3), requests (>=2.25.1) requires idna (>=2.5,<3).
And because sophos-central-api-connector (0.1.4) depends on both idna (>=3.1) and requests (>=2.25.1), sophos-central-api-connector is forbidden.
So, because no versions of sophos-central-api-connector match >0.1.4,<0.2.0
 and test depends on sophos-central-api-connector (^0.1.4), version solving failed.

I only checked this recently, but trying to install via pip is even worse - pip gets stuck trying to resolve the dependencies by slowly backtracking over various versions of different packages, such as boto3 and botocore:

INFO: This is taking longer than usual. You might need to provide the dependency resolver with stricter constraints to reduce runtime. If you want to abort this run, you can press Ctrl + C to do so. To improve how pip performs, tell us what happened here: https://pip.pypa.io/surveys/backtracking
Collecting botocore>=1.20.21
  Downloading botocore-1.20.83-py2.py3-none-any.whl (7.6 MB)
     |████████████████████████████████| 7.6 MB 5.4 MB/s 
Collecting boto3>=1.17.21
  Downloading boto3-1.17.82-py2.py3-none-any.whl (131 kB)
     |████████████████████████████████| 131 kB 4.3 MB/s 
Collecting botocore>=1.20.21
  Downloading botocore-1.20.82-py2.py3-none-any.whl (7.6 MB)
     |████████████████████████████████| 7.6 MB 4.0 MB/s 
Collecting boto3>=1.17.21
  Downloading boto3-1.17.81-py2.py3-none-any.whl (131 kB)
     |████████████████████████████████| 131 kB 4.2 MB/s 
Collecting botocore>=1.20.21
  Downloading botocore-1.20.81-py2.py3-none-any.whl (7.6 MB)
     |████████████████████████████████| 7.6 MB 4.2 MB/s 
Collecting boto3>=1.17.21
  Downloading boto3-1.17.80-py2.py3-none-any.whl (131 kB)
     |████████████████████████████████| 131 kB 4.1 MB/s 
Collecting botocore>=1.20.21
  Downloading botocore-1.20.80-py2.py3-none-any.whl (7.6 MB)
     |████████████████████████████████| 7.6 MB 4.1 MB/s 
...

---

This was fixed in 0750257, but no new releases have been published since then.

I'm able to directly reference this git repo in Poetry instead of downloading from PyPI, but I would appreciate if a new release could be published.

Performance: use requests.Session() to reduce tcp and ssl handshake overhead.

sess = requests.Session()
sess.get()

will have far better performance and use less bandwidth due to not having to repeat tcp handshake and ssl handshakes. My tests show about twice as fast.

See 2 examples:
(venv) 15645(0811)mattjenkins@DatapriseMBP:Automox$ cat test.py test2.py
import requests

for x in range(1,100):
if (result := requests.get('https://www.google.com')):
pass
else:
print(False)
import requests

sess = requests.Session()
for x in range(1,100):
if (result := sess.get('https://www.google.com')):
pass
else:
print(False)

(venv) 15646(0811)mattjenkins@DatapriseMBP:Automox$ time python test.py

real 0m13.911s
user 0m1.774s
sys 0m0.149s
(venv) 15647(0811)mattjenkins@DatapriseMBP:Automox$ time python test2.py

real 0m6.672s
user 0m0.488s
sys 0m0.083s

Errored when pulling alerts/inventory

sophos_central_api_connector_utils.py", line 115, in validate_page_size
elif int(page_size) <= api_conf.max_alerts_page:
ValueError: invalid literal for int() with base 10: ''

Seems like the defaults in sophos_central_api_config.py were not being used. Fixed by entering values in sophos_config.ini

configuration in wrong place

I'm using version 0.1.1 from pypi. I couldn't figure out why it kept saying no section static. So I eventually tracked it down and put a print statement.
(venv) 12286(1602)mattjenkins@DatapriseMBP:sophos_central_api_connector$ python sophos_central_main.py -a static /Users/mattjenkins/venv/lib/python3.8/site-packages/sophos_central_api_connector\config\sophos_config.ini Traceback (most recent call last): File "sophos_central_main.py", line 402, in <module> main(args) File "sophos_central_main.py", line 317, in main client_id, client_secret = get_sophos_creds(sophos_auth, sophos_final_path) File "sophos_central_main.py", line 239, in get_sophos_creds client_id = sophos_conf.get('static', 'client_id') File "/usr/local/Cellar/[email protected]/3.8.5/Frameworks/Python.framework/Versions/3.8/lib/python3.8/configparser.py", line 781, in get d = self._unify_values(section, vars) File "/usr/local/Cellar/[email protected]/3.8.5/Frameworks/Python.framework/Versions/3.8/lib/python3.8/configparser.py", line 1149, in _unify_values raise NoSectionError(section) from None configparser.NoSectionError: No section: 'static' (venv) 12287(1602)mattjenkins@DatapriseMBP:sophos_central_api_connector$

So /Users/mattjenkins/venv/lib/python3.8/site-packages/sophos_central_api_connector\config\sophos_config.ini is converted as so:
(venv) 12287(1602)mattjenkins@DatapriseMBP:sophos_central_api_connector$ readlink -e /Users/mattjenkins/venv/lib/python3.8/site-packages/sophos_central_api_connector\config\sophos_config.ini /Users/mattjenkins/venv/lib/python3.8/site-packages/sophos_central_api_connectorconfigsophos_config.ini

So the config file can actually be found outside the package in the site-packages directory. If I copy the config file to there and fill it out. I still get no section static and errors out.

KeyError: 'static'

Whenever i run that script the KeyError: 'static' message is what i get.
Running on Ubuntu 18 with python 3.6.9

Is the config file nor read?

Pulling alerts results in variable error

No tenant specified. Script will pull some alerts, but eventually errors as below.

ERROR:root:Response Code: 403
ERROR:root:Error Details: b'{\n "error": "forbidden",\n "correlationId": "",\n "requestId": "",\n "message": "Access is denied"\n}'
Traceback (most recent call last):
File "sophos_central_main.py", line 400, in
main(args)
File "sophos_central_main.py", line 357, in main
get_alerts(tenant_info, output, poll, days, reset_flag, page_size, splunk_creds, tenant)
File "sophos_central_main.py", line 129, in get_alerts
json_data = get_api.get_data(tenant_url_data, page_size, tenant_id, api)
File "sophos_central_api_connector\sophos_central_api_get_data.py", line 18, in get_data
ep_item_data = ep_data['items']
TypeError: 'NoneType' object is not subscriptable

Endpoint ID is "encoded" in inventory

Hi.
We notice that any endpoint ID code is actually not the real one but "encoded" and has any couple of character inverted.

If the real endpoint id is somethin live: 123456-7890...
in the inventory the enpoint is: 214365-8709

Thath could be a problem because if i want to use these id to isolate the machine, sophos api goes into and error for id not found

IOC Hunting not possible

While executing the command ioc_hunting.py we get an error message:
... 'error': 'FailedHttpRequest', 'error_message': '403 : [{"error":"forbiddenAccess","message":"Access denied for table"...

Both queries are set up in Sophos Central.

The API User is superadmin.
Other commands (sophos_central_api_*.py) working fine.

Is this an issue due to a change from Sophos Central?

Thanks in advance

Catch Error 403

Hello all,

Getting this error when trying to run inventory of all our clients to a json export:

12/01/2021 07:07:28 PM - INFO - Attempting to get inventory information
12/01/2021 07:07:29 PM - ERROR - Response Code: 403
12/01/2021 07:07:29 PM - ERROR - Error Details: b'{\n "error": "forbidden",\n "correlationId": "5867cac9-997a-43fe-bb6c-e5fd9f994503",\n "requestId": "d0ba43f0-4b54-4075-86fb-c4abdadd7ea8",\n "createdAt": "2021-01-13T02:07:29.234Z",\n "message": "Forbidden"\n}'
Traceback (most recent call last):
File "sophos_central_main.py", line 565, in
main(args)
File "sophos_central_main.py", line 483, in main
get_inventory(tenant_info, output, page_size, splunk_creds, tenant)
File "sophos_central_main.py", line 135, in get_inventory
json_data = get_api.get_data(tenant_url_data, page_size, tenant_id, api)
File "C:\Users\administrator.CORP\Documents\sophos_test\sophos_test\lib\site-packages\sophos_central_api_connector\sophos_central_api_get_data.py", line 30, in get_data
ep_item_data = ep_data['items']
TypeError: 'NoneType' object is not subscriptable

When looking at the list of tenants and where it errors out, this tenant in particular is not managed by us but shows pu in our portal. Is there a way to skip this tenant when building the list, maybe catch the error 403 and move on? I've tried adding an if-else for the tenant ID, but not sure that's sustainable long term as we get more clients in the portal that aren't managed.

Running Python 3.8.3 on Windows 10.

Thanks!

Not reading config file

Upon first run sophos_central_main.py would error with configparser.NoSectionError: No section: 'static' indicating the config file wasn't being read. Updated config\sophos_central_api_config.py paths with a prefixed dot (or removed preceeding slashes).

Tenant Names with Special Characters

We've got some tenants that are using a "/" in their name causing python to not create the file and crash with the following:

16/03/2021 07:38:50 PM - INFO - Verifying output parameter passed Traceback (most recent call last): File "sophos_central_main.py", line 570, in <module> main(args) File "sophos_central_main.py", line 488, in main get_inventory(tenant_info, output, page_size, splunk_creds, tenant) File "sophos_central_main.py", line 140, in get_inventory events = api_output.process_output(output, json_data, tenant_url_data, tenant_id, api, sourcetype_value) File "C:\Users\administrator.CORP\Documents\sophos_mar2021\lib\site-packages\sophos_central_api_connector\sophos_central_api_output.py", line 23, in process_output process_output_json(json_items, filename, api) File "C:\Users\administrator.CORP\Documents\sophos_mar2021\lib\site-packages\sophos_central_api_connector\sophos_central_api_output.py", line 44, in process_output_json with open(os.path.join(final_inv_path, filename), "w", encoding='utf-8') as ep_file: FileNotFoundError: [Errno 2] No such file or directory: 'C:\\Users\\administrator.CORP\\Documents\\sophos_mar2021\\lib\\site-packages\\sophos_central_api_connector\\output\\get_inventory\\**/*** Unlimited_fdb34fa2-fa6b-4217-92e9-76294a158613.json'

Would be good to potentially have a catch statement for special characters and replace them with either underscores or allowable characters.

Thanks!