sysdiglabs / sysdig-sdk-python

Sysdig Platform Python client library

Sysdig Monitor/Secure Python client library

A Python client API for Sysdig Monitor/Sysdig Secure.

This module is a wrapper around the Sysdig Monitor/Sysdig Secure APIs. It exposes most of the sysdig REST API functionality as an easy to use and easy to install Python interface. The repository includes a rich set of examples (in the examples subdir) that quickly address several use cases.

Installation

Automatic with PyPI

pip install sdcclient

Manual (development only)

This method requires Poetry installed

git clone https://github.com/sysdiglabs/sysdig-sdk-python.git
cd python-sdc-client
poetry install

Quick start

• If you are interested in exporting metrics data from Sysdig Monitor, take a look at examples/get_data_simple.py and examples/get_data_advanced.py.
• If you want to programmatically create an alert, refer to examples/create_alert.py
• If you want to programmatically create a dashboard, refer to examples/create_dashboard.py

Usage

Note: in order to use this API you must obtain a Sysdig Monitor/Secure API token. You can get your user's token in the Sysdig Monitor API section of the settings page for monitor or secure.

The library exports two classes, SdMonitorClient and SdSecureClient that are used to connect to Sysdig Monitor/Secure and execute actions. They can be instantiated like this:

from sdcclient import SdMonitorClient

api_token = "MY_API_TOKEN"

#
# Instantiate the Sysdig Monitor client
#
client = SdMonitorClient(api_token)

For backwards compatibility purposes, a third class SdcClient is exported which is an alias of SdMonitorClient.

Once instantiated, all the methods documented below can be called on the object.

Return Values

Every method in the SdMonitorClient/SdSecureClient classes returns a list with two entries. The first one is a boolean value indicating if the call was successful. The second entry depends on the result:

• If the call was successful, it's a dictionary reflecting the json returned by the underlying REST call
• If the call failed, it's a string describing the error

For an example on how to parse this output, take a look at a simple example like get_data_simple.py

Function List & Documentation

Work in progress

Fully documented methods is in our roadmap and will be available soon.

On-Premises Installs

For On-Premises Sysdig Monitor installs, additional configuration is necessary to point to your API server rather than the default SaaS-based one, and also to easily connect when using a self-signed certificate for SSL. One way to handle this is by setting environment variables before running your Python scripts:

export SDC_URL='https://<YOUR-API-SERVER-HOSTNAME-OR-IP>'
export SDC_SSL_VERIFY='false'

Alternatively, you can specify the additional arguments in your Python scripts as you instantiate the SDC client:

client = SdMonitorClient(api_token, sdc_url='https://<YOUR-API-SERVER-HOSTNAME-OR-IP>', ssl_verify=False)

Transitioning from Python to REST

If your goal is to interact with the REST API directly, you can use this Python client library to understand the REST interactions by logging the actions it takes. This is useful because full documentation of the REST API has not yet been created; and also provides a complete example of known working operations.

• Use or modify an example, or write a new script against the Python sdcclient module.
• Log the HTTP requests made by the script.

To log all the requests made by your script in significant detail, add to your script:

import logging
import http.client
http.client.HTTPConnection.debuglevel = 1

logging.basicConfig() # you need to initialize logging, otherwise you will not see anything from requests
logging.getLogger().setLevel(logging.DEBUG)
requests_log = logging.getLogger("requests.packages.urllib3")
requests_log.setLevel(logging.DEBUG)
requests_log.propagate = True

Then run as normal.

sysdig-sdk-python's Issues

sysdig slack notification

Probably not the right place to open git issue, but i don't know where to report it :)

In sysdig when we create slack notification it asks for webhook and channel-name, but the channel-name is not related to slack channel name :( .
slack webhook lets you posts message to any slack channel (we can overwrite the default channel), it is not tied to one channel. Is it something possible configure slack channel name in sysdig to avoid posting messages to just default channel? It is easy to maintain one webhook instead of having multiple webhooks

sdclient.get_notification_ids returns keyerror for slack.

When I run:

slack = [{'type': 'SLACK', 'channel': 'ops-alerts'}]
        slack_id = sdclient.get_notification_ids(slack)
        if not slack_id[0]:
            print "Could not get IDs and hence not creating the alert"
            sys.exit(-1)
        print slack_id

or even just the create_alarm.py example
I get

{'type': 'SLACK', 'channel': 'ops-alerts'} {u'modifiedOn': 1470132611000, u'name': u'sysdig', u'createdOn': 1470132611000, u'enabled': True, u'options': {u'notifyOnOk': False, u'url': u'<REDACTED>'}, u'version': 1, u'type': u'SLACK', u'id': 2}
Traceback (most recent call last):
  File "./create_alert.py", line 35, in <module>
    res = sdclient.get_notification_ids(notify_channels)
  File "/home/giles/work/homeoffice/python-sdc-client/examples/../sdcclient/_client.py", line 131, in get_notification_ids
    if opt['channel'] == c['channel']:
KeyError: 'channel'

Seems to be a bug in the library, doesn't happen with the email and seems to return the slack channel id correctly.

Support for segmentedHistogram panel

dsds

TypeError: string indices must be integers when calling add_dashboard_panel

Hi there,

I'm attempting to add a dashboard panel to a newly-created dashboard when running this script, but I get the following error:

  File "/Users/<REDACTED>/.pyenv/versions/3.8.5/lib/python3.8/site-packages/sdcclient/monitor/_dashboards_v3.py", line 169, in add_dashboard_panel
    new_panel_id = dboard["panels"][-1]["id"] + 1
TypeError: string indices must be integers

It occurs on this line. I can confirm that my local configuration (API key, endpoint URL, instance ID) is correct since I haven't had any issues executing other functions from this SDK.

Working of edit_team

I was trying to use the function of edit team where it overrided all the existing users and added the new user to the team

So currently the previous users were all gone and only the new user I added is there in the team,

Can someone help me in checking the functionality of edit_team? Used the below

https://github.com/draios/python-sdc-client/blob/master/examples/user_team_mgmt.py

currently display name is not considered in scope expression

In creating dashboards from file, we skip scopes defined in file. For that purpose we may have to purposefully built the filter list, but that filter list does not considers variable names. Hence we cannot perfectly create a dashboards from file.

Feature request: Support PromQL queries

Our project using this client used to work with the legacy promlegacy Prometheus metrics, but recently, our instances were upgraded, causing all promlegacy metrics to disappear.

On the dashboard front, we can construct PromQL queries to continue viewing our metrics, but we would like to continue obtaining metrics from PromQL queries in a programmatic manner. Could support be added for PromQL queries to this client?

GetData Multiple Filter Conditions

Hello,

I'm currently trying to use the GetData function to extract data from Sysdig into a .csv file, and I wanted to ask if there was a method to incorporate multiple filter conditions into the current functionality? (For example, both kubernetes.namespace.name=’production’ and container.image=’nginx’ must be met). Thanks ahead of time for the insight!

Best,
Chris

Team switch using SDC Client is not possible. We should deprecate `switch_user_team` method

When our App is used thorough browser, session_Ids are maintained through cookies and sessions are persistent between calls.
Current Team is part of the session and it get loaded from DB (lastUsedTeam) on user login.

On the other hand, when we have requests with SDC_TOKEN instead of cookies, sessions are not persistent, and they last only until request is executed, after which they are destroyed.
That is basically the reason why we have SDC_TOKEN for each (user, team) combination, because SDC_TOKEN identifies USER + TEAM.

Since SDC Client is using SDC_TOKEN, switching team is possible only by acquiring new SDC_TOKEN token for that team.

switch_user_team implementation that we have now is wrong. The effect of its execution is that is it is changing the team for that session, but that session will be destroyed at the end of that request. Next request executed with SDC_TOKEN will again have session for previous team.
Since our client is based on SDC_TOKEN, if someone want to change the team, it should create new client, with new token.

That said we should delete switch_user_team method completely, since it is redundant and can cause misleading feel that team is actually changed.

update alert function

Hi,
I was looking for update alert function, but I don't see such function. Is it available or not?

README debug code is Python 2-specific

We should add a Python 3 version of the code after "To log all the requests made by your script in significant detail..."

update_alert example

Hello, I am trying to get update_alert going, since it would be a useful inclusion in our deployment process, but while playing in bpython, it seems update_alert is not an attribute of SdcClient:

AttributeError: SdcClient instance has no attribute 'update_alert'

I am running with the latest version as of this writing:
sdcclient==0.5.2

Provide Python Module to silent alert by id or regex keyword

We have 10 to 20 alerts per application type and alerts name keep changing.
We Need python method which searches alerts by keyword and silent/flip alert status.
existing implementation silent by alert name really doesn't make sense. to use in the automation world. Nobody is willing to hardcode their CI/CD pipeline with alert name.

scopeExpressionList being overwritten during create_dashboard_from_file

When we create a dashboard using the create_dashboard_from_file function, we pass in a scopeExpressionList in the JSON file as per the V3 Dashboards API https://app.sysdigcloud.com/api/public/docs/index.html#tag/Dashboard-V3.

We noticed that our scopes were being overwritten to an empty list here https://github.com/sysdiglabs/sysdig-sdk-python/blob/master/sdcclient/monitor/_dashboards_v3.py#L287

Not sure if this is a bug, but we skipped this line of code and the Dashboard was created successfully and our dashboard scopes were created as expected.

Do not append "restored by" description if already existing

sysdig-sdk-python/examples/restore_alerts.py

Line 84 in 31be6e0

a['description'] += ' (updated via restore_alerts.py)'

Issue:
Each time you restore alerts "updated via restore_alerts.py" in the description is appended so you get something like: ${DESCRITPION} updated via restore_alerts.py updated via restore_alerts.py updated via restore_alerts.py

Solution:
Remove this or don't append if it already exists - If we must have it, adding a date at the end would be useful that is overwritten each new restore

sdc_url is hardcoded, doesn't allow for on premises use.

Hey guys.

I've found on the file _client.py the line:

def __init__(self, token="", sdc_url='https://app.sysdigcloud.com'):
is hardcoded meaning anyone using on premises sysdig would have to fork and edit manually.

Any chance to add a environment variable or flag to the command to make it more user friendly.

Thanks

Giles

Suggestion: Use OpenAPI spec to define REST interface

Hello!

I have spent a bit of time inspecting this repository and have to built an initial Swagger V2 spec. I am building an application in Go to automatically create alerts and dashboards so I have only implemented those endpoints. Because alerts and dashboards are mounted on different base paths, I have split apart the two into two specs.

Main API: https://app.swaggerhub.com/api/jgensler8/sysdig/1.0.0
Dashboard API: https://app.swaggerhub.com/api/jgensler8/sysdig-ui/1.0.0

Here are sample API clients in Go:
Main API: https://github.com/jgensler8/go-sysdig
Dashboard API: https://github.com/jgensler8/go-sysdig-ui

These generated APIs can still use wrapping code to make the interface a tad bit easier but, IMHO, this outweighs writing a whole REST interface per language.

Please let me know if this is helpful or if I should transfer ownership of the Spec I have written so far!

Dashboards can only be created in readonly mode

The parameter "isShared": true is ignored when creating a new dashboard.

doesn't like - in values

In [358]: dashboardFilter="kubernetes.namespace.name is 'foo-bar'"                                                                                                                                                                            

In [359]: foo=sdclient.create_dashboard_from_view(dashboardName, viewName, dashboardFilter)                                                                                                                                                   

In [360]: foo                                                                                                                                                                                                                                 
Out[360]: 
(False,
 'invalid scope: kubernetes.namespace.name is \'foo-bar\', expecting one of {"\'" \'/[\\\\w\\\\.]+/\' \'"\'}')

In [361]: 

is there any reason why a scope can't be a dict with key values? not sure why you chose converting string when expected value is a simple key value pair.

API discussion

Hey,

lets discuss things here :-)

Things that I mentioned at other places:

• I'd like to have continuous integration testing of the API (I'll do this with my client, at least. Not sure about the API rate limit though?)
• Inconsistent gzipping of responses (get_views, et al.)

Add commands to manage policies/falco rules

Add commands to manage policies and falco rules for new secure product.

We should support the following new admin actions:

• update system falco rules for all customers
• update system falco rules for a given customer
• revert a given customer's falco rules to the version for all customers
• revert the system falco rules for all customers to the version originally shipped with the product
• update user falco rules for a given customer
• get the falco rules file that shipped with the product
• get the global falco rules file for all customers
• get a specific customer's system falco rules file
• get a specific customer's user rules file
• fetch all policy events for a given customer in the last N hours
• create the default set of policies from the current system falco rules file

In addition, add "meta" commands that perform the following:

• "Initialize" a customer's policies and falco rules for secure. This includes setting up their system rules file, initializing their policies, etc.

new function "create_alert_from_jsonfile"

Hi,
I wrote one create alert function which accepts JSON file and creates alerts.
Can you please add this function to sdc client?
Benefits: You don't have to create alert python script for each alert separately. Instead store all your alerts in one json file and pass that file to function.

Function definition:-

def create_alert_from_jsonfile(self, filename,notify=None):

        with open(filename) as file:
            alert_json_data = json.load(file)
        #
        # Get the list of alerts from the server
        #
        res = requests.get(self.url + '/api/alerts', headers=self.hdrs, verify=self.ssl_verify)
        if not self.__checkResponse(res):
            return [False, self.lasterr]
        j = res.json()

        #
        # Populate the alert information,Create the new alert
        #
        for keys,values in alert_json_data.items():
            alert_json = {}
            alert_json["alert"] = values
            if notify != None:
                alert_json['alert']['notificationChannelIds'] = notify
            res = requests.post(self.url + '/api/alerts', headers=self.hdrs, data=json.dumps(alert_json), verify=self.ssl_verify)
            if not self.__checkResponse(res):
                #return [False, self.lasterr]
                print(self.lasterr)
            else:
                print(res.json())
                print("\n")

        return [True]

Example Usage:-

import os
import sys
sys.path.insert(0, os.path.join(os.path.dirname(os.path.realpath(sys.argv[0])), '..'))
from sdcclient import SdcClient

if len(sys.argv) != 2:
    print 'usage: %s <sysdig-token>' % sys.argv[0]
    print 'You can find your token at https://app.sysdigcloud.com/#/settings/user'
    sys.exit(1)

sdc_token = sys.argv[1]

#
# Instantiate the SDC client
#
sdclient = SdcClient(sdc_token)

notify_channels = [ {'type': 'OPSGENIE', 'name': 'Cloud Platform OpsGenie'},
                    {'type': 'WEBHOOK', 'name': 'Cloud Alert Manager'}
                    ]

res = sdclient.get_notification_ids(notify_channels)
if not res[0]:
    print "Could not get IDs and hence not creating the alert: " + res[1]
    sys.exit(-1)

notification_channel_ids = res[1]
res=sdclient.create_alert_from_jsonfile("data", notification_channel_ids)

if not res[0]:
    sys.exit(1)

data file(in json) :-

{
            "alert" : {
                "type" : "MANUAL",
                "name" : "prod-env host down",
                "description" : "triggers when host down",
                "severity" : 2,
                "timespan" : 300000000,
                "condition" : "timeAvg(uptime) = 0",
                "segmentBy" : ["host.mac"],
                "segmentCondition" : {
                    "type": "ANY"
                },
                "filter": "agent.tag.location='prod-env'",
                "enabled" : false,
                "annotations" : {}
            },

            "alert2" : {
                "type" : "MANUAL",
                "name" : "prod-env memory high",
                "description" : "triggers when memory high",
                "severity" : 2,
                "timespan" : 300000000,
                "condition" : "timeAvg(memory.used.percent) >= 90",
                "segmentBy" : ["host.mac"],
                "segmentCondition" : {
                    "type": "ANY"
                },
                "filter": "agent.tag.location='prod-env'",
                "enabled" : false,
                "annotations" : {}
            }
}

using filters like contains,in , not in for alert

Hi,

I am creating alerts using python-sdc-client. I wanted to use filters in alerts for conditions like "contains", "in" , "does not contain", "not in", "starts with" etc.
for ex. container.name 'contains' xyz.

I am not sure how to do that. Documentation does not explain, how to use such conditions.
Can you please help me to figure out?

res = sdclient.create_alert('APITest-KubeApi-Server down(Please Ignore)',
'this alert was automatically created using the python Sysdig Cloud library',
6,
60,
'timeAvg(uptime) = 0',
['host.hostName', 'container.name'],
'ANY',
'agent.tag.kubenode = "controller" and container.name contains "xyz"',
notification_channel_ids,
False)

Variable scope Dashboards

Hello,
is there a way to create a dashboards scope without being defined, meaning:

kubernetes.namespace.name in (var)

Thanks

IAM Authentication throws NameError on the good path

I'm getting the following exception when authenticating with IAM:

Traceback (most recent call last):
  File "/home/jenkins/workspace/Containers-Registry/sync-sysdig-alerts/vaultfiles/scripts/sysdig/sync-alerts.py", line 29, in <module>
    source_client = create_sdcclient_for_instance(source_instance)
  File "/home/jenkins/workspace/Containers-Registry/sync-sysdig-alerts/vaultfiles/scripts/sysdig/sync-alerts.py", line 25, in create_sdcclient_for_instance
    ibm_headers = IbmAuthHelper.get_headers(url, api_key, guid)
  File "/usr/local/lib/python3.7/dist-packages/sdcclient/ibm_auth_helper.py", line 15, in get_headers
    iam_token = IbmAuthHelper.__get_iam_token(url, apikey)
  File "/usr/local/lib/python3.7/dist-packages/sdcclient/ibm_auth_helper.py", line 35, in __get_iam_token
    response = self.http.post(
NameError: name 'self' is not defined

It appears that this change tries to reference self in a static method: 2808518#diff-d3358f2337a151eba234c36ba139abfa6a7a1175da19f9323488a6396afa13dcR35

It can not create 2 alerts with same description and everything else different.

Hi,
I am not sure weather it is intended behaviour or bug. I tried to create 2 alerts having different name and configuration but same description. It gives alert already exists error.
Use-case was, I was putting my runbook url in all alert descriptions. But, as all alert description were same, only one alert got created and other throws alert already exists error.

send notifications to webhook

Hi,
I am using python sdc client for generating my alerts. I wanted to send alerts to "Webhook". Looks like python sdc client doesn't have capability to send alert to "webhook". Can you please add it.
for ex.
notify_channels = [ {'type': 'OPSGENIE', 'name': 'Cloud Platform OpsGenie'},
{'type': 'WEBHOOK', 'name': 'Cloud Alert Manager'}
]

Does urllib3 dependency need to be so strict?

The urllib3 dep was recently bumped to urllib3 = "^2.2.1" which basically forces any consumer of sysdig-sdk-python to also pull in the most recent release (2.2.1) of urllib3 which can be problematic when trying to co-exist with other dependencies in an application.

As this package isn't really using urllib3 directly and is just consuming it via requests, Is there a specific reason not to just match the permissive range (urllib3>=1.21.1,<3) adopted by requests?

Update README

The README hasn't been updated for 3 years, we need to review it and update it if needed.

Update SDK to match Swagger Spec

Current Swagger Spec in Sysdig denotes a bunch of APIs (Beta included)

https://us2.app.sysdig.com/api/docs/swagger-ui.html

However, the Python Client is missing quite a few of these new API definitions.

Please update the Python SDK to true up with the Swagger spec. (And add an experimental flag to enable the Beta APIs)

send notification to opsgiene

Hi,
I wanted to send notification to OpsGenie , i declared like this and it's not working
{'type': 'OPSGENEI', 'channel': 'Cloud Platform OpsGenie'}

viewName in create_dashboard_from_view

I have problem filling in values for viewName parameter for create_dashboard_from_view function.
In your document says look for the view in my explore page.
However, I did not find anything called view on my explore page.
There is just groupings to parse my infrastructure.
Help would be appreciated thanks.

Secure add_policy.py creates a policy without a priority, which will not execute

add_policy.py successfully creates a policy, but doesn't set the priority.
This means that the policy doesn't execute.
Ideally this script needs to also allow specification of a priority, so that a working policy can be added in a single step.

As a side effect, It also causes script list_policies.py to fail

Traceback (most recent call last):
File "./list_policies.py", line 55, in
res[1]['policies'].sort(key=lambda p: priorities.index(p['id']))
File "./list_policies.py", line 55, in
res[1]['policies'].sort(key=lambda p: priorities.index(p['id']))
ValueError: 37 is not in list

(value error 37 is the ID of the newly created policy)

Note FR SMPROD-1442 has been created to improve the API call

No methods for getting all notification channels or getting a single alert

I had a need recently for both of these methods and ended up writing my own. Is this something that I can open a PR for and contribute back to the project? Just wanted to check first beforehand.

Basically, the signature for the methods I wrote:

def get_notification_channels():
  # return list of channels

def get_alert(id):
  # return dictionary containing fields for single alert identified by 'id'

Thanks!

The argument name for function get_events does not work

I tried version 0.8.0 until 0.10.0 and I always retrieve all the results when I execute the function get_events with the argument name filled in.

It seems like the arguments from_ts, to_ts and tags works.

create_alert unable to create PROMETHEUS type alerts.

We have a tool which copies alerts between different instances of sysdig using this SDK. We have found that it is unable to create PROMETHEUS type alerts that have been downloaded from another instance.

What we did

• Downloaded all alerts from Sysdig instance A using get_alerts().
• Substitute notification channels from Sysdig instance B into each alert returned by get_alerts()
• Upload each alert into Sysdig instance B using create_alert(alert_obj=source_alert)

What we expected

All the alerts from Sysdig instance A to be created in Sysdig instance B.

What happened

All alerts are created, except for the Prometheus based alerts, where we see the following error: 'Sorry, something really bad happened with your request (traceId: 03305a64b11f78fd).: system.error'

Steps to recreate

1. Instantiate a client using the SDK.
2. I beleive this code will provoke the error:

client = # instantiate this for your Sysdig instance

source_alert = {
    'id': 539493,
    'version': 35,
    'createdOn': 1603363705000,
    'modifiedOn': 1604405749000,
    'type': 'PROMETHEUS',
    'name': 'Sysdig ST Metrics Absent',
    'description': 'Env: prestage',
    'enabled': True,
    'notificationChannelIds': [87506],    # update this with a valid notification channel for your Sysdig instance
    'severity': 4,
    'timespan': 600000000,
    'customNotification': {'titleTemplate': '{{__alert_name__}} is {{__alert_status__}}', 'useNewTemplate': True},
    'notificationCount': 2,
    'teamId': 4840,
    'autoCreated': False,
    'rateOfChange': False,
    'reNotifyMinutes': 10,
    'reNotify': False,
    'invalidMetrics': [],
    'groupName': 'Default',
    'valid': True,
    'severityLabel': 'LOW',
    'condition': 'absent(ibm_containerregistry_storage) > 0',
    'customerId': 12352,
    'lastCheckTimeInMs': 1606735680000
}

ok, res = client.create_alert(alert_obj=source_alert)
        if not ok:
            print("Failed to create '" + source_alert["name"] + "'. Error: " + res)
        else: 
            print("Successfully created '" + source_alert["name"] + "'")