GithubHelp home page GithubHelp logo

csirtgadgets / csirtgsdk-py-v1 Goto Github PK

View Code? Open in Web Editor NEW
7.0 3.0 5.0 530 KB

the fastest way to publish threat intel

Home Page: https://csirtg.io

License: Mozilla Public License 2.0

Python 100.00%
threatintel threat-sharing

csirtgsdk-py-v1's Introduction

CSIRTG Software Development Kit for Python

The CSIRTG Software Development Kit (SDK) for Python contains library code and examples designed to enable developers to build applications using https://csirtg.io.

Installation

Ubuntu

$ sudo apt-get install -y python-dev python-pip git
$ pip install csirtgsdk

Examples

CLI Examples

Search for an indicator

$ export CSIRTG_TOKEN=1234..
$ csirtg --search example.com

Show a list of feeds (per user)

$ export CSIRTG_TOKEN=1234..
$ csirtg --user csirtgadgets --feeds

Get a feed

$ export CSIRTG_TOKEN=1234..
$ csirtg --user csirtgadgets --feed uce-urls

Create a feed

$ csirtg --user csirtgadgets --new --feed scanners --description 'a feed of port scanners'

Create an indicator within a feed

$ csirtg --user csirtgadgets --feed scanners --new --indicator 1.1.1.1 --tags scanner --comment 'this is a port scanner'

Create an attachment within a feed

$ csirtg --user csirtgadgets --feed uce-attachments --new --attachment 'fax.zip' --description 'file attached in uce email'

SDK

QuickStart

Pulling a Feed

$ export CSIRTG_TOKEN=1234..
from pprint import pprint
from csirtgsdk import feed
rv = feed('csirtgadgets/correlated')
pprint(rv)

{'created_at': '2018-01-17 22:05:04 UTC',
 'description': 'observed across multiple users feeds',
 'indicators': [{'asn': 7922.0,
                 'asn_desc': 'COMCAST CABLE COMMUNICATIONS, LLC',
                 'cc': 'US ',
                 'content': None,
                 'count': 3,
                 'created_at': '2018-10-14 14:53:13 UTC',
                 'description': 'correlated',
                 'firsttime': '2018-10-14 00:11:28 UTC',
                 'id': 12645415,
                 'indicator': '98.220.252.135',
                 'itype': 'ipv4',
                 'lasttime': '2019-01-13 04:03:45 UTC',
                 'portlist': None,
                 'provider': None,
                 'tags': ['login', 'photon', 'hacking', 'telnet', 'scanner'],
                 'updated_at': '2019-01-13 20:59:01 UTC'},
                 ...

Searching for an Indicator

from pprint import pprint
from csirtgsdk import search
rv = search('exmple.com')
pprint(rv)

[{'attachments': [],
  'comments': [],
  'created_at': '2018-01-31 11:34:30 UTC',
  'feed': 'uce-email-addresses',
  'indicator': '[email protected]',
  'lasttime': '2018-01-31 11:34:30 UTC',
  'license': {'name': 'CC BY-SA 4.0',
              'url': 'http://creativecommons.org/licenses/by-sa/4.0/'},
  'location': 'https://csirtg.io/users/csirtgadgets/feeds/uce-email-addresses',
  'portlist': None,
  'tags': ['email-address', 'uce'],
  'updated_at': '2018-01-31 11:34:30 UTC',
  'user': 'csirtgadgets'},
  ...

Create an Indicator

from pprint import pprint
from csirtgsdk import indicator_create
i = {'indicator': 'example.com', 'tags': ['ssh'], 'description': 'this is a test'}
rv = indicator_create('wes/test',i)

{'asn': 15133.0,
 'asn_desc': 'MCI COMMUNICATIONS SERVICES, INC. D/B/A VERIZON BUSINESS',
 'cc': 'US',
 'content': None,
 'count': 1,
 'created_at': '2019-01-13 21:06:13 UTC',
 'description': 'this is a test',
 'feed': 'test',
 'firsttime': '2019-01-13 21:06:13 UTC',
 'id': 13205300,
 'indicator': 'example.com',
 'itype': 'fqdn',
 'lasttime': '2019-01-13 21:06:13 UTC',
 'license': {'name': 'CC BY-SA 4.0',
             'url': 'http://creativecommons.org/licenses/by-sa/4.0/'},
 'location': 'https://csirtg.io/users/wes/feeds/test/indicators/13205300',
 'portlist': None,
 'portlist_src': None,
 'protocol': None,
 'provider': None,
 'updated_at': '2019-01-13 21:06:13 UTC',
 'user': 'wes'}

Advanced SDK

Search for an indicator

from csirtgsdk.search import Search
from pprint import pprint

# Search for an indicator
ret = Search().search('example')

# pretty print the returned data structure
pprint(ret)

Show a list of feeds (per user)

from csirtgsdk.feed import Feed
from pprint import pprint

# Return a list of feeds (per user)
ret = Feed().index('csirtgadgets')

# pprint the returned data structure
pprint(ret)

Get a feed

from csirtgsdk.feed import Feed
from pprint import pprint

# Pull a feed
ret = Feed().show('csirtgadgets', 'uce-urls', limit=25)

# pprint the returned data structure
pprint(ret)

Create a feed

from csirtgsdk.feed import Feed
from pprint import pprint

# Create a feed
ret = Feed().new('csirtgadgets', 'correlated', description='a feed of port scanners')

# pprint the returned data structure
pprint(ret)

Submit a indicator to a feed

from csirtgsdk.indicator import Indicator
from pprint import pprint

i = {
  "user": "csirtgadgets",
  "feed": "scanners",
  "indicator": "1.1.1.1",
  "tags": "scanner",
  "description": "seen port scanning (incomming, tcp, syn, blocked)",
  "portlist": "22",
  "protocol": "TCP",
  "firsttime": "2015-11-22T00:00:00Z",
  "lasttime": "2015-11-23T00:00:00Z",
  "comment": "comment text",
  "attachment": "/tmp/malware.zip"
}

# Submit an indicator
ret = Indicator(i).submit()

# pprint the returned data structure
pprint(ret)

Submit a file to a feed using a filehandle

from csirtgsdk.indicator import Indicator
from pprint import pprint

filename = '/tmp/sample.txt'

# read the file
with open(filename) as f:
   data = f.read()

# Create a dict to submit
i = {
   'user': 'csirtgadgets',
   'feed': 'uce-attachments',
   'tags': 'uce-attachment',
   'description': 'file attached to spam email',
   'attachment': filename
}

# Submit an indicator
ret = Indicator(i).submit()

# pprint the returned data structure
pprint(ret)

Submit a file to a feed using a base64 encoded string

import hashlib
import base64
from csirtgsdk.indicator import Indicator
from pprint import pprint

filename = '/tmp/sample.txt'

# read the file
with open(filename) as f:
  data = f.read()

# Create a dict to submit
i = {
  'user': 'csirtgadgets',
  'feed': 'uce-attachments',
  'indicator': hashlib.sha1(data).hexdigest(),
  'tags': 'uce-attachment',
  'description': 'file attached to spam email',
  'attachment': base64.b64encode(data),
  'attachment_name': filename
}

# Submit an indicator
ret = Indicator(i).submit()

# pprint the returned data structure
pprint(ret)

License and Copyright

Copyright (C) 2019 CSIRT Gadgets

Free use of this software is granted under the terms of the MPL2 License. For details see the file LICENSE included with the distribution.

csirtgsdk-py-v1's People

Contributors

ckrez avatar giovino avatar jeffmurphy avatar wesyoung avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar

csirtgsdk-py-v1's Issues

Creating a new feeds does not seem to work:

cli

$ wf --user giovino --new --feed scanners --description 'a feed of port scanners'

+----------+-------------------------+--------------+------------+
|   name   |       description       |   license    | updated_at |
+----------+-------------------------+--------------+------------+
| scanners | a feed of port scanners | CC BY-SA 4.0 |    None    |
+----------+-------------------------+--------------+------------+

sdk

python new_feed.py 

{u'created_at': None,
 u'description': u'a feed of port scanners',
 u'license': {u'name': u'CC BY-SA 4.0',
              u'url': u'http://creativecommons.org/licenses/by-sa/4.0/'},
 u'name': u'scanners',
 u'updated_at': None,
 u'user': u'giovino'}

Results:

wf --user giovino --feeds

+-----------+----------------------------------+--------------+-------------------------+
|    name   |           description            |   license    |        updated_at       |
+-----------+----------------------------------+--------------+-------------------------+
| test-feed | This feed is used for API test.. | CC BY-SA 4.0 | 2015-10-01 19:34:57 UTC |
|  uce-urls | URLs seen in the message body .. | CC BY-SA 4.0 | 2015-10-21 19:51:09 UTC |
+-----------+----------------------------------+--------------+-------------------------+

timestamp filtering

add "creating at" and "updated at" timestamp filters to the SDK. There is a feature request in the app in addition to this, it may not make sense to work on this feature until the app supports additional features. for this reason, assigning to Wes.

RuntimeWarning(err): line 76, in _post

When trying to submit email data to the UCE feed, I am seeing this error. It is unclear to me from the SDK output what the problem is.

Running the most recently version of the whiteface sdk as of this commit

$ cat email.eml | /opt/py-cgmail/venv/bin/python2.7 whiteface-submit.py 
2015-11-02 15:36:15,036 - INFO - __main__[64] - processing email
2015-11-02 15:36:15,045 - INFO - requests.packages.urllib3.connectionpool[756] - ... InsecurePlatformWarning
2015-11-02 15:36:15,094 - ERROR - whiteface.client[75] - None
Traceback (most recent call last):
  File "whiteface-submit.py", line 96, in <module>
    main()
  File "whiteface-submit.py", line 91, in main
    ret = observable.Observable(user=user, feed=feed, token=token, thing=url, tags='uce', comment=comment).new()
  File "/opt/py-cgmail/venv/local/lib/python2.7/site-packages/whiteface/observable.py", line 52, in new
    data = self._post(uri, data)
  File "/opt/py-cgmail/venv/local/lib/python2.7/site-packages/whiteface/client.py", line 76, in _post
    raise RuntimeWarning(err)
RuntimeWarning: None

NameError: name 'file' is not defined

csirtg --user <username> --feed test-feed --new --indicator '1.1.1.1'
Traceback (most recent call last):
  File "/home/user/code/wf-email/venv/bin/csirtg", line 9, in <module>
    load_entry_point('csirtgsdk==0+unknown', 'console_scripts', 'csirtg')()
  File "/home/user/code/wf-email/venv/lib/python3.4/site-packages/csirtgsdk/client.py", line 243, in main
    o = read_config(args)
  File "/home/user/code/wf-email/venv/lib/python3.4/site-packages/csirtgsdk/utils.py", line 17, in read_config
    f = file(args.config)
NameError: name 'file' is not defined

Improve error of nonexistent user and feed

Improve the error handling of a nonexistent user and non existent feed.

Nonexistent feed:

$ wf --user giovino --feed asdf

2015-11-03 10:52:21,618 - ERROR - whitefacesdk.client[65] - None
Traceback (most recent call last):
  File "...py-whitefacesdk/venv/bin/wf", line 9, in <module>
    load_entry_point('whitefacesdk==0-untagged.60.g8941c94', 'console_scripts', 'wf')()
  File "....whitefacesdk/client.py", line 222, in main
    data = Feed(cli).show(options['user'], options['feed'], limit=options['limit'])
  File "....whitefacesdk/feed.py", line 46, in show
    return self.client.get(uri, params={'limit': limit})
  File "/home/giovino/code/py-whitefacesdk/whitefacesdk/client.py", line 46, in get
    return self._get(uri, params=params)
  File "...whitefacesdk/client.py", line 66, in _get
    raise RuntimeWarning(err)
RuntimeWarning: None

Nonexistent user:

$ wf --user asdf --feed uce-urls

2015-11-03 10:55:56,038 - ERROR - whitefacesdk.client[65] - None
Traceback (most recent call last):
  File "...py-whitefacesdk/venv/bin/wf", line 9, in <module>
    load_entry_point('whitefacesdk==0-untagged.60.g8941c94', 'console_scripts', 'wf')()
  File "...py-whitefacesdk/whitefacesdk/client.py", line 222, in main
    data = Feed(cli).show(options['user'], options['feed'], limit=options['limit'])
  File "...py-whitefacesdk/whitefacesdk/feed.py", line 46, in show
    return self.client.get(uri, params={'limit': limit})
  File "...py-whitefacesdk/whitefacesdk/client.py", line 46, in get
    return self._get(uri, params=params)
  File "...py-whitefacesdk/whitefacesdk/client.py", line 66, in _get
    raise RuntimeWarning(err)
RuntimeWarning: None

Catch status code 500

We need to catch the status code 500 in client.py

2015-11-11 14:17:13,913 - INFO - requests.packages.urllib3.connectionpool[735] - Starting new HTTPS connection (1): whiteface.csirtgadgets.com
<Response [500]>
2015-11-11 14:17:13,969 - ERROR - whitefacesdk.client[106] - None
Traceback (most recent call last):
  File "/home/<path>/<to>/wf-email-urls.py", line 95, in <module>
    main()
  File "/home/<path>/<to>/wf-email-urls.py", line 91, in main
    ret = o.new(user=WHITEFACE_USER, feed=WHITEFACE_FEED)
  File "/opt/py-cgmail/venv/local/lib/python2.7/site-packages/whitefacesdk/observable.py", line 104, in new
    return self.client.post(uri, data)
  File "/opt/py-cgmail/venv/local/lib/python2.7/site-packages/whitefacesdk/client.py", line 72, in post
    return self._post(uri, data)
  File "/opt/py-cgmail/venv/local/lib/python2.7/site-packages/whitefacesdk/client.py", line 107, in _post
    raise RuntimeWarning(err)
RuntimeWarning: None

Better error message when missing .wf.yml

I failed to put .wf.yml in the correct location ~/ and got a misleading error message:

2015-11-03 10:09:53,623 - ERROR - whitefacesdk.client[65] - HTTP Token: Access denied.

Traceback (most recent call last):
  File "/home/giovino/code/py-whitefacesdk/venv/bin/wf", line 9, in <module>
    load_entry_point('whitefacesdk==0-untagged.60.g8941c94', 'console_scripts', 'wf')()
  File "/home/giovino/code/py-whitefacesdk/whitefacesdk/client.py", line 222, in main
    data = Feed(cli).show(options['user'], options['feed'], limit=options['limit'])
  File "/home/giovino/code/py-whitefacesdk/whitefacesdk/feed.py", line 46, in show
    return self.client.get(uri, params={'limit': limit})
  File "/home/giovino/code/py-whitefacesdk/whitefacesdk/client.py", line 46, in get
    return self._get(uri, params=params)
  File "/home/giovino/code/py-whitefacesdk/whitefacesdk/client.py", line 66, in _get
    raise RuntimeWarning(err)
RuntimeWarning: HTTP Token: Access denied.

would be better to say "cannot find .wf.yml configuration file"

limit not being applied on search

csirtg --search 1.1.1.1 --limit 1
+---------+-----------+-----------+----------+----------+----------+--------------+---------+-------------+-------------------------+
|   user  |    feed   | indicator | comments | protocol | portlist | portlist_src |   tags  | description |        updated_at       |
+---------+-----------+-----------+----------+----------+----------+--------------+---------+-------------+-------------------------+
| giovino | test-feed |  1.1.1.1  |    1     |          |          |              | scanner |             | 2015-11-24 16:21:30 UTC |
| giovino | test-feed |  1.1.1.1  |    1     |          |    22    |              | scanner |             | 2015-11-24 20:50:17 UTC |
| giovino | test-feed |  1.1.1.1  |    2     |          |          |              | malware |             | 2015-11-24 14:05:14 UTC |
| giovino | test-feed |  1.1.1.1  |          |          |          |              | scanner |             | 2016-01-18 13:31:20 UTC |

keyerror

wf --user jonahan --feed malware
Traceback (most recent call last):
  File "/usr/local/bin/wf", line 9, in <module>
    load_entry_point('whitefacesdk==0+unknown', 'console_scripts', 'wf')()
  File "/Library/Python/2.7/site-packages/whitefacesdk/client.py", line 249, in main
    format(data).write()
  File "/Library/Python/2.7/site-packages/whitefacesdk/format/table.py", line 17, in write
    for o in self.data['feed']['observables']:
KeyError: 'observables'

how do you get the attachment sample out of wf?

I was expecting to see the base64 encoded bits in the return blob

wf --search da39a3ee5e6b4b0d3255bfef95601890afd80709 --format json | python -m json.tool

{
    "observables": [
        {
            "observable": {
                "comments": [
                    {
                        "comment": {
                            "created_at": "2015-11-24 14:06:43 UTC",
                            "text": "sha512: cf83e1357eefb8bdf1542850d66d8007d620e4050b5715dc83f4a921d36ce9ce47d0d13c5d85f2b0ff8318d2877eec2f63b931bd47417a81a538327af927da3e",
                            "user": "giovino"
                        }
                    },
                    {
                        "comment": {
                            "created_at": "2015-11-24 14:06:43 UTC",
                            "text": "sha1: da39a3ee5e6b4b0d3255bfef95601890afd80709",
                            "user": "giovino"
                        }
                    }
                ],
                "created_at": "2015-11-24 14:06:43 UTC",
                "feed": "test-feed",
                "lasttime": null,
                "license": {
                    "name": "CC BY-SA 4.0",
                    "url": "http://creativecommons.org/licenses/by-sa/4.0/"
                },
                "location": "https://whiteface.csirtgadgets.com/api/users/giovino/feeds/test-feed",
                "portlist": null,
                "tags": [
                    "malware"
                ],
                "thing": "da39a3ee5e6b4b0d3255bfef95601890afd80709",
                "updated_at": "2015-11-24 14:06:43 UTC",
                "user": "giovino"
            }
        }
    ]
}

is upgrading the py-whitefacesdk supported?

If so, it's not...

pip install https://github.com/csirtgadgets/py-whitefacesdk/archive/master.tar.gz --upgrade

------------------------------------------------------------
/path/to/wf-ufw/venv/bin/pip run on Tue Nov 24 14:17:09 2015
Downloading/unpacking https://github.com/csirtgadgets/py-whitefacesdk/archive/master.tar.gz
  Downloading from URL https://github.com/csirtgadgets/py-whitefacesdk/archive/master.tar.gz
  Running setup.py (path:/tmp/pip-nVmVLQ-build/setup.py) egg_info for package from https://github.com/csirtgadgets/py-whitefacesdk/archive/master.tar.gz
    running egg_info
    creating pip-egg-info/whitefacesdk.egg-info
    writing requirements to pip-egg-info/whitefacesdk.egg-info/requires.txt
    writing pip-egg-info/whitefacesdk.egg-info/PKG-INFO
    writing top-level names to pip-egg-info/whitefacesdk.egg-info/top_level.txt
    writing dependency_links to pip-egg-info/whitefacesdk.egg-info/dependency_links.txt
    writing entry points to pip-egg-info/whitefacesdk.egg-info/entry_points.txt
    writing manifest file 'pip-egg-info/whitefacesdk.egg-info/SOURCES.txt'
    warning: manifest_maker: standard file '-c' not found

    reading manifest file 'pip-egg-info/whitefacesdk.egg-info/SOURCES.txt'
    reading manifest template 'MANIFEST.in'
    writing manifest file 'pip-egg-info/whitefacesdk.egg-info/SOURCES.txt'
Cleaning up...
Exception:
Traceback (most recent call last):
  File "/path/to/wf-ufw/venv/local/lib/python2.7/site-packages/pip/basecommand.py", line 122, in main
    status = self.run(options, args)
  File "/path/to/wf-ufw/venv/local/lib/python2.7/site-packages/pip/commands/install.py", line 278, in run
    requirement_set.prepare_files(finder, force_root_egg_info=self.bundle, bundle=self.bundle)
  File "/path/to/wf-ufw/venv/local/lib/python2.7/site-packages/pip/req.py", line 1229, in prepare_files
    req_to_install.run_egg_info()
  File "/path/to/wf-ufw/venv/local/lib/python2.7/site-packages/pip/req.py", line 330, in run_egg_info
    "%(Name)s==%(Version)s" % self.pkg_info())
  File "/path/to/wf-ufw/venv/local/lib/python2.7/site-packages/pip/_vendor/pkg_resources.py", line 2667, in parse
    reqs = list(parse_requirements(s))
  File "/path/to/wf-ufw/venv/local/lib/python2.7/site-packages/pip/_vendor/pkg_resources.py", line 2605, in parse_requirements
    line, p, specs = scan_list(VERSION,LINE_END,line,p,(1,2),"version spec")
  File "/path/to/wf-ufw/venv/local/lib/python2.7/site-packages/pip/_vendor/pkg_resources.py", line 2583, in scan_list
    "Expected ',' or end-of-list in",line,"at",line[p:]
ValueError: ("Expected ',' or end-of-list in", u'whitefacesdk==0+unknown', 'at', u'+unknown')

RuntimeError: invalid indicator (URL)

When submitting a some spam URLs using the SDK I got the following error:

2016-01-11 15:42:39,598 - ERROR - whitefacesdk.client[100] - request failed: 422
Traceback (most recent call last):
  File "wf-email-urls.py", line 99, in <module>
    main()
  File "wf-email-urls.py", line 94, in main
    'comment': comment
  File "wf-email-addresses/venv/local/lib/python2.7/site-packages/whitefacesdk/indicator.py", line 129, in submit
    return self.client.post(uri, data)
  File "wf-email-addresses/venv/local/lib/python2.7/site-packages/whitefacesdk/client.py", line 112, in post
    raise RuntimeError(err)
RuntimeError: invalid indicator: http://www.geldfa.de/3957/generic-ranitidine-online-pharmacy-canadian-zantac-compresse"]Low

Couple of thoughts:

  1. Maybe the url http://www.geldfa.de/3957/generic-ranitidine-online-pharmacy-canadian-zantac-compresse"]Low shouldn't be a invalid indicator as you are able to browse to the URL via Google Chrome.
  2. If/when the SDK does come across a invalid indicator, should it raise a "RuntimeError" and halt all processing. In this instance, there were an array of indicators and it stopped processing in the middle of the array due to the run time error. Would it be better to just log the 'invalid indicator' and continuing processing the array of indicators?

--format csv bug

$ wf --user csirtgadgets --feed port-scanners --format csv

user,feed,thing,comments,protocol,portlist,portlist_src,tags,description,updated_at,firsttime,created_at,otype,lasttime,id
Traceback (most recent call last):
  File "/home/giovino/code/py-whitefacesdk/venv/bin/wf", line 9, in <module>
    load_entry_point('whitefacesdk==0+unknown', 'console_scripts', 'wf')()
  File "/home/giovino/code/py-whitefacesdk/venv/local/lib/python2.7/site-packages/whitefacesdk/client.py", line 250, in main
    format(data).write()
  File "/home/giovino/code/py-whitefacesdk/venv/local/lib/python2.7/site-packages/whitefacesdk/format/format_csv.py", line 40, in write
    t.writerow(o)
  File "/usr/lib/python2.7/csv.py", line 152, in writerow
    return self.writer.writerow(self._dict_to_list(rowdict))
  File "/usr/lib/python2.7/csv.py", line 148, in _dict_to_list
    + ", ".join([repr(x) for x in wrong_fields]))
ValueError: dict contains fields not in fieldnames: u'itype'

Increase verbose output

Increase verbose output, especially when no data is returned.

example:

wf --user csirtgadgets --feed port-scanners -v
2016-01-09 11:11:28,437 - INFO - requests.packages.urllib3.connectionpool[735] - Starting new HTTPS connection (1): whiteface.csirtgadgets.com

AttributeError: 'module' object has no attribute 'test_simple'

Error during:

python setup.py test
running test
running egg_info
writing requirements to whiteface_sdk.egg-info/requires.txt
writing whiteface_sdk.egg-info/PKG-INFO
writing top-level names to whiteface_sdk.egg-info/top_level.txt
writing dependency_links to whiteface_sdk.egg-info/dependency_links.txt
reading manifest file 'whiteface_sdk.egg-info/SOURCES.txt'
reading manifest template 'MANIFEST.in'
writing manifest file 'whiteface_sdk.egg-info/SOURCES.txt'
running build_ext
Traceback (most recent call last):
  File "setup.py", line 27, in <module>
    test_suite = "test"
  File "/usr/lib/python2.7/distutils/core.py", line 151, in setup
    dist.run_commands()
  File "/usr/lib/python2.7/distutils/dist.py", line 953, in run_commands
    self.run_command(cmd)
  File "/usr/lib/python2.7/distutils/dist.py", line 972, in run_command
    cmd_obj.run()
  File "/usr/lib/python2.7/dist-packages/setuptools/command/test.py", line 135, in run
    self.with_project_on_sys_path(self.run_tests)
  File "/usr/lib/python2.7/dist-packages/setuptools/command/test.py", line 116, in with_project_on_sys_path
    func()
  File "/usr/lib/python2.7/dist-packages/setuptools/command/test.py", line 160, in run_tests
    testLoader = cks
  File "/usr/lib/python2.7/unittest/main.py", line 94, in __init__
    self.parseArgs(argv)
  File "/usr/lib/python2.7/unittest/main.py", line 149, in parseArgs
    self.createTests()
  File "/usr/lib/python2.7/unittest/main.py", line 158, in createTests
    self.module)
  File "/usr/lib/python2.7/unittest/loader.py", line 130, in loadTestsFromNames
    suites = [self.loadTestsFromName(name, module) for name in names]
  File "/usr/lib/python2.7/unittest/loader.py", line 103, in loadTestsFromName
    return self.loadTestsFromModule(obj)
  File "/usr/lib/python2.7/dist-packages/setuptools/command/test.py", line 36, in loadTestsFromModule
    tests.append(self.loadTestsFromName(submodule))
  File "/usr/lib/python2.7/unittest/loader.py", line 100, in loadTestsFromName
    parent, obj = obj, getattr(obj, part)
AttributeError: 'module' object has no attribute 'test_simple'

Update cli example usage

example usage:
    $ wf --search example.com
    $ wf --user wes --feeds
    $ wf --user wes --feed scanners --new --observable 1.2.3.4 --portlist 22 --tags ssh,scanner
    $ wf --user wes --feed vnc --new

mirror the readme

cli error when creating feed that already exists

when using the cli to create a feed that already exists, turns an ugly error.

$ csirtg --user user --new --feed larry1 -v
2016-01-27 16:29:42,137 - INFO - csirtgsdk.client[288] - Creating feed larry1 for user user
2016-01-27 16:29:42,142 - INFO - requests.packages.urllib3.connectionpool[735] - Starting new HTTPS connection (1): csirtg.io
/home/user/code/py-csirtgsdk/venv/local/lib/python2.7/site-packages/requests-2.6.2-py2.7.egg/requests/packages/urllib3/util/ssl_.py:90: InsecurePlatformWarning: A true SSLContext object is not available. This prevents urllib3 from configuring SSL appropriately and may cause certain SSL connections to fail. For more information, see https://urllib3.readthedocs.org/en/latest/security.html#insecureplatformwarning.
  InsecurePlatformWarning
Traceback (most recent call last):
  File "/home/user/code/py-csirtgsdk/venv/bin/csirtg", line 9, in <module>
    load_entry_point('csirtgsdk==0.0.0a1-109.gfcb38d9.dirty', 'console_scripts', 'csirtg')()
  File "/home/user/code/py-csirtgsdk/csirtgsdk/client.py", line 289, in main
    feed = Feed(cli).new(options['user'], options['feed'], description=options['description'])
  File "/home/user/code/py-csirtgsdk/csirtgsdk/feed.py", line 27, in new
    body = self.client.post(uri, data)
  File "/home/user/code/py-csirtgsdk/csirtgsdk/client.py", line 96, in post
    body = self.session.post(uri, data=data, verify=self.verify_ssl)
  File "/home/user/code/py-csirtgsdk/venv/local/lib/python2.7/site-packages/requests-2.6.2-py2.7.egg/requests/sessions.py", line 508, in post
    return self.request('POST', url, data=data, json=json, **kwargs)
  File "/home/user/code/py-csirtgsdk/venv/local/lib/python2.7/site-packages/requests-2.6.2-py2.7.egg/requests/sessions.py", line 465, in request
    resp = self.send(prep, **send_kwargs)
  File "/home/user/code/py-csirtgsdk/venv/local/lib/python2.7/site-packages/requests-2.6.2-py2.7.egg/requests/sessions.py", line 594, in send
    history = [resp for resp in gen] if allow_redirects else []
  File "/home/user/code/py-csirtgsdk/venv/local/lib/python2.7/site-packages/requests-2.6.2-py2.7.egg/requests/sessions.py", line 196, in resolve_redirects
    **adapter_kwargs
  File "/home/user/code/py-csirtgsdk/venv/local/lib/python2.7/site-packages/requests-2.6.2-py2.7.egg/requests/sessions.py", line 573, in send
    r = adapter.send(request, **kwargs)
  File "/home/user/code/py-csirtgsdk/venv/local/lib/python2.7/site-packages/requests-2.6.2-py2.7.egg/requests/adapters.py", line 415, in send
    raise ConnectionError(err, request=request)
requests.exceptions.ConnectionError: ('Connection aborted.', ResponseNotReady())

py3 attachment not picked up by api

when submitting a hash as the indicator and a b64 string, the csirtg recored gets created with the hash as the indicator but the attachment bits are slightly dropped.

feed/user required twice when using submit_bulk

It's a little odd that feed/user is required twice when submitting bulk data.

user = 'csirtgadgets'
feed = 'test-feed'

i = {
    'indicator': 'example.com',
    'feed': 'csirtgadgets',
    'user': 'test-feed',
    'comment': 'this is a test',
}

data = []

# Initiate client object
cli = Client(remote=remote, token=token, verify_ssl=verify_ssl)

# Build list of Indicator objects
for x in range(0, 5):
    data.append(
        Indicator(cli, i)
    )

# Call the submit bulk function
ret = cli.submit_bulk(data, user, feed)

Reason is, feed/user is required in the class Indicator object. While on this topic.. it would make more sense to me if user/feed was not in the indicator dictionary but given on the submit:

Indicator(cli, record).submit(user, feed)

format csv: ValueError: dict contains fields not in fieldnames: u'thing'

csirtg --user csirtgadgets --feed uce-urls -l 1 --format csv

Traceback (most recent call last):
  File "/home/x/code/py-csirtgsdk/venv/bin/csirtg", line 9, in <module>
    load_entry_point('csirtgsdk==0.0.0a1+124.g45ecf2c', 'console_scripts', 'csirtg')()
  File "/home/x/code/py-csirtgsdk/csirtgsdk/client.py", line 320, in main
    format(data).write()
  File "/home/x/code/py-csirtgsdk/csirtgsdk/format/format_csv.py", line 49, in write
    t.writerow(o)
  File "/usr/lib/python2.7/csv.py", line 152, in writerow
    return self.writer.writerow(self._dict_to_list(rowdict))
  File "/usr/lib/python2.7/csv.py", line 148, in _dict_to_list
    + ", ".join([repr(x) for x in wrong_fields]))
ValueError: dict contains fields not in fieldnames: u'thing'

cli --feed returning no results

cli:

$ wf --user csirtgadgets --feed port-scanners -d -l 1
2016-01-09 11:26:20,524 - DEBUG - whitefacesdk.client[56] - https://whiteface.csirtgadgets.com/api/users/csirtgadgets/feeds/port-scanners
2016-01-09 11:26:20,524 - DEBUG - whitefacesdk.client[57] - {'limit': '1'}
2016-01-09 11:26:20,530 - INFO - requests.packages.urllib3.connectionpool[735] - Starting new HTTPS connection (1): whiteface.csirtgadgets.com
2016-01-09 11:26:20,870 - DEBUG - requests.packages.urllib3.connectionpool[383] - "GET /api/users/csirtgadgets/feeds/port-scanners?limit=1 HTTP/1.1" 200 None

api:

curl -H "Accept: application/vnd.whiteface.v0" -H "Authorization: Token token=<redacted>" https://whiteface.csirtgadgets.com/api/users/csirtgadgets/feeds/port-scanners?limit=1
{
    "feed": {
        "created_at": "2015-11-05 13:57:04 UTC",
        "description": "hosts blocked in firewall logs",
        "indicators": [
            {
                "indicator": {
                    "comments": [
                        {
                            "comment": {
                                "created_at": "2016-01-09 16:25:48 UTC",
                                "text": null,
                                "user": "csirtgadgets"
                            }
                        }
                    ],
                    "created_at": "2016-01-09 16:25:48 UTC",
                    "description": "sourced from firewall logs (incomming, TCP, Syn, blocked)",
                    "firsttime": null,
                    "id": 137416,
                    "itype": "ipv4",
                    "lasttime": "2016-01-09 16:24:19 UTC",
                    "portlist": "1433",
                    "tags": [
                        "scanner"
                    ],
                    "thing": "114.111.166.45",
                    "updated_at": "2016-01-09 16:25:48 UTC"
                }
            }
        ],
        "license": {
            "name": "CC BY-SA 4.0",
            "url": "http://creativecommons.org/licenses/by-sa/4.0/"
        },
        "name": "port-scanners",
        "updated_at": "2015-11-05 13:57:04 UTC",
        "user": "csirtgadgets"
    }
}

empty feeds throw errers

$ wf --user a --feed fefef
Traceback (most recent call last):
  File "/usr/local/bin/wf", line 164, in <module>
    sys.exit(main())
  File "/usr/local/bin/wf", line 148, in main
    print Table(data=f.show())
  File "/usr/local/lib/python2.7/dist-packages/whiteface/format/table.py", line 15, in __repr__
    for o in self.data['feed']['observables']:
KeyError: 'observables'

Python 3: TypeError: the JSON object must be str, not 'bytes'

Look into making the sdk compatiable with Python 3

2016-01-12 15:15:58,590 - INFO - requests.packages.urllib3.connectionpool[735] - Starting new HTTPS connection (1): whiteface.csirtgadgets.com
Traceback (most recent call last):
  File "test.py", line 110, in <module>
    main()
  File "test.py", line 105, in main
    'comment': comment
  File "wf-email/venv/lib/python3.4/site-packages/whitefacesdk/indicator.py", line 130, in submit
    return self.client.post(uri, data)
  File "wf-email/venv/lib/python3.4/site-packages/whitefacesdk/client.py", line 126, in post
    body = json.loads(body.content)
  File "/usr/lib/python3.4/json/__init__.py", line 312, in loads
    s.__class__.__name__))
TypeError: the JSON object must be str, not 'bytes'

adding .decode() to line 126 in whitefacesdk/client.py addressed this error but more research would need to be done to see what all needs to be done to make this library python 3 compatible.

body = json.loads(body.content.decode())

ssl errors

/Users/wes/.virtualenvs/wf/lib/python2.7/site-packages/requests/packages/urllib3/util/ssl_.py:90: InsecurePlatformWarning: A true SSLContext object is not available. This prevents urllib3 from configuring SSL appropriately and may cause certain SSL connections to fail. For more information, see https://urllib3.readthedocs.org/en/latest/security.html#insecureplatformwarning.
  InsecurePlatformWarning

Clarify no_verify_ssl=False

conversation:

the better way might be to change it at the function level to
verify_tls=True # as the default
and just push the —no-verify logic to each of the clients
meaning, it has to trip verify_tls=False
but that might play out weird, have to test (probably should test, make it less confusing) 

Error when search returned with 0 results & --format csv

$ csirtg -q 1.1.1.56 -v --format csv
2016-01-27 16:20:53,656 - INFO - csirtgsdk.client[259] - Searching for: 1.1.1.56
2016-01-27 16:20:53,662 - INFO - requests.packages.urllib3.connectionpool[735] - Starting new HTTPS connection (1): csirtg.io
2016-01-27 16:20:53,995 - INFO - csirtgsdk.client[261] - Search returned for: 1.1.1.56
user,feed,thing,comments,protocol,portlist,portlist_src,tags,description,updated_at,firsttime,created_at,itype,lasttime,id
Traceback (most recent call last):
  File "/home/user/code/py-csirtgsdk/venv/bin/csirtg", line 9, in <module>
    load_entry_point('csirtgsdk==0.0.0a1-109.gfcb38d9.dirty', 'console_scripts', 'csirtg')()
  File "/home/user/code/py-csirtgsdk/csirtgsdk/client.py", line 263, in main
    format(ret).write()
  File "/home/user/code/py-csirtgsdk/csirtgsdk/format/format_csv.py", line 23, in write
    feedname = self.data['feed']['indicators'][0]['indicator']['feed']
IndexError: list index out of range

AttributeError: 'str' object has no attribute 'timestamp'

Why does submitting a timestamp return the following error? Bug or mis-understanding of the sdk?

from whitefacesdk.client import Client
from whitefacesdk.observable import Observable
from pprint import pprint

remote = 'https://whiteface.csirtgadgets.com/api'
token = ''
verify_ssl = True
limit = 500

record = {
    "user": "giovino",
    "feed": "test-feed",
    "observable": "1.1.1.1",
    "tags": "scanner",
    "description": "seen port scanning (incomming, tcp, syn, blocked)",
    "portlist": "22",
    "protocol": "TCP",
    "firsttime": "2015-11-22T00:00:00Z",
    "lasttime": "2015-11-23T00:00:00Z",
    "comment": {'text': "comment text"}
}

# Initiate client object
cli = Client(remote=remote, token=token, verify_ssl=verify_ssl)

# Submit an observable
ret = Observable(cli, record).submit()

# pprint the returned data structure
pprint(ret)

error:

python gabe/t.py 
Traceback (most recent call last):
  File "gabe/t.py", line 27, in <module>
    ret = Observable(cli, record).submit()
  File "/path/to/py-whitefacesdk/whitefacesdk/observable.py", line 34, in __init__
    self.args.firsttime = arrow.get(self.args.firsttime).strftime("%Y-%m-%dT%H:%M:%S.%fZ").timestamp()
AttributeError: 'str' object has no attribute 'timestamp'

ability to submit malware not tied to an observable

We need to be able to submit malware (a file) without it being tied to an observable.

Example

wf --user giovino --feed malware --new --attachment /tmp/test.exe --tags malware --comment "this is malware"

--feed needs to pull the most recent records

--feed pulls the earliest records, it needs to pull the most recent records.

wf --user giovino --feed test-feed --limit 3
+---------+-----------+---------------------------+----------+----------+----------+---------+-------------+-------------------------+
|   user  |    feed   |           thing           | comments | protocol | portlist |   tags  | description |        updated_at       |
+---------+-----------+---------------------------+----------+----------+----------+---------+-------------+-------------------------+
| giovino | test-feed | 2604:a880:800:10::fd:7001 |    1     |          |  21379   | scanner |             | 2015-10-05 15:16:15 UTC |
| giovino | test-feed | 2604:a880:800:10::fd:7001 |    1     |          |  20547   | scanner |             | 2015-10-05 15:16:16 UTC |
| giovino | test-feed | 2604:a880:800:10::fd:7001 |    1     |          |  21025   | scanner |             | 2015-10-05 15:16:16 UTC |
+---------+-----------+---------------------------+----------+----------+----------+---------+-------------+-------------------------+

Could not find any downloads that satisfy the requirement json (from -r requirements.txt (line 4))

Should json be a requirement? Isn't it a standard lib?

pip install -r requirements.txt 
Requirement already satisfied (use --upgrade to upgrade): requests>=2.0 in /usr/lib/python2.7/dist-packages (from -r requirements.txt (line 1))
Requirement already satisfied (use --upgrade to upgrade): PrettyTable>=0.7.2 in /usr/local/lib/python2.7/dist-packages/prettytable-0.7.2-py2.7.egg (from -r requirements.txt (line 2))
Downloading/unpacking pyyaml (from -r requirements.txt (line 3))
  Downloading PyYAML-3.11.tar.gz (248kB): 248kB downloaded
  Running setup.py (path:/tmp/pip_build_giovino/pyyaml/setup.py) egg_info for package pyyaml

Downloading/unpacking json (from -r requirements.txt (line 4))
  Could not find any downloads that satisfy the requirement json (from -r requirements.txt (line 4))
Cleaning up...
No distributions at all found for json (from -r requirements.txt (line 4))
Storing debug log for failure in /home/giovino/.pip/pip.log

py3 UnicodeDecodeError on file open

csirtg --user xxx --feed xxx --new --attachment samples/BL\ \&\ shipping\ documents.jar --attachment-name test.jar
Traceback (most recent call last):
  File "/home/xxx/code/wf-email/venv/bin/csirtg", line 9, in <module>
    load_entry_point('csirtgsdk==0.0.0a1+128.g73fc660', 'console_scripts', 'csirtg')()
  File "/home/xxx/code/wf-email/venv/lib/python3.4/site-packages/csirtgsdk/client.py", line 330, in main
    ret = Indicator(cli, options).submit()
  File "/home/xxx/code/wf-email/venv/lib/python3.4/site-packages/csirtgsdk/indicator.py", line 139, in submit
    attachment = self._file_to_attachment(self.args.attachment, filename=self.args.attachment_name)
  File "/home/xxx/code/wf-email/venv/lib/python3.4/site-packages/csirtgsdk/indicator.py", line 77, in _file_to_attachment
    data = f.read()
  File "/home/xxx/code/wf-email/venv/lib/python3.4/codecs.py", line 319, in decode
    (result, consumed) = self._buffer_decode(data, self.errors, final)
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xb4 in position 10: invalid start byte

Example file:

type: .zip
password: malware
url: https://dl.dropboxusercontent.com/u/10394183/BL%20%26%20shipping%20documents.jar.zip

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.