eclecticiq / opentaxii Goto Github PK

View Code? Open in Web Editor NEW

182.0 44.0 86.0 1.47 MB

TAXII server implementation in Python from EclecticIQ

License: BSD 3-Clause "New" or "Revised" License

Shell 0.45% Python 99.38% Dockerfile 0.18%

taxii stix cti python taxii-server threatintel threat-sharing cyber-threat-intelligence

opentaxii's Introduction

OpenTAXII

TAXII server implementation in Python from EclecticIQ.

OpenTAXII is a robust Python implementation of TAXII Services that delivers rich feature set and friendly pythonic API built on top of well designed application.

OpenTAXII is guaranteed to be compatible with Cabby, TAXII client library.

Source | Documentation | Information | Download

State of the project

We have made the decision to consider this project feature-complete. It means we still maintain it, however we focus only on bug fixes. Still, we’re very open to external contributions - if you know how to fix an issue and you can open a PR, we will be very grateful.

Getting started

See the documentation.

Getting started with OpenTAXII using Docker

OpenTAXII can also be run using docker. This guide assumes that you have access to a local or remote docker server, and won't go into the setup of docker.

To get a default (development) instance using docker

$ docker run -d -p 9000:9000 eclecticiq/opentaxii

To have the instance preloaded with example data, see the documentation on docker volumes.

NOTE: OpenTAXII is now accessible through port 9000, with data stored locally in a SQLite databases optionally using services/collections/accounts defined in data-configuration.yml

More documentation on running OpenTAXII in a container is found in the OpenTAXII Docker Documentation.

Feedback

You are encouraged to provide feedback by commenting on open issues or sending us email at [email protected]

opentaxii's People

Contributors

Stargazers

Watchers

Forkers

clever-crow-consulting dandye jiangzhw mostini cosive dicksonchow lazypareto bjigmp chorsley floatingghost opaopa6969 python3pkg madsc13ntist cjltsod xiaochao aukjan vovanec ninoseki xautomator dhkron cyhook davidonzo yangzhihobg mustafaugurhancar ivanapm jnorsa zentavr awesome-security max-static abshkd arcsector orsinium-forks iwm911 thunderingswolf marcelslotema epuzanov tanushree543 mnaderipoor binaryflesh mbellioum synsec thetraker yuk1h1ra pynpy zach-threatq 2509934810 fagan2888 ababook alyagomaa sanyakapoor danielrichdc tom-donahue marjatech l1kw1d golucky5 hartl3y94 shawnhank avibek ibluerose frankfanslc thecatoverlord deekshiththimmegowda horustaxi magsenabbethales alphalabsusa gzerphisa usama0121 atissan94 andrewbeard rdpickard seegersinvegas matrixsociety tech1media skbankjoy benjwebster rubicondimitri arikansm stevenjpearson 1260228859 aams-eam moniqueprinsloo santoshkumar89 iab-ai davidtekeshe

opentaxii's Issues

opentaxii.persistence.sqldb.SQLDatabaseAPI.get_service() incorrectly calls Service.get() instead of Service.query.get()

Found this when a unit test of mine failed with AttributeError:

ret_se = SQLDatabaseAPI.get_service(self, service_id)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <(snipped) object at 0x7fe80d007ad0>, service_id = 'test_cm_svc-A'

    def get_service(self, service_id):
>       return conv.to_service_entity(Service.get(service_id))
E       AttributeError: type object 'Service' has no attribute 'get'

../../../../../../../local/lib/python2.7/site-packages/opentaxii/persistence/sqldb/api.py:57: AttributeError
================================================================================================================== 1 failed, 3 passed in 0.53 seconds ==================================================================================================================

How could I use my own XSD to verify the XML(stix)？

I need to use the particular XSD to verify the XML, but I can not change the XSD to replace the TAXII_XMLMessageBinding_Schema_11.xsd ? T_T How to do Q_Q?

How to create a service that can be used for multiple purposes.

Like hail a taxii server which provides a single service for both discovery and collection management http://hailataxii.com/taxii-discovery-service.
I basically want to integrate stix feeds with QRadar using its Threat Intelligence plugin and it called 1 service for both discovery and collection, I followed http://www.opentaxii.org/ in which I can't find any configuration for such purpose.
For this purpose what should I do extend the public code API or just need to modify the service configuration file.

MySQL - (Probably) Schema issues causing feed corruption

Hoi there - just a quick and hopefully easy one here

I was trying to deploy OpenTAXII using MySQL as the backend, and found that content blocks and inbox messages get truncated to 64k, causing the feed to corrupt and crash any client that attempted to parse it.

Luckily I found out why - in the Persistence Database:

-> describe content_blocks;
+------------------+--------------+------+-----+---------+----------------+
| Field            | Type         | Null | Key | Default | Extra          |
+------------------+--------------+------+-----+---------+----------------+
| date_created     | datetime     | YES  |     | NULL    |                |
| id               | int(11)      | NO   | PRI | NULL    | auto_increment |
| message          | text         | YES  |     | NULL    |                |
| timestamp_label  | datetime     | YES  | MUL | NULL    |                |
| inbox_message_id | int(11)      | YES  | MUL | NULL    |                |
| content          | text         | YES  |     | NULL    |                |
| binding_id       | varchar(300) | YES  | MUL | NULL    |                |
| binding_subtype  | varchar(300) | YES  | MUL | NULL    |                |
+------------------+--------------+------+-----+---------+----------------+

content uses the TEXT type, which in MySQL is Limited to 65535 bytes - apparently using LONGTEXT would fix it

I'd go PR it, but I'm not familiar with SqlAlchemy to not break anything else

Thanks ❤️

docker won't start up

docker won't start up the image and keeps giving me this error:

6b4710cb13f6c87571d9fcd9dc37105e316209e8e5ae09847ef0cf90060b0a53
docker: Error response from daemon: invalid header field value "oci runtime error: container_linux.go:247: starting container process caused \"process_linux.go:334: running prestart hook 0 caused \\\"fork/exec /usr/bin/dockerd (deleted): no such file or directory\\\"\"\n".

Don't use built-in fuction names (type, id, ...) as variable names

This is a proposal in regard to a stylistic property of the source code rather than addressing a real bug or vulnerability.

https://github.com/Intelworks/taxii-server/blob/adding-tests/taxii_server/server.py#L29

I'd recommend not using special names like id and type as variable names. The above link is just an example, I've seen more of these overridings in taxii-server. Even though it's legal in Python, these are actually built-in functions:

https://docs.python.org/2/library/functions.html#id
https://docs.python.org/2/library/functions.html#type

In the best case this ambiguity can cause some initial confusion to the human reader of the source code, and in the worst case this may lead to programming errors (if type is expected to be the built-in function returning the type of a variable when the contents of the local type variable is expected and vice versa).

How can I poll data from the Server using cabby client.

I want to poll data from the server using cabby.
I have pushed the stix content successfully using taxii-push --host localhost:9000 --discovery /services/discovery-a --content-file ~/Downloads/OpenTAXII-master/examples/stix/stuxnet.stix.xml --binding "urn:stix.mitre.org:xml:1.1.1" --subtype custom-subtype --path /services/inbox-a --dest collection-B
But when I am polling data using taxii-poll --host localhost:9000 --discovery /services/discovery-a --binding urn:taxii.mitre.org:message:xml:1.1 --path services/poll-a -c collection-B

I get the following response.

 2016-03-17 18:47:20,468 INFO: Polling using data binding: ['urn:taxii.mitre.org:message:xml:1.1']
 2016-03-17 18:47:20,470 INFO: Sending Poll_Request to http://localhost:9000/services/poll-a
 2016-03-17 18:47:20,478 INFO: Starting new HTTP connection (1): localhost
 2016-03-17 18:47:20,488 INFO: Response received for Poll_Request from http://localhost:9000/services/poll-a
 2016-03-17 18:47:20,488 ERROR: UNAUTHORIZED:

Where I am wrong please guide me.

How to change the default configuration.

Hi,
I am following this installation guide for Opentaxii server.
http://opentaxii.readthedocs.org/en/stable/configuration.html
Now I want to bind or run the Opentaxii server on 0.0.0.0 but I don't find configuration file to change.
I have created a custom configuration file and export in the Environmental Variable but still default is running.

Next thing how can I import stix data into the server using API any example will be greatly appreciated.

Edited: I install it in virtual Environment.

How to Load External Threat Feed

I don't understand how to load external threat feeds into OpenTAXII. For example, how would I go about loading the feeds from HailATaxii into my OpenTAXII server?

opentaxii.persistence.SQLDatabaseAPI.create_inbox_message() ignores entity.message_id

create_inbox_message() creates an InboxMessage object, before adding it to the database session and committing, however, if the method's InboxMessageEntity argument named entity has its message_id attribute set, its value is not passed through to InboxMessage.message_id, even though InboxMessage does have a message_id attribute, per its model definition.

Since InboxMessage.message_id's database column is nullable, this does not raise an exception nor cause any other fatal error condition, but I believe it is an important omission and unexpected (mis)behaviour.

As with the other issues I have raised, this was noticed in unit tests against my own code - an assertion failed when I was unable to query an InboxMessage that had just been created, by the message_id of the InboxMessageEntity it was created from.

I have applied a very simple fix:

   def create_inbox_message(self, entity):

        if entity.destination_collections:
            names = json.dumps(entity.destination_collections)
        else:
            names = None

        #**NEW**
        message_id = entity.message_id if entity.message_id else None

        begin = entity.exclusive_begin_timestamp_label
        end = entity.inclusive_end_timestamp_label

        message = InboxMessage(
            original_message=entity.original_message,
            content_block_count=entity.content_block_count,
            destination_collections=names,

            service_id=entity.service_id,

            #**NEW **
            message_id=message_id,

            result_id=entity.result_id,
            record_count=entity.record_count,
            partial_count=entity.partial_count,

            subscription_collection_name=entity.subscription_collection_name,
            subscription_id=entity.subscription_id,

            exclusive_begin_timestamp_label=begin,
            inclusive_end_timestamp_label=end
        )

        self.db.session.add(message)
        self.db.session.commit()

        return conv.to_inbox_message_entity(message)

Client certificate authentication - unauthorized

Dears,

I am trying to authenticate towards my TAXII server using a client certificate + key file. The gunicorn server is started with the ca-certs that signed my client certificate.
Server certificate is well validated by the client, but the other way round, I can't get opentaxii to validate the user certificate:

taxii-poll --path https://taxi.local.test:9000/services/poll --collection collection --cert /tmp/client.crt --key /tmp/client.key

ERROR: UNAUTHORIZED

I get the same response back if I am using the cabby python library.
username / password authentication works fine thou, but it is not a possible option for me.

Is it enough to have the client cert signed by a trusted ca or is there any other setup to be done (i.e. define the clients CN's somewhere in config e.g.)?

Pardon me if the question has been answered already or if it is documented somewhere, but I couldn't find it.

many thanks in advance for your help!

BR's
irving

Make sure to minimise the 'count' function for PostgreSQL persistence

We should try to minimise the use of 'count' function for PostgreSQL persistence implementation, as it can become very costly on large collections. (e.g. in delete content blocks function)

User authorization per collection

Are there currently plans or recommended approaches for authorizing user access to particular collections? Moreover, is there a way to have some collections as read-only and some as read-write per-user?

Additionally, this would mean that users are only shown the collections they have access to when they access the discovery service.

This is something along the lines of what is discussed at TAXIIProject/django-taxii-services#35.

MemError with taxii-push command

I had been successfully pushing STIX files to my OpenTAXII until a few days ago since it started failing with the following MemError on the gunicorn service run:

{"timestamp": "2017-04-12T07:27:29.439255Z", "logger": "opentaxii.utils", "api": "opentaxii.persistence.sqldb.SQLDatabaseAPI", "event": "api.initialized", "level": "info"}
{"timestamp": "2017-04-12T07:27:29.535209Z", "logger": "opentaxii.utils", "api": "opentaxii.auth.sqldb.SQLDatabaseAPI", "event": "api.initialized", "level": "info"}
{"timestamp": "2017-04-12T07:27:29.536024Z", "logger": "opentaxii.server", "event": "opentaxii.server_configured", "level": "info"}
("'b3BlbnRheGlpOmZvcnRpbmV0'", 24)
[2017-04-12 09:27:51,420] ERROR in app: Exception on /services/inbox-a [POST]
Traceback (most recent call last):
  File "/home/fortinet/venv/local/lib/python2.7/site-packages/flask/app.py", line 1982, in wsgi_app
    response = self.full_dispatch_request()
  File "/home/fortinet/venv/local/lib/python2.7/site-packages/flask/app.py", line 1614, in full_dispatch_request
    rv = self.handle_user_exception(e)
  File "/home/fortinet/venv/local/lib/python2.7/site-packages/flask/app.py", line 1517, in handle_user_exception
    reraise(exc_type, exc_value, tb)
  File "/home/fortinet/venv/local/lib/python2.7/site-packages/flask/app.py", line 1612, in full_dispatch_request
    rv = self.dispatch_request()
  File "/home/fortinet/venv/local/lib/python2.7/site-packages/flask/app.py", line 1598, in dispatch_request
    return self.view_functions[rule.endpoint](**req.view_args)
  File "/home/fortinet/venv/local/lib/python2.7/site-packages/opentaxii/middleware.py", line 76, in wrapper
    return _process_with_service(service)
  File "/home/fortinet/venv/local/lib/python2.7/site-packages/opentaxii/middleware.py", line 142, in _process_with_service
    get_content_type(request.headers), request.data)
  File "/home/fortinet/venv/local/lib/python2.7/site-packages/opentaxii/taxii/utils.py", line 46, in parse_message
    result = validator_parser.validator.validate_string(body)
  File "/home/fortinet/venv/local/lib/python2.7/site-packages/libtaxii/validation.py", line 170, in validate_string
    etree_xml = parse(xml_string)
  File "/home/fortinet/venv/local/lib/python2.7/site-packages/libtaxii/common.py", line 36, in parse
    e = etree.parse(s, get_xml_parser()).getroot()
  File "src/lxml/lxml.etree.pyx", line 3427, in lxml.etree.parse (src/lxml/lxml.etree.c:81117)
  File "src/lxml/parser.pxi", line 1811, in lxml.etree._parseDocument (src/lxml/lxml.etree.c:117848)
  File "src/lxml/parser.pxi", line 1837, in lxml.etree._parseDocumentFromURL (src/lxml/lxml.etree.c:118195)
  File "src/lxml/parser.pxi", line 1741, in lxml.etree._parseDocFromFile (src/lxml/lxml.etree.c:117107)
  File "src/lxml/parser.pxi", line 1138, in lxml.etree._BaseParser._parseDocFromFile (src/lxml/lxml.etree.c:111653)
  File "src/lxml/parser.pxi", line 595, in lxml.etree._ParserContext._handleParseResultDoc (src/lxml/lxml.etree.c:105109)
  File "src/lxml/parser.pxi", line 706, in lxml.etree._handleParseResult (src/lxml/lxml.etree.c:106817)
  File "src/lxml/parser.pxi", line 629, in lxml.etree._raiseParseError (src/lxml/lxml.etree.c:105589)
MemoryError

Any idea hwo to fix this?
Thanks in advance,

Unsafe XML processing in taxii / query_handlers / base_handlers.py

Hey there,

See here:

https://github.com/Intelworks/taxii-server/blob/master/taxii/query_handlers/base_handlers.py#L611

Here you are using lxml's.etree's default XML parser, which does not guard against certain XML exploitation techniques that were recently remediated.

I suggest you do the following. Create a project-wide XML parser instance with safe settings and use it where applicable. Look at the most recent django-taxii-services for example:

from libtaxii.common import parse
# ...
# ...
etree_content = parse(content_block.content)

Where libtaxii.common contains the following:

def parse(s):
    """
    Uses the default parser to parse a string or file-like object

    :param s: The XML String or File-like object to parse
    :return: an etree._Element
    """

    try:
        e = etree.parse(s, get_xml_parser()).getroot()
    except IOError:
        e = etree.XML(s, get_xml_parser())

    return e


def get_xml_parser():
    """Return the XML parser currently in use.

    If one has not already been set (via :py:func:`set_xml_parser()`), a new
    ``etree.XMLParser`` is constructed with ``no_network=True`` and
    ``huge_tree=True``.
    """
    global _XML_PARSER
    if _XML_PARSER is None:
        _XML_PARSER = etree.XMLParser(attribute_defaults=False,
                                      dtd_validation=False,
                                      load_dtd=False,
                                      no_network=True,
                                      ns_clean=True,
                                      recover=False,
                                      remove_blank_text=False,
                                      remove_comments=False,
                                      remove_pis=False,
                                      strip_cdata=True,
                                      compact=True,
                                      # collect_ids=True,
                                      resolve_entities=False,
                                      huge_tree=False)

    return _XML_PARSER

How to change binding IP address?

Hi Team,

I am totally new to this OpenTaxII Git- Somehow, I successfully installed the whole set up in a ubuntu VM. But I cant access the taxii server VIA my host machine. Then i came to know the address to binded(lo Localhost). can anyone help me change this binding to run on my "eth1" interface.

I have changed the defaults.yml files. Then also its not replicating. Thanks in advance.

Postgre - Polling returns Binascii format

Sorry, gotta bother you again.

I've run into a really weird issue - I switched to Postgres as the backend as
recommended, but it seems that everything is being sent around in what looks like
python's binascii format.

I've switched to sqlite and it works now, so no urgency, it's just a bit awkward >__<

# Create tables
$ opentaxii-create-services -c config/services.yaml
$ opentaxii-create-collections -c config/collections.yaml
# Account already created
$ opentaxii-run-dev # all runs fine

# Push a test file
$ taxii-push --username USER --password PASS --dest COLL --discovery
http://localhost/services/discovery -f test_files/campaign-v-actors.xml

2017-07-27 10:31:11,025 INFO: Sending Discovery_Request to
http://localhost/services/discovery
2017-07-27 10:31:11,027 INFO: Starting new HTTP connection (1): localhost
2017-07-27 10:31:11,208 INFO: 4 services discovered
2017-07-27 10:31:11,210 INFO: Sending Inbox_Message to http://localhost/services/inbox
2017-07-27 10:31:11,219 INFO: Starting new HTTP connection (1): localhost
2017-07-27 10:31:11,393 INFO: Content block successfully pushed

$ taxii-poll --username USER --password PASS -c COLL --discovery
http://localhost/services/discovery

2017-07-27 10:31:37,044 INFO: Polling using data binding: ALL
2017-07-27 10:31:37,048 INFO: Sending Discovery_Request to
http://localhost/services/discovery
2017-07-27 10:31:37,050 INFO: Starting new HTTP connection (1): localhost
2017-07-27 10:31:37,182 INFO: 4 services discovered
2017-07-27 10:31:37,183 INFO: Sending Poll_Request to http://localhost/services/poll
2017-07-27 10:31:37,185 INFO: Starting new HTTP connection (1): localhost

\x3c737469783a535449585f5 .. SNIP SNIP .. Basically a LOT of binascii

It's weird, I can poll taxiistand just fine, so it's not the client :<

Looks like it's stored in that format in the database - is that intended?

I've attached configuration files as well, I don't think there's anything wrong with
them though

Any idea what could cause this?

Versions:
sqlalchemy 1.2.0b2 (persists on 1.1.2)
cabby 0.1.18
OpenTaxii git HEAD

I think those are the only relevant ones

services.txt
config.txt
collections.txt

AttributeError: 'list' object has no attribute 'get'

Hi all,

While running the following command

opentaxii-create-services -c config/services.yaml

I ran into the following error

Traceback (most recent call last):
  File "/usr/local/bin/opentaxii-create-services", line 11, in <module>
    load_entry_point('opentaxii==0.1.9a1', 'console_scripts', 'opentaxii-create-services')()
  File "/usr/local/lib/python3.4/dist-packages/opentaxii-0.1.9a1-py3.4.egg/opentaxii/cli/persistence.py", line 24, in create_services
    services = services_config.get('services', [])
AttributeError: 'list' object has no attribute 'get'

Can anyone help me, please?

Thanks

Make sure that OpenTAXII supports version argument in libtaxii's generate_message_id

Some time ago libtaxii added optional argument to its generate_message_id() function (https://github.com/TAXIIProject/libtaxii/blob/master/libtaxii/common.py#L82) that forces the function to create better IDs (uuid4) for TAXII 1.1 messages. OpenTAXII needs to use that feature.

opentaxii-delete-blocks SQL error

Haven't investigated yet, and not sure if the command needs to be updated to reflect any schema changes:

$ opentaxii-delete-blocks -c mycollection --begin 2017-07-01T00:00:0Z
...
sqlalchemy.exc.IntegrityError: (psycopg2.IntegrityError) update or delete on table "content_blocks" violates foreign key constraint "collection_to_content_block_content_block_id_fkey" on table "collection_to_content_block"
DETAIL:  Key (id)=(1) is still referenced from table "collection_to_content_block".
 [SQL: 'DELETE FROM content_blocks WHERE content_blocks.id IN (SELECT content_blocks.id \nFROM data_collections JOIN collection_to_content_block AS collection_to_content_block_1 ON data_collections.id = collection_to_content_block_1.collection_id JOIN content_blocks ON content_blocks.id = collection_to_content_block_1.content_block_id \nWHERE data_collections.id = %(id_1)s AND content_blocks.timestamp_label > %(timestamp_label_1)s)'] [parameters: {'id_1': 2, 'timestamp_label_1': '2017-07-01T00:00:0Z'}]

services.yml and collections.yml missing from example directory

Links in https://opentaxii.readthedocs.io/en/stable/configuration.html#creating-services-and-collections to these two files are also broken, which makes completing the short getting-started-guide/tutorial rather difficult.

OpenTAXII feeds / collections not recognised by Qradar Threat Intelligence App Platform

Hi,

When I tried to poll from my OpenTAXII server using Qradar's Threat Intelligence App platform, it throws an error - Failed to get list of collections from "https://domain:port/services/discovery". This is even though my collection ("collection") has 12 items. May I know if there's anyone who managed to connect their TAXII server with Qradar SIEM platform?

postgresql not working to connect to opentaxii keeps failing

i am running an ubuntu AWS instance and running open taxii in a virtual env on AWS and can't get to get the service to boot up with:
"gunicorn opentaxii.http:app --bind 0.0.0.0:8000"
this has worked before when i had the files and just changed their directories to another volume so data wouldn't be lost.
the config file shows:

---

domain: "localhost:9000"
support_basic_auth: yes
save_raw_inbox_messages: yes

persistence_api:
  class: opentaxii.persistence.sqldb.SQLDatabaseAPI
  parameters:
    db_connection: postgresql://***USERNAME***:***PASSWORD***@***DB INSTANCE ID***.***AWS_ENDPOINT_ADDRESS***:5432/***DB NAME***
    create_tables: yes

auth_api:
  class: opentaxii.auth.sqldb.SQLDatabaseAPI
  parameters:
    db_connection: postgresql://***USERNAME***:***PASSWORD***@***DB INSTANCE ID***.***AWS_ENDPOINT_ADDRESS***:5432/***DB NAME***
    create_tables: yes
    secret: ***this is not to be seen***

logging:
  opentaxii: info
  root: info

xml_parser_supports_huge_tree: yes

hooks:

both of the db_connection: were the exact same.
it try's to boot up the dies and says this:

[2017-11-30 00:21:45 +0000] [10495] [INFO] Starting gunicorn 19.7.1
[2017-11-30 00:21:45 +0000] [10495] [INFO] Listening at: http://0.0.0.0:8000 (10495)
[2017-11-30 00:21:45 +0000] [10495] [INFO] Using worker: sync
[2017-11-30 00:21:45 +0000] [10500] [INFO] Booting worker with pid: 10500
[2017-11-30 00:21:45 +0000] [10495] [INFO] Shutting down: Master
[2017-11-30 00:21:45 +0000] [10495] [INFO] Reason: Worker failed to boot.

it workered just fine with the data file config but now this is broken since i changed the database.
please help i need this fixed soon and i do not know what to do.

Error creating services and collections

While configuring an OpenTaxii instance and running both commands below,
opentaxii-create-collections -c collections.yml
opentaxii-create-services -c services.yml
I get this error message:
Traceback (most recent call last):
File "/home/fortinet/venv3/bin/opentaxii-create-services", line 11, in
load_entry_point('opentaxii==0.1.9a1', 'console_scripts', 'opentaxii-create-services')()
File "/home/fortinet/venv3/lib/python3.5/site-packages/opentaxii-0.1.9a1-py3.5.egg/opentaxii/cli/persistence.py", line 23, in create_services
services_config = anyconfig.load(args.config, forced_type="yaml")
File "/home/fortinet/venv3/lib/python3.5/site-packages/anyconfig/api.py", line 377, in load
**options)
File "/home/fortinet/venv3/lib/python3.5/site-packages/anyconfig/api.py", line 262, in single_load
cnf = psr.load(path_or_stream, **options)
File "/home/fortinet/venv3/lib/python3.5/site-packages/anyconfig/backend/base.py", line 290, in load
cnf = self.load_from_path(path_or_stream, container, **options)
File "/home/fortinet/venv3/lib/python3.5/site-packages/anyconfig/backend/base.py", line 424, in load_from_path
return self.load_from_stream(self.ropen(filepath), container, **kwargs)
File "/home/fortinet/venv3/lib/python3.5/site-packages/anyconfig/backend/base.py", line 75, in wrapper
return func(*args[1:], **kwargs)
File "/home/fortinet/venv3/lib/python3.5/site-packages/anyconfig/backend/yaml.py", line 167, in _yml_load
return container() if ret is None else container(ret)
ValueError: dictionary update sequence element #0 has length 8; 2 is required

Any hint is more than welcome.
Thanks in advance.
\Max

Can't reconnect until invalid transaction is rolled back

OpenTaxii server responds with ERROR to all requests following some period of inactivity - e.g. overnight. The only fix is to kill and restart the OpenTaxii server.

Using OpenTaxii with MySQL.

The error "Can't reconnect until invalid transaction is rolled back" features in the log files.

Facing ssl error while sending https request to opentaxii

Hi,

I am facing ssl error on trying to send a discovery request with https binding, I am using libtaxii as client, what could be the issue? doesnt opentaxii support https? how to make this certificate validation work?
stuck on this issue for quite long time,..any leads would be helpful.

root@vatd:~# discovery_client -u https://10.213.18.230:9001/services/discovery-a
Request:

Message Type: Discovery_Request
Message ID: 4007619629629755912

Traceback (most recent call last):
File "/usr/lib/python2.7/site-packages/libtaxii-1.1.111-py2.7.egg/libtaxii/scripts/init.py", line 375, in call
url.port)
File "/usr/lib/python2.7/site-packages/libtaxii-1.1.111-py2.7.egg/libtaxii/clients.py", line 346, in call_taxii_service2
response = urllib.request.urlopen(req)
File "/usr/lib64/python2.7/urllib2.py", line 154, in urlopen
return opener.open(url, data, timeout)
File "/usr/lib64/python2.7/urllib2.py", line 429, in open
response = self._open(req, data)
File "/usr/lib64/python2.7/urllib2.py", line 447, in _open
'_open', req)
File "/usr/lib64/python2.7/urllib2.py", line 407, in _call_chain
result = func(*args)
File "/usr/lib/python2.7/site-packages/libtaxii-1.1.111-py2.7.egg/libtaxii/clients.py", line 374, in https_open
return self.do_open(self.get_connection, req)
File "/usr/lib64/python2.7/urllib2.py", line 1200, in do_open
raise URLError(err)
URLError: <urlopen error EOF occurred in violation of protocol (_ssl.c:590)>

content_blocks hex encoded

hello,
I got a opentaxii server based on a postgres db. Everything seems working fine but content_blocs/Content is hex-encoded. I pushed in a stix xml. Following the output of a taxii-poll :

(venv) ubuntu@taxiiserver:~$ taxii-poll --host 192.168.15.84 --port 9000 --path /services/poll --collection collection --username taxii --password taxii
2017-08-08 15:35:38,465 INFO: Polling using data binding: ALL
2017-08-08 15:35:38,468 INFO: Sending Poll_Request to http://192.168.15.84:9000/services/poll
\x3c737469783a535449585f5061636b61676520786d6c6e733a7873693d22687474703a2f2f7777772e77332e6f72672f323030312f584d4c536368656d612d696e7374616e63652220786d6c6e733a737469783d22687474703a2f2f737469782e6d697472652e6f72672f737469782d312220786d6c6e733a696e64696361746f723d22687474703a2f2f737469782e6d697472652e6f72672f496e64696361746f722d322220786d6c6e733a6379626f783d22687474703a2f2f6379626f782e6d697472652e6f72672f6379626f782d322220786d6c6e733a416464726573734f626a6563743d22687474703a2f2f6379626f782e6d697472652e6f72672f6f626a6563747323416464726573734f626a6563742d322220786d6c6e733a6379626f78566f636162733d22687474703a2f2f6379626f782e6d697472652e6f72672f64656661756c745f766f636162756c61726965732d322220786d6c6e733a73746978566f636162733d22687474703a2f2f737469782e6d697472652e6f72672f64656661756c745f766f636162756c61726965732d312220786d6c6e733a6578616d706c653d22687474703a2f2f6578616d706c652e636f6d2f2220786d6c6e733a74617869693d22687474703a2f2f74617869692e6d697472652e6f72672f6d657373616765732f74617869695f786d6c5f62696e64696e672d312220786d6c6e733a74617869695f31313d22687474703a2f2f74617869692e6d697472652e6f72672f6d657373616765732f74617869695f786d6c5f62696e64696e672d312e312220786d6c6e733a7464713d22687474703a2f2f74617869692e6d697472652e6f72672f71756572792f74617869695f64656661756c745f71756572792d3122207873693a736368656d614c6f636174696f6e3d222020202020687474703a2f2f737469782e6d697472652e6f72672f737469782d31202e2e2f737469785f636f72652e7873642020202020687474703a2f2f737469782e6d697472652e6f72672f496e64696361746f722d32202e2e2f696e64696361746f722e7873642020202020687474703a2f2f6379626f782e6d697472652e6f72672f64656661756c745f766f636162756c61726965732d32202e2e2f6379626f782f6379626f785f64656661756c745f766f636162756c61726965732e7873642020202020687474703a2f2f737469782e6d697472652e6f72672f64656661756c745f766f636162756c61726965732d31202e2e2f737469785f64656661756c745f766f636162756c61726965732e7873642020202020687474703a2f2f6379626f782e6d697472652e6f72672f6f626a6563747323416464726573734f626a6563742d32202e2e2f6379626f782f6f626a656374732f416464726573735f4f626a6563742e787364222069643d226578616d706c653a535449585061636b6167652d33336665336232322d303230312d343763662d383564302d393763303231363435323864222076657273696f6e3d22312e302e31223e0a202020203c737469783a535449585f4865616465723e0a20202020202020203c737469783a5469746c653e4578616d706c652077617463686c697374207468617420636f6e7461696e7320495020696e666f726d6174696f6e2e3c2f737469783a5469746c653e0a20202020202020203c737469783a5061636b6167655f496e74656e74207873693a747970653d2273746978566f636162733a5061636b616765496e74656e74566f6361622d312e30223e496e64696361746f7273202d2057617463686c6973743c2f737469783a5061636b6167655f496e74656e743e0a202020203c2f737469783a535449585f4865616465723e0a202020203c737469783a496e64696361746f72733e0a20202020202020203c737469783a496e64696361746f72207873693a747970653d22696e64696361746f723a496e64696361746f7254797065222069643d226578616d706c653a496e64696361746f722d33336665336232322d303230312d343763662d383564302d393763303231363435323864223e0a2020202020202020202020203c696e64696361746f723a54797065207873693a747970653d2273746978566f636162733a496e64696361746f7254797065566f6361622d312e30223e49502057617463686c6973743c2f696e64696361746f723a547970653e0a2020202020202020202020203c696e64696361746f723a4465736372697074696f6e3e53616d706c65204950204164647265737320496e64696361746f7220666f7220746869732077617463686c6973742e205468697320636f6e7461696e73206f6e6520696e64696361746f722077697468206120736574206f662074687265652049502061646472657373657320696e207468652077617463686c6973742e3c2f696e64696361746f723a4465736372697074696f6e3e0a2020202020202020202020203c696e64696361746f723a4f627365727661626c652069643d226578616d706c653a4f627365727661626c652d31633739383236322d613463642d343334642d613935382d383834643639383063343539223e0a202020202020202020202020202020203c6379626f783a4f626a6563742069643d226578616d706c653a4f626a6563742d31393830636534332d386530332d343930622d383633612d656134303464313232343265223e0a20202020202020202020202020202020202020203c6379626f783a50726f70657274696573207873693a747970653d22416464726573734f626a6563743a416464726573734f626a65637454797065222063617465676f72793d22697076342d61646472223e0a2020202020202020202020202020202020202020202020203c416464726573734f626a6563743a416464726573735f56616c756520636f6e646974696f6e3d22457175616c7322206170706c795f636f6e646974696f6e3d22414e59223e31302e302e302e302323636f6d6d61232331302e302e302e312323636f6d6d61232331302e302e302e323c2f416464726573734f626a6563743a416464726573735f56616c75653e0a20202020202020202020202020202020202020203c2f6379626f783a50726f706572746965733e0a202020202020202020202020202020203c2f6379626f783a4f626a6563743e0a2020202020202020202020203c2f696e64696361746f723a4f627365727661626c653e0a20202020202020203c2f737469783a496e64696361746f723e0a202020203c2f737469783a496e64696361746f72733e0a3c2f737469783a535449585f5061636b6167653e0a20202020
\x3c737469783a535449585f5061636b61676520786d6c6e733a7873693d22687474703a2f2f7777772e77332e6f72672f323030312f584d4c536368656d612d696e7374616e63652220786d6c6e733a737469783d22687474703a2f2f737469782e6d697472652e6f72672f737469782d312220786d6c6e733a696e64696361746f723d22687474703a2f2f737469782e6d697472652e6f72672f496e64696361746f722d322220786d6c6e733a6379626f783d22687474703a2f2f6379626f782e6d697472652e6f72672f6379626f782d322220786d6c6e733a416464726573734f626a6563743d22687474703a2f2f6379626f782e6d697472652e6f72672f6f626a6563747323416464726573734f626a6563742d322220786d6c6e733a6379626f78566f636162733d22687474703a2f2f6379626f782e6d697472652e6f72672f64656661756c745f766f636162756c61726965732d322220786d6c6e733a73746978566f636162733d22687474703a2f2f737469782e6d697472652e6f72672f64656661756c745f766f636162756c61726965732d312220786d6c6e733a6578616d706c653d22687474703a2f2f6578616d706c652e636f6d2f2220786d6c6e733a74617869693d22687474703a2f2f74617869692e6d697472652e6f72672f6d657373616765732f74617869695f786d6c5f62696e64696e672d312220786d6c6e733a74617869695f31313d22687474703a2f2f74617869692e6d697472652e6f72672f6d657373616765732f74617869695f786d6c5f62696e64696e672d312e312220786d6c6e733a7464713d22687474703a2f2f74617869692e6d697472652e6f72672f71756572792f74617869695f64656661756c745f71756572792d3122207873693a736368656d614c6f636174696f6e3d222020202020687474703a2f2f737469782e6d697472652e6f72672f737469782d31202e2e2f737469785f636f72652e7873642020202020687474703a2f2f737469782e6d697472652e6f72672f496e64696361746f722d32202e2e2f696e64696361746f722e7873642020202020687474703a2f2f6379626f782e6d697472652e6f72672f64656661756c745f766f636162756c61726965732d32202e2e2f6379626f782f6379626f785f64656661756c745f766f636162756c61726965732e7873642020202020687474703a2f2f737469782e6d697472652e6f72672f64656661756c745f766f636162756c61726965732d31202e2e2f737469785f64656661756c745f766f636162756c61726965732e7873642020202020687474703a2f2f6379626f782e6d697472652e6f72672f6f626a6563747323416464726573734f626a6563742d32202e2e2f6379626f782f6f626a656374732f416464726573735f4f626a6563742e787364222069643d226578616d706c653a535449585061636b6167652d33336665336232322d303230312d343763662d383564302d393763303231363435323864222076657273696f6e3d22312e302e31223e0a202020203c737469783a535449585f4865616465723e0a20202020202020203c737469783a5469746c653e4578616d706c652077617463686c697374207468617420636f6e7461696e7320495020696e666f726d6174696f6e2e3c2f737469783a5469746c653e0a20202020202020203c737469783a5061636b6167655f496e74656e74207873693a747970653d2273746978566f636162733a5061636b616765496e74656e74566f6361622d312e30223e496e64696361746f7273202d2057617463686c6973743c2f737469783a5061636b6167655f496e74656e743e0a202020203c2f737469783a535449585f4865616465723e0a202020203c737469783a496e64696361746f72733e0a20202020202020203c737469783a496e64696361746f72207873693a747970653d22696e64696361746f723a496e64696361746f7254797065222069643d226578616d706c653a496e64696361746f722d33336665336232322d303230312d343763662d383564302d393763303231363435323864223e0a2020202020202020202020203c696e64696361746f723a54797065207873693a747970653d2273746978566f636162733a496e64696361746f7254797065566f6361622d312e30223e49502057617463686c6973743c2f696e64696361746f723a547970653e0a2020202020202020202020203c696e64696361746f723a4465736372697074696f6e3e53616d706c65204950204164647265737320496e64696361746f7220666f7220746869732077617463686c6973742e205468697320636f6e7461696e73206f6e6520696e64696361746f722077697468206120736574206f662074687265652049502061646472657373657320696e207468652077617463686c6973742e3c2f696e64696361746f723a4465736372697074696f6e3e0a2020202020202020202020203c696e64696361746f723a4f627365727661626c652069643d226578616d706c653a4f627365727661626c652d31633739383236322d613463642d343334642d613935382d383834643639383063343539223e0a202020202020202020202020202020203c6379626f783a4f626a6563742069643d226578616d706c653a4f626a6563742d31393830636534332d386530332d343930622d383633612d656134303464313232343265223e0a20202020202020202020202020202020202020203c6379626f783a50726f70657274696573207873693a747970653d22416464726573734f626a6563743a416464726573734f626a65637454797065222063617465676f72793d22697076342d61646472223e0a2020202020202020202020202020202020202020202020203c416464726573734f626a6563743a416464726573735f56616c756520636f6e646974696f6e3d22457175616c7322206170706c795f636f6e646974696f6e3d22414e59223e31302e302e302e302323636f6d6d61232331302e302e302e312323636f6d6d61232331302e302e302e323c2f416464726573734f626a6563743a416464726573735f56616c75653e0a20202020202020202020202020202020202020203c2f6379626f783a50726f706572746965733e0a202020202020202020202020202020203c2f6379626f783a4f626a6563743e0a2020202020202020202020203c2f696e64696361746f723a4f627365727661626c653e0a20202020202020203c2f737469783a496e64696361746f723e0a202020203c2f737469783a496e64696361746f72733e0a3c2f737469783a535449585f5061636b6167653e0a20202020
2017-08-08 15:35:38,691 INFO: 2 blocks polled

opentaxii-create-services: ValueError: dictionary update sequence element #0 has length 1; 2 is required

Hello,
In attempting to set up the configuration for OpenTAXII's services, the following error is encountered on running opentaxii-create-services

2017-05-11T05:53:38.428849Z [opentaxii.utils] info: api.initialized {timestamp=2017-05-11T05:53:38.428849Z, logger=opentaxii.utils, api=opentaxii.persistence.sqldb.SQLDatabaseAPI, event=api.initialized, level=info}
2017-05-11T05:53:38.433150Z [opentaxii.utils] info: api.initialized {timestamp=2017-05-11T05:53:38.433150Z, logger=opentaxii.utils, api=opentaxii.auth.sqldb.SQLDatabaseAPI, event=api.initialized, level=info}
2017-05-11T05:53:38.433341Z [opentaxii.server] info: opentaxii.server_configured {timestamp=2017-05-11T05:53:38.433341Z, logger=opentaxii.server, event=opentaxii.server_configured, level=info}
Loading: services.yml
Traceback (most recent call last):
  File "/usr/local/bin/opentaxii-create-services", line 11, in <module>
    sys.exit(create_services())
  File "/usr/local/lib/python2.7/dist-packages/opentaxii/cli/persistence.py", line 28, in create_services
    services_config)
  File "/usr/local/lib/python2.7/dist-packages/opentaxii/persistence/manager.py", line 62, in create_services_from_object
    service = blob_to_service_entity(blob)
  File "/usr/local/lib/python2.7/dist-packages/opentaxii/taxii/converters.py", line 332, in blob_to_service_entity
    properties = dict(blob)
ValueError: dictionary update sequence element #0 has length 1; 2 is required

This error is encountered both when I used the default services.yml file provided with the installation, and when utilizing a custom services.yml set up.

Any pointers on the right direction?

Thank you!

opentaxii-run-dev cannot load custom config file

Hi, I created a custom config file with different parameters (for example different domain and port) and made the command export OPENTAXII_CONFIG=/home/taxii/Programming/OpenTAXII/opentaxii/config.yml, but after the execution opentaxii-run-dev, the program always load default parameters (domain localhost and port 9000)

Thank you

Incorrect datetime value for column timestamp_label

I get the following error when attempting a taxii-push.

(1292, "Incorrect datetime value: '2017-05-18 20:44:03.599933+00:00' for column 'timestamp_label' at row 1")

The strange thing is when I run
opentaxii-run-dev
taxii-push -f sample.xml --path http://localhost:9000/services/inbox --dest collection

I do not get the error. So I guess the dev version doesn't actually update the database/only uses the default config.

It occurs when I use
gunicorn opentaxii.http:app --bind 192.168.56.2:9000
taxii-push -f sample.xml --path http://192.168.56.2:9000/services/inbox --dest collection

I do suspect it has something to do with the fact that SQLAlchemy doesnt comply with mysql datetime standard (http://stackoverflow.com/questions/15821953/mysql-and-python-datetime)

Any assistance would be appreciated, I assume the fix would be to just remove the timezone for that field somewhere...

Thanks

Question: Docker Container Download Delays with Kitematic?

I'm attempting to load the Docker OpenTAXII Image using Kitematic (https://kitematic.com) under Mac OSX after an hour and a half it's sitting at ~96%. Anyone else seeing this and have any suggestions?

Unable to Run Opentaxii with Gunicorn - "Worker failed to boot"

I am attempting to run opentaxii with Gunicorn in "production mode" according to these instructions. I receive an error message indicating Worker failed to boot. I have turned on debug logging, but am not able to get any additional debug information.

What is the problem? What else can I do to get additional debug information?

(opentaxii-venv)[root@source ~]# /usr/local/opentaxii/opentaxii-venv/bin/gunicorn opentaxii.http:app --log-level debug --log-file - --bind localhost:9000
[2016-04-20 14:51:12 +0000] [9110] [DEBUG] Current configuration:
  proxy_protocol: False
  worker_connections: 1000
  statsd_host: None
  max_requests_jitter: 0
  post_fork: <function post_fork at 0x7fb87cd50938>
  pythonpath: None
  enable_stdio_inheritance: False
  worker_class: sync
  ssl_version: 3
  suppress_ragged_eofs: True
  syslog: False
  syslog_facility: user
  when_ready: <function when_ready at 0x7fb87cd50668>
  pre_fork: <function pre_fork at 0x7fb87cd507d0>
  cert_reqs: 0
  preload_app: False
  keepalive: 2
  accesslog: None
  group: 0
  graceful_timeout: 30
  do_handshake_on_connect: False
  spew: False
  workers: 1
  proc_name: None
  sendfile: None
  pidfile: None
  umask: 0
  on_reload: <function on_reload at 0x7fb87cd50500>
  pre_exec: <function pre_exec at 0x7fb87cd50ed8>
  worker_tmp_dir: None
  post_worker_init: <function post_worker_init at 0x7fb87cd50aa0>
  limit_request_fields: 100
  on_exit: <function on_exit at 0x7fb87cd585f0>
  config: None
  secure_scheme_headers: {'X-FORWARDED-PROTOCOL': 'ssl', 'X-FORWARDED-PROTO': 'https', 'X-FORWARDED-SSL': 'on'}
  proxy_allow_ips: ['127.0.0.1']
  pre_request: <function pre_request at 0x7fb87cd580c8>
  post_request: <function post_request at 0x7fb87cd581b8>
  user: 0
  forwarded_allow_ips: ['127.0.0.1']
  worker_int: <function worker_int at 0x7fb87cd50c08>
  threads: 1
  max_requests: 0
  limit_request_line: 4094
  access_log_format: %(h)s %(l)s %(u)s %(t)s "%(r)s" %(s)s %(b)s "%(f)s" "%(a)s"
  certfile: None
  worker_exit: <function worker_exit at 0x7fb87cd58320>
  chdir: /root
  paste: None
  default_proc_name: opentaxii.http:app
  errorlog: -
  loglevel: debug
  logconfig: None
  syslog_addr: udp://localhost:514
  syslog_prefix: None
  daemon: False
  ciphers: TLSv1
  on_starting: <function on_starting at 0x7fb87cd50398>
  worker_abort: <function worker_abort at 0x7fb87cd50d70>
  bind: ['localhost:9000']
  raw_env: []
  reload: False
  check_config: False
  limit_request_field_size: 8190
  nworkers_changed: <function nworkers_changed at 0x7fb87cd58488>
  timeout: 30
  ca_certs: None
  django_settings: None
  tmp_upload_dir: None
  keyfile: None
  backlog: 2048
  logger_class: gunicorn.glogging.Logger
  statsd_prefix:
[2016-04-20 14:51:12 +0000] [9110] [INFO] Starting gunicorn 19.4.5
[2016-04-20 14:51:12 +0000] [9110] [DEBUG] Arbiter booted
[2016-04-20 14:51:12 +0000] [9110] [INFO] Listening at: http://127.0.0.1:9000 (9110)
[2016-04-20 14:51:12 +0000] [9110] [INFO] Using worker: sync
[2016-04-20 14:51:12 +0000] [9115] [INFO] Booting worker with pid: 9115
[2016-04-20 14:51:13 +0000] [9110] [DEBUG] 1 workers
[2016-04-20 14:51:13 +0000] [9110] [INFO] Shutting down: Master
[2016-04-20 14:51:13 +0000] [9110] [INFO] Reason: Worker failed to boot.

Here is some additional information about my environment. I am running a CentOS 6 box in VirtualBox on Mac OS X.

(opentaxii-venv)[root@source ~]# uname -a
Linux source 2.6.32-573.el6.x86_64 #1 SMP Thu Jul 23 15:44:03 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux

(opentaxii-venv)[root@source ~]# python
Python 2.7.8 (default, Oct  1 2015, 20:09:21)
[GCC 4.4.7 20120313 (Red Hat 4.4.7-16)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>>

(opentaxii-venv)[root@source ~]# pip list
anyconfig (0.5.0)
blinker (1.4)
cabby (0.1.10)
colorlog (2.6.1)
Flask (0.10.1)
furl (0.4.93)
gunicorn (19.4.5)
itsdangerous (0.24)
Jinja2 (2.8)
libtaxii (1.1.109)
lxml (3.6.0)
MarkupSafe (0.23)
opentaxii (0.1.5)
orderedmultidict (0.7.6)
pip (1.4.1)
PyJWT (1.4.0)
python-dateutil (2.5.2)
pytz (2016.3)
PyYAML (3.11)
requests (2.9.1)
setuptools (0.9.8)
six (1.10.0)
SQLAlchemy (1.0.12)
structlog (16.0.0)
Werkzeug (0.11.8)
wsgiref (0.1.2)

Curl to /management/auth broken

Hello guys,

I'm curling with the following command:

curl -v -H "Content-Type: application/json" -X POST -d '{"username":"admin","password":"pass"}' http://localhost:9000/management/auth>

Where admin and pass are the details specified in the accounts section of the OPENTAXII_CONFIG file. However I'm getting back this output:

*   Trying ::1...
* TCP_NODELAY set
* connect to ::1 port 9000 failed: Connection refused
*   Trying 127.0.0.1...
* TCP_NODELAY set
* Connected to localhost (127.0.0.1) port 9000 (#0)
< POST /management/auth HTTP/1.1
< Host: localhost:9000
< User-Agent: curl/7.52.1
< Accept: */*
< Content-Type: application/json
< Content-Length: 39
<
* upload completely sent off: 39 out of 39 bytes
127.0.0.1 - - [19/Oct/2017 10:20:33] "POST /management/auth HTTP/1.1" 401 -
* HTTP 1.0, assume close after body
< HTTP/1.0 401 UNAUTHORIZED
< Content-Type: text/html
< Content-Length: 339
< Server: Werkzeug/0.12.2 Python/2.7.13
< Date: Thu, 19 Oct 2017 17:20:33 GMT
<


<title>401 Unauthorized</title>
Unauthorized
The server could not verify that you are authorized to access the URL requested.  You either supplied the wrong credentials (e.g. a bad password), or your browser doesn't understand how to supply the credentials required.
* Curl_http_done: called premature == 0
* Closing connection 0

I'm curious if this is a working bug, or if auth via json token is being deprecated.

Thank you guys!

`opentaxii-delete-blocks` not deleting records from `collection_to_content_block`

Apparently when deleting content-blocks by using the python script opentaxii-delete-blocks, only the content blocks themselves are deleted.
The connection between collection and content blocks is not deleted.

To reproduce:

Push a record to a collection using cabby
Delete all content blocks from the collection by opentaxii-delete-blocks
Try to insert again -> will throw an exception.

This is the exception I see:

  "event": "Status exception",
  "logger": "opentaxii.middleware",
  "level": "warning",
  "timestamp": "2017-11-30T16:45:35.240165Z",
  "exception": "Traceback (most recent call last):
  File \"/usr/local/lib/python3.5/dist-packages/sqlalchemy/engine/base.py\", line 1182, in _execute_context
    context)
  File \"/usr/local/lib/python3.5/dist-packages/sqlalchemy/engine/default.py\", line 470, in do_execute
    cursor.execute(statement, parameters)
sqlite3.IntegrityError: UNIQUE constraint failed: collection_to_content_block.collection_id, collection_to_content_block.content_block_id

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File \"/usr/local/lib/python3.5/dist-packages/opentaxii/taxii/services/abstract.py\", line 83, in process
    response_message = handler.handle_message(self, message)
  File \"/usr/local/lib/python3.5/dist-packages/opentaxii/taxii/services/handlers/inbox_message_handlers.py\", line 126, in handle_message
    return InboxMessage11Handler.handle_message(service, request)
  File \"/usr/local/lib/python3.5/dist-packages/opentaxii/taxii/services/handlers/inbox_message_handlers.py\", line 65, in handle_message
    inbox_message_id=inbox_message.id if inbox_message else None)
  File \"/usr/local/lib/python3.5/dist-packages/opentaxii/persistence/manager.py\", line 160, in create_content
    content, collection_ids=collection_ids, service_id=service_id)
  File \"/usr/local/lib/python3.5/dist-packages/opentaxii/persistence/sqldb/api.py\", line 249, in create_content_block
    self._attach_content_to_collections(content, collection_ids)
  File \"/usr/local/lib/python3.5/dist-packages/opentaxii/persistence/sqldb/api.py\", line 267, in _attach_content_to_collections
    collections=new_collections.count())
  File \"/usr/local/lib/python3.5/dist-packages/sqlalchemy/orm/query.py\", line 3089, in count
    return self.from_self(col).scalar()
  File \"/usr/local/lib/python3.5/dist-packages/sqlalchemy/orm/query.py\", line 2843, in scalar
    ret = self.one()
  File \"/usr/local/lib/python3.5/dist-packages/sqlalchemy/orm/query.py\", line 2814, in one
    ret = self.one_or_none()
  File \"/usr/local/lib/python3.5/dist-packages/sqlalchemy/orm/query.py\", line 2784, in one_or_none
    ret = list(self)
  File \"/usr/local/lib/python3.5/dist-packages/sqlalchemy/orm/query.py\", line 2854, in __iter__
    self.session._autoflush()
  File \"/usr/local/lib/python3.5/dist-packages/sqlalchemy/orm/session.py\", line 1422, in _autoflush
    util.raise_from_cause(e)
  File \"/usr/local/lib/python3.5/dist-packages/sqlalchemy/util/compat.py\", line 203, in raise_from_cause
    reraise(type(exception), exception, tb=exc_tb, cause=cause)
  File \"/usr/local/lib/python3.5/dist-packages/sqlalchemy/util/compat.py\", line 187, in reraise
    raise value
  File \"/usr/local/lib/python3.5/dist-packages/sqlalchemy/orm/session.py\", line 1412, in _autoflush
    self.flush()
  File \"/usr/local/lib/python3.5/dist-packages/sqlalchemy/orm/session.py\", line 2192, in flush
    self._flush(objects)
  File \"/usr/local/lib/python3.5/dist-packages/sqlalchemy/orm/session.py\", line 2312, in _flush
    transaction.rollback(_capture_exception=True)
  File \"/usr/local/lib/python3.5/dist-packages/sqlalchemy/util/langhelpers.py\", line 66, in __exit__
    compat.reraise(exc_type, exc_value, exc_tb)
  File \"/usr/local/lib/python3.5/dist-packages/sqlalchemy/util/compat.py\", line 187, in reraise
    raise value
  File \"/usr/local/lib/python3.5/dist-packages/sqlalchemy/orm/session.py\", line 2276, in _flush
    flush_context.execute()
  File \"/usr/local/lib/python3.5/dist-packages/sqlalchemy/orm/unitofwork.py\", line 389, in execute
    rec.execute(self)
  File \"/usr/local/lib/python3.5/dist-packages/sqlalchemy/orm/unitofwork.py\", line 503, in execute
    self.dependency_processor.process_saves(uow, states)
  File \"/usr/local/lib/python3.5/dist-packages/sqlalchemy/orm/dependency.py\", line 1092, in process_saves
    secondary_update, secondary_delete)
  File \"/usr/local/lib/python3.5/dist-packages/sqlalchemy/orm/dependency.py\", line 1136, in _run_crud
    connection.execute(statement, secondary_insert)
  File \"/usr/local/lib/python3.5/dist-packages/sqlalchemy/engine/base.py\", line 945, in execute
    return meth(self, multiparams, params)
  File \"/usr/local/lib/python3.5/dist-packages/sqlalchemy/sql/elements.py\", line 263, in _execute_on_connection
    return connection._execute_clauseelement(self, multiparams, params)
  File \"/usr/local/lib/python3.5/dist-packages/sqlalchemy/engine/base.py\", line 1053, in _execute_clauseelement
    compiled_sql, distilled_params
  File \"/usr/local/lib/python3.5/dist-packages/sqlalchemy/engine/base.py\", line 1189, in _execute_context
    context)
  File \"/usr/local/lib/python3.5/dist-packages/sqlalchemy/engine/base.py\", line 1402, in _handle_dbapi_exception
    exc_info
  File \"/usr/local/lib/python3.5/dist-packages/sqlalchemy/util/compat.py\", line 203, in raise_from_cause
    reraise(type(exception), exception, tb=exc_tb, cause=cause)
  File \"/usr/local/lib/python3.5/dist-packages/sqlalchemy/util/compat.py\", line 186, in reraise
    raise value.with_traceback(tb)
  File \"/usr/local/lib/python3.5/dist-packages/sqlalchemy/engine/base.py\", line 1182, in _execute_context
    context)
  File \"/usr/local/lib/python3.5/dist-packages/sqlalchemy/engine/default.py\", line 470, in do_execute
    cursor.execute(statement, parameters)
sqlalchemy.exc.IntegrityError: (raised as a result of Query-invoked autoflush; consider using a session.no_autoflush block if this flush is occurring prematurely) (sqlite3.IntegrityError) UNIQUE constraint failed: collection_to_content_block.collection_id, collection_to_content_block.content_block_id [SQL: 'INSERT INTO collection_to_content_block (collection_id, content_block_id) VALUES (?, ?)'] [parameters: (1, 1)]

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File \"/usr/local/lib/python3.5/dist-packages/flask/app.py\", line 1612, in full_dispatch_request
    rv = self.dispatch_request()
  File \"/usr/local/lib/python3.5/dist-packages/flask/app.py\", line 1598, in dispatch_request
    return self.view_functions[rule.endpoint](**req.view_args)
  File \"/usr/local/lib/python3.5/dist-packages/opentaxii/middleware.py\", line 76, in wrapper
    return _process_with_service(service)
  File \"/usr/local/lib/python3.5/dist-packages/opentaxii/middleware.py\", line 154, in _process_with_service
    response_message = service.process(request.headers, taxii_message)
  File \"/usr/local/lib/python3.5/dist-packages/opentaxii/taxii/services/abstract.py\", line 89, in process
    in_response_to=message.message_id)
  File \"/usr/local/lib/python3.5/dist-packages/opentaxii/taxii/exceptions.py\", line 48, in raise_failure
    tb=tb)
  File \"/usr/local/lib/python3.5/dist-packages/six.py\", line 692, in reraise
    raise value.with_traceback(tb)
  File \"/usr/local/lib/python3.5/dist-packages/opentaxii/taxii/services/abstract.py\", line 83, in process
    response_message = handler.handle_message(self, message)
  File \"/usr/local/lib/python3.5/dist-packages/opentaxii/taxii/services/handlers/inbox_message_handlers.py\", line 126, in handle_message
    return InboxMessage11Handler.handle_message(service, request)
  File \"/usr/local/lib/python3.5/dist-packages/opentaxii/taxii/services/handlers/inbox_message_handlers.py\", line 65, in handle_message
    inbox_message_id=inbox_message.id if inbox_message else None)
  File \"/usr/local/lib/python3.5/dist-packages/opentaxii/persistence/manager.py\", line 160, in create_content
    content, collection_ids=collection_ids, service_id=service_id)
  File \"/usr/local/lib/python3.5/dist-packages/opentaxii/persistence/sqldb/api.py\", line 249, in create_content_block
    self._attach_content_to_collections(content, collection_ids)
  File \"/usr/local/lib/python3.5/dist-packages/opentaxii/persistence/sqldb/api.py\", line 267, in _attach_content_to_collections
    collections=new_collections.count())
  File \"/usr/local/lib/python3.5/dist-packages/sqlalchemy/orm/query.py\", line 3089, in count
    return self.from_self(col).scalar()
  File \"/usr/local/lib/python3.5/dist-packages/sqlalchemy/orm/query.py\", line 2843, in scalar
    ret = self.one()
  File \"/usr/local/lib/python3.5/dist-packages/sqlalchemy/orm/query.py\", line 2814, in one
    ret = self.one_or_none()
  File \"/usr/local/lib/python3.5/dist-packages/sqlalchemy/orm/query.py\", line 2784, in one_or_none
    ret = list(self)
  File \"/usr/local/lib/python3.5/dist-packages/sqlalchemy/orm/query.py\", line 2854, in __iter__
    self.session._autoflush()
  File \"/usr/local/lib/python3.5/dist-packages/sqlalchemy/orm/session.py\", line 1422, in _autoflush
    util.raise_from_cause(e)
  File \"/usr/local/lib/python3.5/dist-packages/sqlalchemy/util/compat.py\", line 203, in raise_from_cause
    reraise(type(exception), exception, tb=exc_tb, cause=cause)
  File \"/usr/local/lib/python3.5/dist-packages/sqlalchemy/util/compat.py\", line 187, in reraise
    raise value
  File \"/usr/local/lib/python3.5/dist-packages/sqlalchemy/orm/session.py\", line 1412, in _autoflush
    self.flush()
  File \"/usr/local/lib/python3.5/dist-packages/sqlalchemy/orm/session.py\", line 2192, in flush
    self._flush(objects)
  File \"/usr/local/lib/python3.5/dist-packages/sqlalchemy/orm/session.py\", line 2312, in _flush
    transaction.rollback(_capture_exception=True)
  File \"/usr/local/lib/python3.5/dist-packages/sqlalchemy/util/langhelpers.py\", line 66, in __exit__
    compat.reraise(exc_type, exc_value, exc_tb)
  File \"/usr/local/lib/python3.5/dist-packages/sqlalchemy/util/compat.py\", line 187, in reraise
    raise value
  File \"/usr/local/lib/python3.5/dist-packages/sqlalchemy/orm/session.py\", line 2276, in _flush
    flush_context.execute()
  File \"/usr/local/lib/python3.5/dist-packages/sqlalchemy/orm/unitofwork.py\", line 389, in execute
    rec.execute(self)
  File \"/usr/local/lib/python3.5/dist-packages/sqlalchemy/orm/unitofwork.py\", line 503, in execute
    self.dependency_processor.process_saves(uow, states)
  File \"/usr/local/lib/python3.5/dist-packages/sqlalchemy/orm/dependency.py\", line 1092, in process_saves
    secondary_update, secondary_delete)
  File \"/usr/local/lib/python3.5/dist-packages/sqlalchemy/orm/dependency.py\", line 1136, in _run_crud
    connection.execute(statement, secondary_insert)
  File \"/usr/local/lib/python3.5/dist-packages/sqlalchemy/engine/base.py\", line 945, in execute
    return meth(self, multiparams, params)
  File \"/usr/local/lib/python3.5/dist-packages/sqlalchemy/sql/elements.py\", line 263, in _execute_on_connection
    return connection._execute_clauseelement(self, multiparams, params)
  File \"/usr/local/lib/python3.5/dist-packages/sqlalchemy/engine/base.py\", line 1053, in _execute_clauseelement
    compiled_sql, distilled_params
  File \"/usr/local/lib/python3.5/dist-packages/sqlalchemy/engine/base.py\", line 1189, in _execute_context
    context)
  File \"/usr/local/lib/python3.5/dist-packages/sqlalchemy/engine/base.py\", line 1402, in _handle_dbapi_exception
    exc_info
  File \"/usr/local/lib/python3.5/dist-packages/sqlalchemy/util/compat.py\", line 203, in raise_from_cause
    reraise(type(exception), exception, tb=exc_tb, cause=cause)
  File \"/usr/local/lib/python3.5/dist-packages/sqlalchemy/util/compat.py\", line 186, in reraise
    raise value.with_traceback(tb)
  File \"/usr/local/lib/python3.5/dist-packages/sqlalchemy/engine/base.py\", line 1182, in _execute_context
    context)
  File \"/usr/local/lib/python3.5/dist-packages/sqlalchemy/engine/default.py\", line 470, in do_execute
    cursor.execute(statement, parameters)
opentaxii.taxii.exceptions.FailureStatus: (raised as a result of Query-invoked autoflush; consider using a session.no_autoflush block if this flush is occurring prematurely) (sqlite3.IntegrityError) UNIQUE constraint failed: collection_to_content_block.collection_id, collection_to_content_block.content_block_id [SQL: 'INSERT INTO collection_to_content_block (collection_id, content_block_id) VALUES (?, ?)'] [parameters: (1, 1)]"
}

How to Age-Out (Purge Old) Data?

Is there a mechanism to purge or age-out old data with OpenTAXII? How do I ensure that the data store doesn't grow without bounds?

Case insensitive scheme matching

Change this line on https://github.com/Intelworks/OpenTAXII/blob/master/opentaxii/taxii/services/abstract.py#L106

if scheme and not address.startswith(scheme):

into

if scheme and not address.lower().startswith(scheme.lower()):

Didn't test this but looks logical to me.

Installation problems with MySQL

Hey there, I hope I'm not doing anything stupid in my installation process, but I can't seem to get the TAXII server running with MySQL

I'm using Python3.5 if that helps. Should be compatible I think.

Process:

git clone [repo URL]
cd OpenTAXII
sudo python3 setup.py install
mysql -u [username] -p
  # Create dbs taxiipersist and taxiiauth
export OPENTAXII_CONFIG=[full path]
opentaxii-create-services -c services.yaml

I then get a big damn list of errors that I piped out, attached as errors.txt

All my config files are attached as well, just in case I somehow messed up there.

Thanks in advance :)

config_files.zip
errors.txt

poll data so slow when i pushed 5milion data to opentaxii

I think i need create mysql index.But i do not know which field the index create on.

opentaxii.persistence.sqldb.SQLDatabaseAPI.update_service() unconditionally returns None instead of a ServiceEntity as probably intended

Also discovered by one of my unit tests.

The line https://github.com/EclecticIQ/OpenTAXII/blob/master/opentaxii/persistence/sqldb/api.py#L70 assigns the return value of db.session.add() to the local variable service. However, the .add() method, having no return statement, implicitly returns None. This means that None is unconditionally assigned to service, and .update_service() returns the return value of return conv.to_service_entity(service), which in turn unconditionally returns None instead of a ServiceEntity.

The return value of db.session.add() should be ignored, which means service will still be bound to a Service object, and the subsequent call to .to_service_entity() will be able to return a ServiceEntity which will in turn be returned by .update_service(), which is almost certainly the intended behaviour.

Use Python lint tool prior to public release

Same motivation as posited in eclecticiq/cabby#16 applies here.

(this was run on taxii-server-adding-tests)

$ find -name '*.py' -exec pyflakes {} \;
./tests/services/test_discovery.py:11: 'from fixtures import *' used; unable to detect undefined names
./tests/services/fixtures.py:1: 'from libtaxii.constants import *' used; unable to detect undefined names
./tests/services/utils.py:4: 'from taxii_server.taxii.http import *' used; unable to detect undefined names
./tests/services/test_inbox.py:14: 'from fixtures import *' used; unable to detect undefined names
./tests/services/test_inbox.py:60: local variable 'collections' is assigned to but never used
./tests/test_http.py:7: 'ST_NOT_FOUND' imported but unused
./tests/test_http.py:7: 'ST_SUCCESS' imported but unused
./tests/test_http.py:12: 'from taxii_server.taxii.http import *' used; unable to detect undefined names
./taxii_server/middleware.py:3: 'jsonify' imported but unused
./taxii_server/middleware.py:6: 'from taxii.http import *' used; unable to detect undefined names
./taxii_server/middleware.py:15: 'configure_logging' imported but unused
./taxii_server/server.py:1: 'namedtuple' imported but unused
./taxii_server/options.py:2: 'sys' imported but unused
./taxii_server/options.py:7: 'ContentBinding' imported but unused
./taxii_server/options.py:8: 'SimpleRenderer' imported but unused
./taxii_server/taxii/services/collection_management.py:8: redefinition of unused 'CollectionInformationRequestHandler' from line 8
./taxii_server/taxii/services/collection_management.py:8: redefinition of unused 'SubscriptionRequestHandler' from line 8
./taxii_server/taxii/services/collection_management.py:29: undefined name 'DiscoveryService'
./taxii_server/taxii/services/handlers/inbox_message_handlers.py:4: 'from libtaxii.constants import *' used; unable to detect undefined names
./taxii_server/taxii/services/handlers/inbox_message_handlers.py:6: 'StatusMessageException' imported but unused
./taxii_server/taxii/services/handlers/poll_request_handlers.py:8: 'from libtaxii.constants import *' used; unable to detect undefined names
./taxii_server/taxii/services/handlers/poll_request_handlers.py:94: local variable 'response' is assigned to but never used
./taxii_server/taxii/services/handlers/base_handlers.py:1: 'from libtaxii.constants import *' used; unable to detect undefined names
./taxii_server/taxii/services/handlers/base_handlers.py:4: 'StatusMessageException' imported but unused
./taxii_server/taxii/services/handlers/poll_fulfilment_request_handlers.py:6: 'from libtaxii.constants import *' used; unable to detect undefined names
./taxii_server/taxii/services/handlers/__init__.py:2: 'DiscoveryRequestHandler' imported but unused
./taxii_server/taxii/services/handlers/__init__.py:3: 'InboxMessageHandler' imported but unused
./taxii_server/taxii/services/handlers/subscription_request_handlers.py:7: 'from libtaxii.constants import *' used; unable to detect undefined names
./taxii_server/taxii/services/abstract.py:40: local variable 'e' is assigned to but never used
./taxii_server/taxii/services/__init__.py:2: 'InboxService' imported but unused
./taxii_server/taxii/services/__init__.py:3: 'DiscoveryService' imported but unused
./taxii_server/taxii/transform.py:2: 'tdq' imported but unused
./taxii_server/taxii/transform.py:6: 'from libtaxii.constants import *' used; unable to detect undefined names
./taxii_server/taxii/entities.py:3: 'from bindings import *' used; unable to detect undefined names
./taxii_server/taxii/http.py:2: 'from libtaxii.constants import *' used; unable to detect undefined names
./taxii_server/taxii/http.py:4: 'from bindings import *' used; unable to detect undefined names
./taxii_server/taxii/http.py:5: 'raise_failure' imported but unused
./taxii_server/taxii/http.py:6: 'parse_message' imported but unused
./taxii_server/taxii/bindings.py:5: 'from libtaxii.constants import *' used; unable to detect undefined names
./taxii_server/taxii/utils.py:2: 'dateutil' imported but unused
./taxii_server/taxii/utils.py:5: 'from libtaxii.constants import *' used; unable to detect undefined names
./taxii_server/taxii/status.py:1: 'from libtaxii.constants import *' used; unable to detect undefined names
./taxii_server/taxii/status.py:6: 'deepcopy' imported but unused
./taxii_server/taxii/status.py:8: 'from http import *' used; unable to detect undefined names
./taxii_server/persistence/sql/models.py:1: 'backref' imported but unused
./taxii_server/persistence/sql/models.py:2: 'Index' imported but unused
./taxii_server/persistence/sql/models.py:2: 'UniqueConstraint' imported but unused
./taxii_server/persistence/sql/models.py:2: 'PrimaryKeyConstraint' imported but unused
./taxii_server/persistence/sql/models.py:3: 'Enum' imported but unused
./taxii_server/persistence/sql/models.py:3: 'Date' imported but unused
./taxii_server/persistence/sql/models.py:6: 'event' imported but unused
./taxii_server/persistence/sql/models.py:86: redefinition of function 'content_binding' from line 82
./taxii_server/persistence/sql/models.py:127: redefinition of function 'supported_content' from line 123
./taxii_server/persistence/sql/__init__.py:1: 'sqlalchemy' imported but unused
./taxii_server/persistence/sql/__init__.py:4: 'from taxii_server.taxii.entities import *' used; unable to detect undefined names
./taxii_server/utils.py:1: 'os' imported but unused
./taxii_server/utils.py:2: 'sys' imported but unused
./taxii_server/utils.py:3: 'logging' imported but unused
./taxii_server/utils.py:4: 'structlog' imported but unused

Make URL harvesting impossible

At the moment it is possible to discover URLs of the protected services.

When client sends a request without a token (or with invalid token), OpenTAXII doesn't know if authentication is required for the service, so first it has to configure service instance. If a service does not exist, 404 NOT FOUND will be returned. if service is configured and requires authentication, UNAUTHORIZED will be returned.

So it is possible to walk all possible URLs and log the ones with UNAUTHORIZED. This is a configuration leak.

Example API queries on "discovery" and "collection" failure

Example API queries on "discovery" and "collection" fail freshly installed instanced OpenTAXII server (documentation steps)

$ pip install cabby
Requirement already satisfied: cabby in ./venv/lib/python2.7/site-packages
Requirement already satisfied: six in ./venv/lib/python2.7/site-packages (from cabby)
Requirement already satisfied: requests>=2.7.0 in ./venv/lib/python2.7/site-packages (from cabby)
Requirement already satisfied: pytz>=2014.10 in ./venv/lib/python2.7/site-packages (from cabby)
Requirement already satisfied: colorlog in ./venv/lib/python2.7/site-packages (from cabby)
Requirement already satisfied: libtaxii>=1.1.109 in ./venv/lib/python2.7/site-packages (from cabby)
Requirement already satisfied: furl>=0.4.7 in ./venv/lib/python2.7/site-packages (from cabby)
Requirement already satisfied: lxml>=2.2.3 in ./venv/lib/python2.7/site-packages (from libtaxii>=1.1.109->cabby)
Requirement already satisfied: python-dateutil>=1.4.1 in ./venv/lib/python2.7/site-packages (from libtaxii>=1.1.109->cabby)
Requirement already satisfied: orderedmultidict>=0.7.8 in ./venv/lib/python2.7/site-packages (from furl>=0.4.7->cabby)

$ taxii-discovery --path http://localhost:9000/services/discovery-a
2017-03-27 15:15:07,511 INFO: Sending Discovery_Request to http://localhost:9000/services/discovery-a
2017-03-27 15:15:07,537 ERROR: HTTP Error: status code 404

$ taxii-collections --path http://localhost:9000/services/collection-management-a
2017-03-27 15:15:24,341 INFO: Sending Collection_Information_Request to http://localhost:9000/services/collection-management-a
2017-03-27 15:15:24,371 ERROR: HTTP Error: status code 404

We checked the server is alive:
$ curl http://localhost:9000/management/health
{
"alive": true
}

Many thanks for your help.

Using a more permanent database

Is there any documentation about how to configure OpenTAXII to use a database rather than SQLite.
I'm looking to 'productionise' OpenTAXII.
Thanks.

Service FEED not available for OpenTAXII published sites

I am trying to access some data that was successfully pushed in the OpenTAXII server data collection. The said data can be retrieved with taxii-poll CLI, but this is failing if queried from an Anomali-STAXX server.

The error message is: "Service FEED not available for site https://domain:9000/services/discovery-a" while accessing via https://opentaxii.etlab.net:9000/services/discovery-a

And is "Service FEED not available for site http://opentaxii.etlab.net:9000/services/poll-a" while accessing via http://opentaxii.etlab.net:9000/services/poll-a

For reference, here is the CLI command used to push content into OpenTAXII:
taxii-push -f indicators.xml --path http://domain:9000/services/inbox-a --dest collection-A
And here is the CLI command used to access the same data from OpenTAXII:
taxii-poll --path http://domain:9000/services/poll-a -c collection-A --username opentaxii --password test

Many thanks for your help.

Collection Volume describes a rate not a full count

In TAXII 1.1 spec "Collection Volume" is defined as

"... indicates the typical number of records added to this Data Collection daily. This represents a "typical" value and the producer is under no obligation to keep the Data Collection volume at the given level."

In current implementation volume represents total count of content blocks in a collection

SSL23_GET_SERVER_HELLO ERROR when stix connected to opentaxii

when i use stix server discover my opentaxii server. it shows error:140770FC:SSL routines:SSL23_GET_SERVER_HELLO:unknown protocol.

I run my opentaxii server follow the document http://www.opentaxii.org/en/stable/installation.html

My opentaxii version is 0.18

ERROR: Extra content at the end of the document (taxii-poll command)

I get

2017-04-12 09:43:11,079 ERROR: Extra content at the end of the document, line 655441, column 53 (line 655441)

After the taxii-poll command prints out lines extracted from OpenTAXII server.
Many thanks in advance for your help.

Could not load stix file error (Probably huge XML tree)

Hi,

I ran into an error while trying to pull data from a TAXII Server in STIX Format. This is the traceback

Traceback (most recent call last):
  File "/usr/local/lib/python3.4/dist-packages/misp_stix_converter-0.2.9-py3.4.egg/misp_stix_converter/converters/convert.py", line 156, in manage_stix
    stix_package = STIXPackage.from_xml(stix)
  File "/usr/local/lib/python3.4/dist-packages/stix/core/stix_package.py", line 249, in from_xml
    return entity_parser.parse_xml(xml_file, encoding=encoding)
  File "/usr/local/lib/python3.4/dist-packages/mixbox/parser.py", line 179, in parse_xml
    xml_etree = get_etree(xml_file, encoding=encoding)
  File "/usr/local/lib/python3.4/dist-packages/mixbox/xml.py", line 55, in get_etree
    return etree.parse(doc, parser=parser)
  File "src/lxml/lxml.etree.pyx", line 3427, in lxml.etree.parse (src/lxml/lxml.etree.c:81117)
  File "src/lxml/parser.pxi", line 1832, in lxml.etree._parseDocument (src/lxml/lxml.etree.c:118116)
  File "src/lxml/parser.pxi", line 1852, in lxml.etree._parseFilelikeDocument (src/lxml/lxml.etree.c:118399)
  File "src/lxml/parser.pxi", line 1747, in lxml.etree._parseDocFromFilelike (src/lxml/lxml.etree.c:117187)
  File "src/lxml/parser.pxi", line 1162, in lxml.etree._BaseParser._parseDocFromFilelike (src/lxml/lxml.etree.c:111914)
  File "src/lxml/parser.pxi", line 595, in lxml.etree._ParserContext._handleParseResultDoc (src/lxml/lxml.etree.c:105109)
  File "src/lxml/parser.pxi", line 706, in lxml.etree._handleParseResult (src/lxml/lxml.etree.c:106817)
  File "src/lxml/parser.pxi", line 635, in lxml.etree._raiseParseError (src/lxml/lxml.etree.c:105671)
  File "/home/misp/MISP-Taxii-Server/fsisac.xml", line 611
lxml.etree.XMLSyntaxError: AttValue: ' expected, line 611, column 87

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.4/dist-packages/opentaxii-0.1.9a1-py3.4.egg/opentaxii/taxii/services/abstract.py", line 83, in process
    response_message = handler.handle_message(self, message)
  File "/usr/local/lib/python3.4/dist-packages/opentaxii-0.1.9a1-py3.4.egg/opentaxii/taxii/services/handlers/inbox_message_handlers.py", line 126, in handle_message
    return InboxMessage11Handler.handle_message(service, request)
  File "/usr/local/lib/python3.4/dist-packages/opentaxii-0.1.9a1-py3.4.egg/opentaxii/taxii/services/handlers/inbox_message_handlers.py", line 65, in handle_message
    inbox_message_id=inbox_message.id if inbox_message else None)
  File "/usr/local/lib/python3.4/dist-packages/opentaxii-0.1.9a1-py3.4.egg/opentaxii/persistence/manager.py", line 164, in create_content
    collection_ids=collection_ids, service_id=service_id)
  File "/usr/local/lib/python3.4/dist-packages/blinker-1.4-py3.4.egg/blinker/base.py", line 267, in send
    for receiver in self.receivers_for(sender)]
  File "/usr/local/lib/python3.4/dist-packages/blinker-1.4-py3.4.egg/blinker/base.py", line 267, in <listcomp>
    for receiver in self.receivers_for(sender)]
  File "/usr/local/lib/python3.4/dist-packages/misp_taxii_hooks-0.2-py3.4.egg/misp_taxii_hooks/hooks.py", line 59, in post_stix
    package = pymisp.tools.stix.load_stix(content_block.content)
  File "/usr/local/lib/python3.4/dist-packages/pymisp/tools/stix.py", line 17, in load_stix
    stix = convert.load_stix(stix)
  File "/usr/local/lib/python3.4/dist-packages/misp_stix_converter-0.2.9-py3.4.egg/misp_stix_converter/converters/convert.py", line 132, in load_stix
    return manage_stix(f)
  File "/usr/local/lib/python3.4/dist-packages/misp_stix_converter-0.2.9-py3.4.egg/misp_stix_converter/converters/convert.py", line 160, in manage_stix
    raise STIXLoadError("Could not load stix file. {0}".format(ex))
misp_stix_converter.errors.STIXLoadError: Could not load stix file. AttValue: ' expected, line 611, column 87 (fsisac.xml, line 611)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.4/dist-packages/Flask-0.12.1-py3.4.egg/flask/app.py", line 1612, in full_dispatch_request
    rv = self.dispatch_request()
  File "/usr/local/lib/python3.4/dist-packages/Flask-0.12.1-py3.4.egg/flask/app.py", line 1598, in dispatch_request
    return self.view_functions[rule.endpoint](**req.view_args)
  File "/usr/local/lib/python3.4/dist-packages/opentaxii-0.1.9a1-py3.4.egg/opentaxii/middleware.py", line 76, in wrapper
    return _process_with_service(service)
  File "/usr/local/lib/python3.4/dist-packages/opentaxii-0.1.9a1-py3.4.egg/opentaxii/middleware.py", line 154, in _process_with_service
    response_message = service.process(request.headers, taxii_message)
  File "/usr/local/lib/python3.4/dist-packages/opentaxii-0.1.9a1-py3.4.egg/opentaxii/taxii/services/abstract.py", line 89, in process
    in_response_to=message.message_id)
  File "/usr/local/lib/python3.4/dist-packages/opentaxii-0.1.9a1-py3.4.egg/opentaxii/taxii/exceptions.py", line 48, in raise_failure
    tb=tb)
  File "/usr/local/lib/python3.4/dist-packages/six.py", line 685, in reraise
    raise value.with_traceback(tb)
  File "/usr/local/lib/python3.4/dist-packages/opentaxii-0.1.9a1-py3.4.egg/opentaxii/taxii/services/abstract.py", line 83, in process
    response_message = handler.handle_message(self, message)
  File "/usr/local/lib/python3.4/dist-packages/opentaxii-0.1.9a1-py3.4.egg/opentaxii/taxii/services/handlers/inbox_message_handlers.py", line 126, in handle_message
    return InboxMessage11Handler.handle_message(service, request)
  File "/usr/local/lib/python3.4/dist-packages/opentaxii-0.1.9a1-py3.4.egg/opentaxii/taxii/services/handlers/inbox_message_handlers.py", line 65, in handle_message
    inbox_message_id=inbox_message.id if inbox_message else None)
  File "/usr/local/lib/python3.4/dist-packages/opentaxii-0.1.9a1-py3.4.egg/opentaxii/persistence/manager.py", line 164, in create_content
    collection_ids=collection_ids, service_id=service_id)
  File "/usr/local/lib/python3.4/dist-packages/blinker-1.4-py3.4.egg/blinker/base.py", line 267, in send
    for receiver in self.receivers_for(sender)]
  File "/usr/local/lib/python3.4/dist-packages/blinker-1.4-py3.4.egg/blinker/base.py", line 267, in <listcomp>
    for receiver in self.receivers_for(sender)]
  File "/usr/local/lib/python3.4/dist-packages/misp_taxii_hooks-0.2-py3.4.egg/misp_taxii_hooks/hooks.py", line 59, in post_stix
    package = pymisp.tools.stix.load_stix(content_block.content)
  File "/usr/local/lib/python3.4/dist-packages/pymisp/tools/stix.py", line 17, in load_stix
    stix = convert.load_stix(stix)
  File "/usr/local/lib/python3.4/dist-packages/misp_stix_converter-0.2.9-py3.4.egg/misp_stix_converter/converters/convert.py", line 132, in load_stix
    return manage_stix(f)
  File "/usr/local/lib/python3.4/dist-packages/misp_stix_converter-0.2.9-py3.4.egg/misp_stix_converter/converters/convert.py", line 160, in manage_stix
    raise STIXLoadError("Could not load stix file. {0}".format(ex))
opentaxii.taxii.exceptions.FailureStatus: Could not load stix file. AttValue: ' expected, line 611, column 87 (fsisac.xml, line 611), event=Status exception}

I checked the XML and it is correct (unfortunately I can't share it)
This is the configuration file

---
domain: "localhost:9000"
support_basic_auth: yes

save_raw_inbox_messages: yes
xml_parser_supports_huge_tree: no

persistence_api:
  class: opentaxii.persistence.sqldb.SQLDatabaseAPI
  parameters:
    db_connection: mysql://taxii:Password1234@localhost/taxiipersist
    create_tables: yes

auth_api:
  class: opentaxii.auth.sqldb.SQLDatabaseAPI
  parameters:
    db_connection: mysql://taxii:Password1234@localhost/taxiiauth
    create_tables: yes
    secret: XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX

logging:
  opentaxii: debug
  root: debug

hooks: misp_taxii_hooks.hooks
# Sample configuration for misp_taxii_server

zmq:
    host: 192.168.56.50
    port: 50000

misp:
    url: "https://192.168.56.50"
    api: XXXXXXXXXXXXXXXXXXXXXXXXX

I tried to change the xml_parser_supports_huge_tree option in yes, but nothing changed.
The XML block size that causes this error is 65535 bytes.

Can anyone help me?

Thanks in advance

ERROR: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:590)

Hi there!

I created a TAXII Server and used gunicorn as per the guide in http://www.opentaxii.org/en/stable/running.html. However, i'm receiving this error:

command: taxii-poll --path https://<url>:<port>/services/poll --collection collection --username <username> --password <password> --port <port> --verify <path-to-my-ca-chain.cert.pem>

2017-01-20 19:28:49,036 INFO: Polling using data binding: ALL
2017-01-20 19:28:49,045 INFO: Sending Poll_Request to https://<url>:<port>/services/poll
2017-01-20 19:28:50,163 ERROR: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:590)

the poll only works if i used --verify no in my commands. Is there a way to let my self-signed certificate be verified?

Collection not found, but also can't be created

Hello there EclecticIQ,

First off let me say this is a fantastic product and I can't wait to see how it develops. I do, however have a bit of a problem.

I'm using cabby in conjunction with opentaxii as suggested, but I am using MySQL instead of PostGRE. I created my services without a problem with opentaxii-create-services -c and I created my collections the first time without a problem as well with opentaxii-create-collections -c. However, when i use cabby's taxii-push to push items to the collection I have, it doesn't work, throwing the following error:

674 ERROR: NOT_FOUND: ITEM=taxiistand; The Data Collection was not found

Next, when I go to my collections and try to recreate them with opentaxii-create-collections -c collections.yaml, the command throws the following error:

sqlalchemy.exc.IntegrityError: (sqlite3.IntegrityError) UNIQUE constraint failed: data_collections.name [SQL: 'INSERT INTO data_collections (date_created, name, type, description, accept_all_content, bindings, available, volume) VALUES (?, ?, ?, ?, ?, ?, ?, ?)'] [parameters: ('2017-09-28 16:29:49.880883', 'taxiistand', 'DATA_SET', None, 1, '[]', 1, 0)

I'm not quite sure what to do. Thanks for all your help! I'll be attaching my collections.yaml and services.yaml as well

services.txt
collections.txt