freetaxii / server Goto Github PK
View Code? Open in Web Editor NEWA cyber threat intelligence server based on TAXII 2 and written in Golang
License: Apache License 2.0
A cyber threat intelligence server based on TAXII 2 and written in Golang
License: Apache License 2.0
The issue is that server is not compatible with the current release of libstix2-0.6.1 (or the latest source in the repo), but it also isnt' compatible with the previous release libstix2-0.5.1 either.
Here is the output compiling server when using libstix2-0.6.1
$ env GOOS=linux GOARCH=amd64 go build -v freetaxii.go
../../../libstix2/datastore/sqlite3/s_baseobject.go:16:2: cannot find package "github.com/freetaxii/libstix2/objects/baseobject" in any of:
/usr/local/Cellar/go/1.15.2/libexec/src/github.com/freetaxii/libstix2/objects/baseobject (from $GOROOT)
/Users/don/go/src/github.com/freetaxii/libstix2/objects/baseobject (from $GOPATH)
../../internal/config/serverConfig.go:13:2: cannot find package "github.com/freetaxii/libstix2/resources/apiroot" in any of:
/usr/local/Cellar/go/1.15.2/libexec/src/github.com/freetaxii/libstix2/resources/apiroot (from $GOROOT)
/Users/don/go/src/github.com/freetaxii/libstix2/resources/apiroot (from $GOPATH)
../../../libstix2/datastore/datastore.go:8:8: cannot find package "github.com/freetaxii/libstix2/resources/collections" in any of:
/usr/local/Cellar/go/1.15.2/libexec/src/github.com/freetaxii/libstix2/resources/collections (from $GOROOT)
/Users/don/go/src/github.com/freetaxii/libstix2/resources/collections (from $GOPATH)
../../internal/config/serverConfig.go:15:2: cannot find package "github.com/freetaxii/libstix2/resources/discovery" in any of:
/usr/local/Cellar/go/1.15.2/libexec/src/github.com/freetaxii/libstix2/resources/discovery (from $GOROOT)
/Users/don/go/src/github.com/freetaxii/libstix2/resources/discovery (from $GOPATH)
../../../libstix2/datastore/sqlite3/t_collectiondata.go:18:2: cannot find package "github.com/freetaxii/libstix2/resources/envelope" in any of:
/usr/local/Cellar/go/1.15.2/libexec/src/github.com/freetaxii/libstix2/resources/envelope (from $GOROOT)
/Users/don/go/src/github.com/freetaxii/libstix2/resources/envelope (from $GOPATH)
../../../libstix2/datastore/sqlite3/t_collectiondataManifest.go:16:2: cannot find package "github.com/freetaxii/libstix2/resources/manifest" in any of:
/usr/local/Cellar/go/1.15.2/libexec/src/github.com/freetaxii/libstix2/resources/manifest (from $GOROOT)
/Users/don/go/src/github.com/freetaxii/libstix2/resources/manifest (from $GOPATH)
../../internal/handlers/contentHandler.go:18:2: cannot find package "github.com/freetaxii/libstix2/resources/status" in any of:
/usr/local/Cellar/go/1.15.2/libexec/src/github.com/freetaxii/libstix2/resources/status (from $GOROOT)
/Users/don/go/src/github.com/freetaxii/libstix2/resources/status (from $GOPATH)
../../internal/handlers/errors.go:13:2: cannot find package "github.com/freetaxii/libstix2/resources/taxiierror" in any of:
/usr/local/Cellar/go/1.15.2/libexec/src/github.com/freetaxii/libstix2/resources/taxiierror (from $GOROOT)
/Users/don/go/src/github.com/freetaxii/libstix2/resources/taxiierror (from $GOPATH)
../../../libstix2/datastore/sqlite3/t_collectiondata.go:19:2: cannot find package "github.com/freetaxii/libstix2/resources/versions" in any of:
/usr/local/Cellar/go/1.15.2/libexec/src/github.com/freetaxii/libstix2/resources/versions (from $GOROOT)
/Users/don/go/src/github.com/freetaxii/libstix2/resources/versions (from $GOPATH)
Here is the output compiling server when using libstix2-0.5.1
$ env GOOS=linux GOARCH=amd64 go build -v freetaxii.go
internal/unsafeheader
runtime/internal/sys
runtime/internal/atomic
internal/race
internal/cpu
runtime/internal/math
sync/atomic
unicode
unicode/utf8
internal/bytealg
container/list
crypto/internal/subtle
math/bits
crypto/subtle
internal/testlog
unicode/utf16
math
vendor/golang.org/x/crypto/cryptobyte/asn1
internal/nettrace
vendor/golang.org/x/crypto/internal/subtle
encoding
github.com/freetaxii/libstix2/resources/versions
github.com/freetaxii/libstix2/defs
github.com/freetaxii/libstix2/vocabs
runtime
internal/reflectlite
sync
internal/singleflight
math/rand
errors
sort
internal/oserror
io
strconv
vendor/golang.org/x/net/dns/dnsmessage
syscall
bytes
hash
reflect
crypto
strings
crypto/internal/randutil
crypto/hmac
bufio
crypto/rc4
vendor/golang.org/x/crypto/hkdf
regexp/syntax
hash/crc32
time
internal/syscall/unix
internal/syscall/execenv
vendor/golang.org/x/text/transform
path
regexp
context
internal/poll
html
github.com/freetaxii/libstix2/stixid
os
encoding/binary
internal/fmtsort
path/filepath
crypto/cipher
fmt
net
crypto/sha512
crypto/aes
crypto/des
math/big
crypto/ed25519/internal/edwards25519
crypto/md5
crypto/sha1
crypto/sha256
encoding/hex
encoding/base64
io/ioutil
encoding/pem
net/url
vendor/golang.org/x/crypto/chacha20
vendor/golang.org/x/crypto/poly1305
vendor/golang.org/x/sys/cpu
vendor/golang.org/x/crypto/curve25519
vendor/golang.org/x/crypto/chacha20poly1305
encoding/json
database/sql/driver
crypto/elliptic
crypto/rand
crypto/dsa
encoding/asn1
github.com/freetaxii/libstix2/resources/envelope
crypto/ed25519
crypto/rsa
vendor/golang.org/x/crypto/cryptobyte
github.com/google/uuid
crypto/x509/pkix
database/sql
github.com/freetaxii/libstix2/timestamp
github.com/pborman/uuid
github.com/freetaxii/libstix2/objects/properties
github.com/freetaxii/libstix2/resources/properties
github.com/freetaxii/libstix2/objects/baseobject
crypto/ecdsa
github.com/freetaxii/libstix2/resources/manifest
github.com/freetaxii/libstix2/resources/collections
crypto/x509
github.com/freetaxii/libstix2/objects/indicator
github.com/gologme/log
github.com/freetaxii/libstix2/datastore
github.com/freetaxii/libstix2/resources/apiroot
github.com/freetaxii/libstix2/resources/discovery
github.com/mattn/go-sqlite3
log
compress/flate
vendor/golang.org/x/text/unicode/bidi
github.com/freetaxii/libstix2/datastore/sqlite3
compress/gzip
crypto/tls
vendor/golang.org/x/text/unicode/norm
vendor/golang.org/x/text/secure/bidirule
net/textproto
vendor/golang.org/x/net/http2/hpack
mime
mime/quotedprintable
net/http/internal
mime/multipart
vendor/golang.org/x/net/idna
github.com/freetaxii/libstix2/objects
github.com/freetaxii/libstix2/resources/status
github.com/freetaxii/libstix2/resources/taxiierror
text/template/parse
github.com/pborman/getopt
vendor/golang.org/x/net/http/httpguts
vendor/golang.org/x/net/http/httpproxy
text/template
net/http/httptrace
html/template
net/http
github.com/freetaxii/server/internal/headers
github.com/gorilla/mux
github.com/freetaxii/server/internal/config
github.com/freetaxii/server/internal/handlers
../../internal/handlers/contentHandler.go:332:10: assignment mismatch: 2 variables but objects.Decode returns 3 values
root@root:~/go/src/github.com/freetaxii/server/cmd/freetaxii# go build freetaxii.go
../../../libstix2/datastore/sqlite3/s_baseobject.go:16:2: cannot find package "github.com/freetaxii/libstix2/objects/baseobject" in any of:
/usr/lib/go-1.14/src/github.com/freetaxii/libstix2/objects/baseobject (from $GOROOT)
/root/go/src/github.com/freetaxii/libstix2/objects/baseobject (from $GOPATH)
../../internal/config/serverConfig.go:13:2: cannot find package "github.com/freetaxii/libstix2/resources/apiroot" in any of:
/usr/lib/go-1.14/src/github.com/freetaxii/libstix2/resources/apiroot (from $GOROOT)
/root/go/src/github.com/freetaxii/libstix2/resources/apiroot (from $GOPATH)
../../../libstix2/datastore/datastore.go:8:8: cannot find package "github.com/freetaxii/libstix2/resources/collections" in any of:
/usr/lib/go-1.14/src/github.com/freetaxii/libstix2/resources/collections (from $GOROOT)
/root/go/src/github.com/freetaxii/libstix2/resources/collections (from $GOPATH)
../../internal/config/serverConfig.go:15:2: cannot find package "github.com/freetaxii/libstix2/resources/discovery" in any of:
/usr/lib/go-1.14/src/github.com/freetaxii/libstix2/resources/discovery (from $GOROOT)
/root/go/src/github.com/freetaxii/libstix2/resources/discovery (from $GOPATH)
../../../libstix2/datastore/sqlite3/t_collectiondata.go:18:2: cannot find package "github.com/freetaxii/libstix2/resources/envelope" in any of:
/usr/lib/go-1.14/src/github.com/freetaxii/libstix2/resources/envelope (from $GOROOT)
/root/go/src/github.com/freetaxii/libstix2/resources/envelope (from $GOPATH)
../../../libstix2/datastore/sqlite3/t_collectiondataManifest.go:16:2: cannot find package "github.com/freetaxii/libstix2/resources/manifest" in any of:
/usr/lib/go-1.14/src/github.com/freetaxii/libstix2/resources/manifest (from $GOROOT)
/root/go/src/github.com/freetaxii/libstix2/resources/manifest (from $GOPATH)
../../internal/handlers/contentHandler.go:18:2: cannot find package "github.com/freetaxii/libstix2/resources/status" in any of:
/usr/lib/go-1.14/src/github.com/freetaxii/libstix2/resources/status (from $GOROOT)
/root/go/src/github.com/freetaxii/libstix2/resources/status (from $GOPATH)
../../internal/handlers/errors.go:13:2: cannot find package "github.com/freetaxii/libstix2/resources/taxiierror" in any of:
/usr/lib/go-1.14/src/github.com/freetaxii/libstix2/resources/taxiierror (from $GOROOT)
/root/go/src/github.com/freetaxii/libstix2/resources/taxiierror (from $GOPATH)
../../../libstix2/datastore/sqlite3/t_collectiondata.go:19:2: cannot find package "github.com/freetaxii/libstix2/resources/versions" in any of:
/usr/lib/go-1.14/src/github.com/freetaxii/libstix2/resources/versions (from $GOROOT)
/root/go/src/github.com/freetaxii/libstix2/resources/versions (from $GOPATH)
Installing on a fresh Ubuntu Bionic docker image the freetaxii server fails to build.
root@4de5dbc2492a:/# go get -u -v github.com/freetaxii/server/cmd/freetaxii
github.com/freetaxii/server (download)
created GOPATH=/root/go; see 'go help gopath'
github.com/freetaxii/libstix2 (download)
github.com/pborman/uuid (download)
github.com/google/uuid (download)
github.com/gologme/log (download)
github.com/mattn/go-sqlite3 (download)
github.com/gorilla/mux (download)
github.com/pborman/getopt (download)
github.com/freetaxii/libstix2/resources/versions
github.com/freetaxii/libstix2/defs
github.com/freetaxii/libstix2/vocabs
github.com/freetaxii/libstix2/resources/envelope
github.com/google/uuid
github.com/gologme/log
github.com/freetaxii/libstix2/timestamp
github.com/freetaxii/libstix2/stixid
github.com/freetaxii/libstix2/objects/properties
github.com/gorilla/mux
github.com/freetaxii/server/internal/headers
github.com/mattn/go-sqlite3
github.com/pborman/uuid
github.com/pborman/getopt
github.com/freetaxii/libstix2/resources/properties
github.com/freetaxii/libstix2/objects/baseobject
github.com/freetaxii/libstix2/resources/manifest
github.com/freetaxii/libstix2/resources/collections
github.com/freetaxii/libstix2/datastore
github.com/freetaxii/libstix2/resources/apiroot
github.com/freetaxii/libstix2/objects/indicator
github.com/freetaxii/libstix2/resources/discovery
github.com/freetaxii/server/internal/config
github.com/freetaxii/libstix2/objects/attackpattern
github.com/freetaxii/libstix2/objects/campaign
github.com/freetaxii/libstix2/objects/courseofaction
github.com/freetaxii/libstix2/objects/identity
github.com/freetaxii/libstix2/objects/infrastructure
github.com/freetaxii/libstix2/objects/intrusionset
github.com/freetaxii/libstix2/objects/malware
github.com/freetaxii/libstix2/objects/observeddata
github.com/freetaxii/libstix2/objects/relationship
github.com/freetaxii/libstix2/objects/report
github.com/freetaxii/libstix2/objects/sighting
github.com/freetaxii/libstix2/objects/threatactor
github.com/freetaxii/libstix2/objects/tool
github.com/freetaxii/libstix2/objects/vulnerability
github.com/freetaxii/libstix2/resources/status
github.com/freetaxii/libstix2/resources/taxiierror
github.com/freetaxii/libstix2/objects
github.com/freetaxii/server/internal/handlers
# github.com/freetaxii/server/internal/handlers
root/go/src/github.com/freetaxii/server/internal/handlers/contentHandler.go:332:14: assignment mismatch: 3 variables but 2 values
github.com/freetaxii/libstix2/datastore/sqlite3
root@4de5dbc2492a:/# cd ~/go/src/github.com/freetaxii/server/cmd/freetaxii
root@4de5dbc2492a:~/go/src/github.com/freetaxii/server/cmd/freetaxii# go build freetaxii.go
# github.com/freetaxii/server/internal/handlers
../../internal/handlers/contentHandler.go:332:14: assignment mismatch: 3 variables but 2 values
The offending line in contentHandler.go seems to be:
o, id, err := objects.Decode(v)
It looks like the libstix2 library was updated in Jan to use a different format, one which returns a different number of arguments
The following command which download the previous release of libstix2 fixes the problem.
cd ~/go/src/github.com/freetaxii
rm -rf libstix2
wget https://github.com/freetaxii/libstix2/archive/v0.5.1.tar.gz
tar -zxvf v0.5.1.tar.gz
ln -s libstix2-0.5.1 libstix2
root@4de5dbc2492a:~/go/src/github.com/freetaxii# cd ~/go/src/github.com/freetaxii/server/cmd/freetaxii
root@4de5dbc2492a:~/go/src/github.com/freetaxii/server/cmd/freetaxii# go build freetaxii.go
root@4de5dbc2492a:~/go/src/github.com/freetaxii/server/cmd/freetaxii#
Hi,
Thanks for working on an implementation of taxii2 server. I'm trying to install and get freetaxii to run but I've been getting errors one after another. I've sort of managed to work through most of it but now stuck at another error.
This is my folder structure
/Users/user/workspace/go/src/github.com/freetaxii/server/cmd/freetaxii
And this is the error I'm getting now if I execute the freetaxii binary.
2018/12/16 22:53:10 database execution error getting base index: %!(EXTRA sqlite3.Error=no such table: s_base_object)
Any advise on how to solve this?
Hi Bret,
thanks for sharing this work on a TAXII 2.0 server and especially the test server.
I'm starting a TAXII 2.0 javascript client library at: https://github.com/workingDog/taxii2lib
I'm having some problems when trying to connect to your test server. I get:
Failed to load https://test.freetaxii.com:8000/taxii/: Response to preflight request
doesn't pass access control check: No 'Access-Control-Allow-Origin'
header is present on the requested resource. Origin 'http://localhost:8000'
is therefore not allowed access. The response had HTTP status code 405.
If an opaque response serves your needs, set the request's mode to 'no-cors'
to fetch the resource with CORS disabled.
I've tried many different headers and modes for my requests and also
browser (Chrome) settings on my part but to no avail. I just cannot get pass this.
Do you have any advice on how to overcome this?
Ringo
Hi all,
Probably my poor knowledge with go, but I can't succeed to build freetaxii.
Here is the error I have (extract only one)
package github.com/freetaxii/libstix2/resources/collections: cannot find package "github.com/freetaxii/libstix2/resources/collections" in any of:
/usr/local/go/src/github.com/freetaxii/libstix2/resources/collections (from $GOROOT)
/root/myGOProjects/src/github.com/freetaxii/libstix2/resources/collections (from $GOPATH)
If I go to the folder /root/myGOProjects/src/github.com/freetaxii/libstix2/resources then I have only these files/folder :
doc.go helpers resources.go
Thanks in advance for your kind help
--Kripskroll
It seems that new STIX objects are created every time the objects endpoint is queried. Every time I request it, the id/created/modified properties of all the objects are different. This makes it very hard to get an object by its id.
When testing the /<api-root>/collections/<id>/objects/
endpoint, it seems that the server returns JSON if the Accept header is set to application/vnd.oasis.taxii+json
but returns HTML if it is set to application/vnd.oasis.stix+json
. Looking at Section 5.3 in the spec it looks like it should accept the STIX media type.
I'm testing retrieving the collection with this:
`
<title>TAXII2</title> <script> function loadTaxii() { var xhttp = new XMLHttpRequest(); xhttp.onreadystatechange = function() { if (this.readyState == 4 && this.status == 200) { document.getElementById("demo").innerHTML = this.responseText; } }; xhttp.open("GET", "https://test.freetaxii.com:8000/api1/collections/9cfa669c-ee94-4ece-afd2-f8edac37d8fd/", true); xhttp.setRequestHeader('Accept', 'application/vnd.oasis.taxii+json'); xhttp.setRequestHeader('Content-type', 'application/vnd.oasis.taxii+json'); xhttp.send(); }; loadTaxii(); </script> `I get:
{"id":"9cfa669c-ee94-4ece-afd2-f8edac37d8fd","title":"Emerging Threats - Block Rules - Compromised IPs","description":"A collection of blocking rule indicators from Emerging Threats. Pulled live from: https://rules.emergingthreats.net/blockrules/emerging-botcc.excluded","can_read":true,"media_types":["application/vnd.oasis.stix+json"]}
which does not contain the required can_write attribute.
I'm getting errors trying to use python-stix2
to request data from freetaxii:
collection = Collection("https://test.freetaxii.com:8000/osint/collections/2b37444f-3659-4dba-ad8e-960b4e61a546/")
# supply the TAXII2 collection to TAXIICollection
tc_source = TAXIICollectionSource(collection)
# build your filter -- kinda seems broken now
# f1 = Filter("type","=", "indicator")
#retrieve the STIX objects
results = tc_source.query(allow_custom=True)
Error is:
TAXIIServiceException: Unexpected Response Content-Type: application/vnd.oasis.stix+json; version=2.1; charset=utf-8
Is there going to be a future when a more stable backend DB will be allowed?
I used to be able to test my code using your very useful test taxii 2.0 server at:
https://test.freetaxii.com:8000/
do you have a test taxii 2.1 server?
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.