benb196 / crashplan-ffs-puller Goto Github PK
View Code? Open in Web Editor NEWA third party Golang application used for pulling Code 42 Crashplan File Forensic Search logs from their API.
License: MIT License
A third party Golang application used for pulling Code 42 Crashplan File Forensic Search logs from their API.
License: MIT License
There are a couple incorrect elasticsearch fields in the es standardized mapping
The following error should not result in a panic, instead it should result in the query being retried. A max of 10 retries should be allowed until a fatal error/panic is thrown. This limit is to prevent the program from building up a massive queue of in progress/failed queries which will eventually become impossible to handle.
error getting file events for ffs query: main_query
panic: Error with gathering file events POST: 500 Internal Server Error
goroutine 191656 [running]:
github.com/BenB196/crashplan-ffs-puller/ffsEvent.queryFetcher(0xc00012ed90, 0xa, 0xc00010c660, 0x17, 0xc00012eda0, 0xf, 0xc00012ed9a, 0x2, 0xc00012ed9c, 0x3, ...)
/app/ffsEvent/ffsEvent.go:234 +0x2802
created by github.com/BenB196/crashplan-ffs-puller/ffsEvent.FFSQuery.func5
/app/ffsEvent/ffsEvent.go:184 +0x1d5
Currently, Elasticsearch index insertion is based off of onOrBefore time or time.Now. It would be nice to also support insertion based off of EventTimestamp or InsertionTimestamp as these can differ depending on the query that is being run.
crashplan-ffs-puller/ffsEvent/ffsEvent.go
Line 185 in 5b39c83
Causes a WaitGroup is reused before previous Wait has returned
best_compression should be enabled on the Elasticsearch index template as it provides better space optimization. This should actually be a configuration value that can be enabled/disabled
Currently there are no logging levels. This is something that should be added.
The _all field is not needed as this type of search should not be done against this data as all fields a specific and know. This field just causes extra overhead which is not needed.
https://www.elastic.co/guide/en/elasticsearch/reference/5.5/mapping-all-field.html
The following is not handled but should be:
error getting file events for ffs query: main_query
panic: Error with gathering file events POST: 400 Bad Request
goroutine 144713 [running]:
github.com/BenB196/crashplan-ffs-puller/ffsEvent.queryFetcher(0xc000025390, 0xa, 0xc000022a00, 0x17, 0xc0000253a0, 0xf, 0xc00002539a, 0x2, 0xc00002539c, 0x3, ...)
/app/ffsEvent/ffsEvent.go:249 +0x2c05
github.com/BenB196/crashplan-ffs-puller/ffsEvent.queryFetcher(0xc000025390, 0xa, 0xc000022a00, 0x17, 0xc0000253a0, 0xf, 0xc00002539a, 0x2, 0xc00002539c, 0x3, ...)
/app/ffsEvent/ffsEvent.go:241 +0x281d
created by github.com/BenB196/crashplan-ffs-puller/ffsEvent.FFSQuery.func5
/app/ffsEvent/ffsEvent.go:185 +0x1e9
There is some poor logic with the use of WaitGroups in this application. This poor logic causes the following error:
sync: WaitGroup is reused before previous Wait has returned
Waitgroup logic needs to be re-evaluated.
The following error is not properly handled, and should result in retry attempts.
panic: stream error: stream ID x; PROTOCOL_ERROR
goroutine 225026 [running]:
github.com/BenB196/crashplan-ffs-puller/ffsEvent.queryFetcher(0xc000027390, 0xa, 0xc000022a00, 0x17, 0xc0000273a0, 0xf, 0xc00002739a, 0x2, 0xc00002739c, 0x3, ...)
/app/ffsEvent/ffsEvent.go:251 +0x2cdf
github.com/BenB196/crashplan-ffs-puller/ffsEvent.queryFetcher(0xc000027390, 0xa, 0xc000022a00, 0x17, 0xc0000273a0, 0xf, 0xc00002739a, 0x2, 0xc00002739c, 0x3, ...)
/app/ffsEvent/ffsEvent.go:243 +0x282f
created by github.com/BenB196/crashplan-ffs-puller/ffsEvent.FFSQuery.func5
/app/ffsEvent/ffsEvent.go:185 +0x1e9
Currently query retries, retry immediately. This can causes a too many request error if there is actually something wrong with the API end point. Therefore query retries should be slept for queryInterval before firing again.
The following error can be thrown, but should be handled:
error getting file events for ffs query: main_query
panic: stream error: stream ID 2223; INTERNAL_ERROR
goroutine 32318 [running]:
github.com/BenB196/crashplan-ffs-puller/ffsEvent.queryFetcher(0xc000116c90, 0xa, 0xc0000fc580, 0x17, 0xc000116ca0, 0xf, 0xc000116c9a, 0x2, 0xc000116c9c, 0x3, ...)
/app/ffsEvent/ffsEvent.go:233 +0x1d6a
created by github.com/BenB196/crashplan-ffs-puller/ffsEvent.FFSQuery.func5
/app/ffsEvent/ffsEvent.go:180 +0x206
Passing "geoPoint":{} to elasticsearch causes it to error and not record the doc.
The following error is not properly handled.
error getting file events for ffs query: main_query
panic: Post https://forensicsearch-default.prod.ffs.us2.code42.com/forensic-search/queryservice/api/v1/fileevent/export: unexpected EOF
goroutine 605928 [running]:
github.com/BenB196/crashplan-ffs-puller/ffsEvent.queryFetcher(0xc000128d90, 0xa, 0xc0000f6660, 0x17, 0xc000128da0, 0xf, 0xc000128d9a, 0x2, 0xc000128d9c, 0x3, ...)
/app/ffsEvent/ffsEvent.go:251 +0x2d3b
created by github.com/BenB196/crashplan-ffs-puller/ffsEvent.FFSQuery.func5
/app/ffsEvent/ffsEvent.go:185 +0x1e9
Currently retrying queries increments the in progress query count of the prometheus metric. This should not be done as the actual number of in progress queries has not changed.
Currently POST: 429 Too Many Requests are unhandled and result in a panic. Potentially handle this thing.
OneDrive Events aren't getting pulled.
Currently if 0001-01-01T00:00:00Z is passed as a date, it is considered a real value.
These values should be dropped as null.
See error below:
panic: Error with gathering file events POST: 504 Gateway Timeout
goroutine 67027614 [running]:
github.com/BenB196/crashplan-ffs-puller/ffsEvent.queryFetcher(0xc000024df0, 0xa, 0xc000022420, 0x17, 0xc000024e00, 0xf, 0xc000024dfa, 0x2, 0xc000024dfc, 0x3, ...)
/app/ffsEvent/ffsEvent.go:251 +0x3878
created by github.com/BenB196/crashplan-ffs-puller/ffsEvent.FFSQuery.func5
/app/ffsEvent/ffsEvent.go:185 +0x205
The following error can be thrown and should be handled.
panic: Post https://forensicsearch-default.prod.ffs.us2.code42.com/forensic-search/queryservice/api/v1/fileevent/export: read tcp 10.111.1.8:51048->3.87.164.248:443: read: connection reset by peer
goroutine 422582 [running]:
github.com/BenB196/crashplan-ffs-puller/ffsEvent.queryFetcher(0xc000024da0, 0xa, 0xc000022680, 0x17, 0xc000024db0, 0xf, 0xc000024daa, 0x2, 0xc000024dac, 0x3, ...)
/app/ffsEvent/ffsEvent.go:249 +0x2c05
created by github.com/BenB196/crashplan-ffs-puller/ffsEvent.FFSQuery.func5
/app/ffsEvent/ffsEvent.go:185 +0x1e9
panic: read tcp 10.111.1.8:51048->3.87.164.248:443: read: connection reset by peer
goroutine 422473 [running]:
github.com/BenB196/crashplan-ffs-puller/ffsEvent.queryFetcher(0xc000024da0, 0xa, 0xc000022680, 0x17, 0xc000024db0, 0xf, 0xc000024daa, 0x2, 0xc000024dac, 0x3, ...)
/app/ffsEvent/ffsEvent.go:249 +0x2c05
created by github.com/BenB196/crashplan-ffs-puller/ffsEvent.FFSQuery.func5
/app/ffsEvent/ffsEvent.go:185 +0x1e9
This is not something that is actually used in 7.0 elasticsearch
The elastic standardized output puts Exposure under cloud. This should be directly under file_event instead.
Currently if you need to point to an elasticsearch cluster with non-public certs, you need to be able to pass a CA to docker. While this is possible, there is no way to properly update ca's without executing the command after start. This is intended to fix that.
Crashplan FFS privateIpAddress field returns invalid IPv6 addresses:
Are invalid.
Are the proper valid IP Addresses.
Under certain circumstances, the Prometheus metric for In Progress Queries can become a negative value.
Some people may want to do additional data enrichment with FFS logs. Therefore having the ability to output logs to logstash is a nice add.
Currently both the FileEvent field and Geolocation field are flattened into the base JSON object. Some people may want the ability to have these no flattened and in their own sub-objects. The ability to enable this would be nice to have.
When using IP-API integration, there is the possibility that the publicIPAddress field can be "". If this occurs the app will panic.
This adds support for 2 new fields:
Currently field names are locked into the ones given by Crashplan FFS. However a lot of these fields don't make usable sense when trying to integrate with other indexes (ex: osHostname should really be either hostname or host, and deviceUsername should be username).
Should allow for people to create custom mappings so that they can integrate the FFS into their own indexes.
Currently only camalCase is supported. Snake_case option would be nice.
The output location for logstash output does not validate properly.
Following error can be thrown
panic: reflect: slice index out of range [recovered]
panic: reflect: slice index out of range
goroutine 82 [running]:
encoding/json.(*encodeState).marshal.func1(0xc0049ef9e8)
/usr/local/go/src/encoding/json/encode.go:305 +0x9a
panic(0x9c2ca0, 0xb7ee60)
/usr/local/go/src/runtime/panic.go:679 +0x1b2
reflect.Value.Index(0x9b4b60, 0xc00000c080, 0x197, 0x27, 0xa20100, 0xc00025c720, 0x199)
/usr/local/go/src/reflect/value.go:949 +0x1fe
encoding/json.arrayEncoder.encode(0xc000b9be60, 0xc0000ea070, 0x9b4b60, 0xc00000c080, 0x197, 0xc0049e0100)
/usr/local/go/src/encoding/json/encode.go:791 +0x8d
encoding/json.sliceEncoder.encode(0xc0002e7710, 0xc0000ea070, 0x9b4b60, 0xc00000c080, 0x197, 0x4f0100)
/usr/local/go/src/encoding/json/encode.go:765 +0x8f
encoding/json.ptrEncoder.encode(0xc0002e77a0, 0xc0000ea070, 0x99cd00, 0xc00000c080, 0x16, 0x990100)
/usr/local/go/src/encoding/json/encode.go:810 +0xb1
encoding/json.(*encodeState).reflectValue(0xc0000ea070, 0x99cd00, 0xc00000c080, 0x16, 0x100)
/usr/local/go/src/encoding/json/encode.go:337 +0x82
encoding/json.(*encodeState).marshal(0xc0000ea070, 0x99cd00, 0xc00000c080, 0x100, 0x0, 0x0)
/usr/local/go/src/encoding/json/encode.go:309 +0x10b
encoding/json.Marshal(0x99cd00, 0xc00000c080, 0x1000, 0xc0013d0000, 0xc000532030, 0x0, 0x0)
/usr/local/go/src/encoding/json/encode.go:161 +0x52
github.com/BenB196/crashplan-ffs-puller/eventOutput.WriteInProgressQueries(0xc00011cef0, 0xe, 0xc00011e5e0, 0x17, 0xc00011cf00, 0xf, 0xc00011cefe, 0x2, 0xc00011cf10, 0x3, ...)
/app/eventOutput/fileHandler.go:279 +0x2fe
github.com/BenB196/crashplan-ffs-puller/ffsEvent.FFSQuery.func2(0xc00000c080, 0xc0004282d0, 0xc00000a1e0, 0xc000212780)
/app/ffsEvent/ffsEvent.go:99 +0x23c
created by github.com/BenB196/crashplan-ffs-puller/ffsEvent.FFSQuery
/app/ffsEvent/ffsEvent.go:91 +0x67d
The retrycount is never properly initialized. This results in an infinite for loop of retries.
Elasticsearch breaks when you remote nodes. Setting elastic.SetSniff() to false resolves this issue.
crashplan-ffs-go-pkg v0.0.7 adds support for the new FFS event fields
Currently InProgressQueries and LastCompletedQuery are checked/written on an 100ms. It may be better/more efficient to instead handle writes when the variables are updated instead.
Saw retry event, but retry number never went above 0.
Currently logstash connection uses DialTCP, instead should use net.Dialer and sent a proper timeout.
Currently the output of information does not match the standards of elasticsearch (https://www.elastic.co/guide/en/beats/devguide/current/event-conventions.html). A config option to support this style output should be added.
type ElasticFFSEvent struct {
FileEvent ElasticFileEvent `json:"file_event"`
Geoip *Geoip `json:"geoip"`
}
type ElasticFileEvent struct {
EventId string `json:"eventId"`
EventType string `json:"eventType"`
EventTimestamp *time.Time `json:"event_timestamp,omitempty"`
InsertionTimestamp *time.Time `json:"insertion_timestamp,omitempty"`
FilePath string `json:"file_path,omitempty"`
FileName string `json:"file_name"`
FileType string `json:"file_type,omitempty"`
FileCategory string `json:"file_category,omitempty"`
FileSize *int `json:"file_size"`
FileOwner []string `json:"file_owner,omitempty"` //Array of owners
Md5Checksum string `json:"md5_checksum,omitempty"`
Sha256Checksum string `json:"sha256_checksum,omitempty"`
CreatedTimestamp *time.Time `json:"created_timestamp,omitempty"`
ModifyTimestamp *time.Time `json:"modify_timestamp,omitempty"`
DeviceUsername string `json:"device_username,omitempty"`
DeviceUid string `json:"device_uid,omitempty"`
UserUid string `json:"user_uid,omitempty"`
OsHostname string `json:"os_hostname,omitempty"`
DomainName string `json:"domain_name,omitempty"`
PublicIpAddress string `json:"public_ip_address,omitempty"`
PrivateIpAddresses []string `json:"private_ip_addresses,omitempty"` //Array of IP address strings
Actor string `json:"actor,omitempty"`
DirectoryId []string `json:"directory_id,omitempty"` //An array of something, I am not sure
Source string `json:"source,omitempty"`
Url string `json:"url,omitempty"`
Shared string `json:"shared,omitempty"`
SharedWith []string `json:"shared-with,omitempty"` //An array of strings (Mainly Email Addresses)
SharingTypeAdded []string `json:"sharing_type_added,omitempty"`
CloudDriveId string `json:"cloud_drive_id,omitempty"`
DetectionSourceAlias string `json:"detection_source_alias,omitempty"`
FileId string `json:"file_id,omitempty"`
Exposure []string `json:"exposure,omitempty"`
ProcessOwner string `json:"process_owner,omitempty"`
ProcessName string `json:"process_name,omitempty"`
RemovableMediaVendor string `json:"removable_media_vendor,omitempty"`
RemovableMediaName string `json:"removable_media_name,omitempty"`
RemovableMediaSerialNumber string `json:"removable_media_serial_number,omitempty"`
RemovableMediaCapacity *int `json:"removable_media_capacity,omitempty"`
RemovableMediaBusType string `json:"removable_media_bus_type,omitempty"`
RemovableMediaMediaName string `json:"removable_media-media_name,omitempty"`
RemovableMediaVolumeName string `json:"removable_media_volume_name,omitempty"`
RemovableMediaPartitionId string `json:"removable_media_partition_id,omitempty"`
SyncDestination string `json:"sync_destination,omitempty"`
}
type Geoip struct {
Status string `json:"status,omitempty"`
Message string `json:"message,omitempty"`
Continent string `json:"continent,omitempty"`
ContinentCode string `json:"continent_code,omitempty"`
Country string `json:"country_name,omitempty"`
CountryCode string `json:"country_code2,omitempty"`
Region string `json:"region,omitempty"`
RegionName string `json:"region_name,omitempty"`
City string `json:"city_name,omitempty"`
District string `json:"district,omitempty"`
ZIP string `json:"postal_code,omitempty"`
Lat float32 `json:"latitude,omitempty"`
Lon float32 `json:"longitude,omitempty"`
Timezone string `json:"timezone,omitempty"`
Currency string `json:"currency,omitempty"`
ISP string `json:"isp,omitempty"`
Org string `json:"org,omitempty"`
AS string `json:"as,omitempty"`
ASName string `json:"as_name,omitempty"`
Reverse string `json:"reverse,omitempty"`
Mobile bool `json:"mobile,omitempty"`
Proxy bool `json:"proxy,omitempty"`
Query string `json:"ip,omitempty"`
GeoPoint *GeoPoint `json:"location,omitempty"`
}
type GeoPoint struct {
Lat float32 `json:"lat,omitempty"`
Lon float32 `json:"lon,omitempty"`
}
Currently, refresh_interval is unset, which may not be desirable for everyone. A configuration option for setting this value should be added.
https://sematext.com/blog/elasticsearch-refresh-interval-vs-indexing-performance/
Logstash timeout is currently set to 30 seconds. Can safely set this to 5 minutes probably.
This makes the output, more closely related to the FFS out, while making some small changes to be more in line with Elasticsearch standards
If the inProgressQueries files exists, the app will fail to start with the following error:
error getting old in progress queries
panic: error: parsing in progress queries from: /crashplan-ffs-puller/<query_name>inProgressQueries.json unexpected end of JSON input
goroutine 18 [running]:
github.com/BenB196/crashplan-ffs-puller/ffsEvent.FFSQuery(0xc000130600, 0x39, 0xc00005a070, 0x6b, 0xc000122e00, 0x1, 0x4, 0x1, 0x1f90, 0xc000026cf0, ...)
/app/ffsEvent/ffsEvent.go:44 +0x10f5
created by main.main.func1
/app/main.go:46 +0x15f
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.