GithubHelp home page GithubHelp logo

benb196 / crashplan-ffs-puller Goto Github PK

View Code? Open in Web Editor NEW
2.0 2.0 0.0 1.4 MB

A third party Golang application used for pulling Code 42 Crashplan File Forensic Search logs from their API.

License: MIT License

Go 99.38% Dockerfile 0.62%

crashplan-ffs-puller's People

Contributors

benb196 avatar

Stargazers

avatar avatar

Watchers

avatar avatar

crashplan-ffs-puller's Issues

Bug: 500 Internal Server Error should be properly handled

The following error should not result in a panic, instead it should result in the query being retried. A max of 10 retries should be allowed until a fatal error/panic is thrown. This limit is to prevent the program from building up a massive queue of in progress/failed queries which will eventually become impossible to handle.

error getting file events for ffs query: main_query
panic: Error with gathering file events POST: 500 Internal Server Error

goroutine 191656 [running]:
github.com/BenB196/crashplan-ffs-puller/ffsEvent.queryFetcher(0xc00012ed90, 0xa, 0xc00010c660, 0x17, 0xc00012eda0, 0xf, 0xc00012ed9a, 0x2, 0xc00012ed9c, 0x3, ...)
        /app/ffsEvent/ffsEvent.go:234 +0x2802
created by github.com/BenB196/crashplan-ffs-puller/ffsEvent.FFSQuery.func5
        /app/ffsEvent/ffsEvent.go:184 +0x1d5

Bug: Handle 400 Bad Request

The following is not handled but should be:

error getting file events for ffs query: main_query
panic: Error with gathering file events POST: 400 Bad Request

goroutine 144713 [running]:
github.com/BenB196/crashplan-ffs-puller/ffsEvent.queryFetcher(0xc000025390, 0xa, 0xc000022a00, 0x17, 0xc0000253a0, 0xf, 0xc00002539a, 0x2, 0xc00002539c, 0x3, ...)
        /app/ffsEvent/ffsEvent.go:249 +0x2c05
github.com/BenB196/crashplan-ffs-puller/ffsEvent.queryFetcher(0xc000025390, 0xa, 0xc000022a00, 0x17, 0xc0000253a0, 0xf, 0xc00002539a, 0x2, 0xc00002539c, 0x3, ...)
        /app/ffsEvent/ffsEvent.go:241 +0x281d
created by github.com/BenB196/crashplan-ffs-puller/ffsEvent.FFSQuery.func5
        /app/ffsEvent/ffsEvent.go:185 +0x1e9

Bug: Poor logic with WaitGroups

There is some poor logic with the use of WaitGroups in this application. This poor logic causes the following error:

sync: WaitGroup is reused before previous Wait has returned

Waitgroup logic needs to be re-evaluated.

Bug: Handle stream error: stream ID x; PROTOCOL_ERROR

The following error is not properly handled, and should result in retry attempts.

panic: stream error: stream ID x; PROTOCOL_ERROR

goroutine 225026 [running]:
github.com/BenB196/crashplan-ffs-puller/ffsEvent.queryFetcher(0xc000027390, 0xa, 0xc000022a00, 0x17, 0xc0000273a0, 0xf, 0xc00002739a, 0x2, 0xc00002739c, 0x3, ...)
        /app/ffsEvent/ffsEvent.go:251 +0x2cdf
github.com/BenB196/crashplan-ffs-puller/ffsEvent.queryFetcher(0xc000027390, 0xa, 0xc000022a00, 0x17, 0xc0000273a0, 0xf, 0xc00002739a, 0x2, 0xc00002739c, 0x3, ...)
        /app/ffsEvent/ffsEvent.go:243 +0x282f
created by github.com/BenB196/crashplan-ffs-puller/ffsEvent.FFSQuery.func5
        /app/ffsEvent/ffsEvent.go:185 +0x1e9

Bug: Add sleep to query retries

Currently query retries, retry immediately. This can causes a too many request error if there is actually something wrong with the API end point. Therefore query retries should be slept for queryInterval before firing again.

Bug: stream error: stream ID x; INTERNAL_ERROR is not handled

The following error can be thrown, but should be handled:

error getting file events for ffs query: main_query
panic: stream error: stream ID 2223; INTERNAL_ERROR

goroutine 32318 [running]:
github.com/BenB196/crashplan-ffs-puller/ffsEvent.queryFetcher(0xc000116c90, 0xa, 0xc0000fc580, 0x17, 0xc000116ca0, 0xf, 0xc000116c9a, 0x2, 0xc000116c9c, 0x3, ...)
        /app/ffsEvent/ffsEvent.go:233 +0x1d6a
created by github.com/BenB196/crashplan-ffs-puller/ffsEvent.FFSQuery.func5
        /app/ffsEvent/ffsEvent.go:180 +0x206

Bug: Handle unexpected EOF

The following error is not properly handled.

error getting file events for ffs query: main_query
panic: Post https://forensicsearch-default.prod.ffs.us2.code42.com/forensic-search/queryservice/api/v1/fileevent/export: unexpected EOF

goroutine 605928 [running]:
github.com/BenB196/crashplan-ffs-puller/ffsEvent.queryFetcher(0xc000128d90, 0xa, 0xc0000f6660, 0x17, 0xc000128da0, 0xf, 0xc000128d9a, 0x2, 0xc000128d9c, 0x3, ...)
        /app/ffsEvent/ffsEvent.go:251 +0x2d3b
created by github.com/BenB196/crashplan-ffs-puller/ffsEvent.FFSQuery.func5
        /app/ffsEvent/ffsEvent.go:185 +0x1e9

Bug: Handle POST: 504 Gateway Timeout

See error below:

panic: Error with gathering file events POST: 504 Gateway Timeout

goroutine 67027614 [running]:
github.com/BenB196/crashplan-ffs-puller/ffsEvent.queryFetcher(0xc000024df0, 0xa, 0xc000022420, 0x17, 0xc000024e00, 0xf, 0xc000024dfa, 0x2, 0xc000024dfc, 0x3, ...)
        /app/ffsEvent/ffsEvent.go:251 +0x3878
created by github.com/BenB196/crashplan-ffs-puller/ffsEvent.FFSQuery.func5
        /app/ffsEvent/ffsEvent.go:185 +0x205

Bug: Handle connection reset by peer

The following error can be thrown and should be handled.

panic: Post https://forensicsearch-default.prod.ffs.us2.code42.com/forensic-search/queryservice/api/v1/fileevent/export: read tcp 10.111.1.8:51048->3.87.164.248:443: read: connection reset by peer

goroutine 422582 [running]:
github.com/BenB196/crashplan-ffs-puller/ffsEvent.queryFetcher(0xc000024da0, 0xa, 0xc000022680, 0x17, 0xc000024db0, 0xf, 0xc000024daa, 0x2, 0xc000024dac, 0x3, ...)
        /app/ffsEvent/ffsEvent.go:249 +0x2c05
created by github.com/BenB196/crashplan-ffs-puller/ffsEvent.FFSQuery.func5
        /app/ffsEvent/ffsEvent.go:185 +0x1e9
panic: read tcp 10.111.1.8:51048->3.87.164.248:443: read: connection reset by peer

goroutine 422473 [running]:
github.com/BenB196/crashplan-ffs-puller/ffsEvent.queryFetcher(0xc000024da0, 0xa, 0xc000022680, 0x17, 0xc000024db0, 0xf, 0xc000024daa, 0x2, 0xc000024dac, 0x3, ...)
        /app/ffsEvent/ffsEvent.go:249 +0x2c05
created by github.com/BenB196/crashplan-ffs-puller/ffsEvent.FFSQuery.func5
        /app/ffsEvent/ffsEvent.go:185 +0x1e9

Feature: Support reloading of trusted CA in docker container

Currently if you need to point to an elasticsearch cluster with non-public certs, you need to be able to pass a CA to docker. While this is possible, there is no way to properly update ca's without executing the command after start. This is intended to fix that.

Enhancement: Add the ability to have nested fields

Currently both the FileEvent field and Geolocation field are flattened into the base JSON object. Some people may want the ability to have these no flattened and in their own sub-objects. The ability to enable this would be nice to have.

Bug: reflect: slice index out of range

Following error can be thrown

panic: reflect: slice index out of range [recovered]
        panic: reflect: slice index out of range

goroutine 82 [running]:
encoding/json.(*encodeState).marshal.func1(0xc0049ef9e8)
        /usr/local/go/src/encoding/json/encode.go:305 +0x9a
panic(0x9c2ca0, 0xb7ee60)
        /usr/local/go/src/runtime/panic.go:679 +0x1b2
reflect.Value.Index(0x9b4b60, 0xc00000c080, 0x197, 0x27, 0xa20100, 0xc00025c720, 0x199)
        /usr/local/go/src/reflect/value.go:949 +0x1fe
encoding/json.arrayEncoder.encode(0xc000b9be60, 0xc0000ea070, 0x9b4b60, 0xc00000c080, 0x197, 0xc0049e0100)
        /usr/local/go/src/encoding/json/encode.go:791 +0x8d
encoding/json.sliceEncoder.encode(0xc0002e7710, 0xc0000ea070, 0x9b4b60, 0xc00000c080, 0x197, 0x4f0100)
        /usr/local/go/src/encoding/json/encode.go:765 +0x8f
encoding/json.ptrEncoder.encode(0xc0002e77a0, 0xc0000ea070, 0x99cd00, 0xc00000c080, 0x16, 0x990100)
        /usr/local/go/src/encoding/json/encode.go:810 +0xb1
encoding/json.(*encodeState).reflectValue(0xc0000ea070, 0x99cd00, 0xc00000c080, 0x16, 0x100)
        /usr/local/go/src/encoding/json/encode.go:337 +0x82
encoding/json.(*encodeState).marshal(0xc0000ea070, 0x99cd00, 0xc00000c080, 0x100, 0x0, 0x0)
        /usr/local/go/src/encoding/json/encode.go:309 +0x10b
encoding/json.Marshal(0x99cd00, 0xc00000c080, 0x1000, 0xc0013d0000, 0xc000532030, 0x0, 0x0)
        /usr/local/go/src/encoding/json/encode.go:161 +0x52
github.com/BenB196/crashplan-ffs-puller/eventOutput.WriteInProgressQueries(0xc00011cef0, 0xe, 0xc00011e5e0, 0x17, 0xc00011cf00, 0xf, 0xc00011cefe, 0x2, 0xc00011cf10, 0x3, ...)
        /app/eventOutput/fileHandler.go:279 +0x2fe
github.com/BenB196/crashplan-ffs-puller/ffsEvent.FFSQuery.func2(0xc00000c080, 0xc0004282d0, 0xc00000a1e0, 0xc000212780)
        /app/ffsEvent/ffsEvent.go:99 +0x23c
created by github.com/BenB196/crashplan-ffs-puller/ffsEvent.FFSQuery
        /app/ffsEvent/ffsEvent.go:91 +0x67d

Enhancement: Add an Elastic standard output.

Currently the output of information does not match the standards of elasticsearch (https://www.elastic.co/guide/en/beats/devguide/current/event-conventions.html). A config option to support this style output should be added.

type ElasticFFSEvent struct {
	FileEvent	ElasticFileEvent	`json:"file_event"`
	Geoip		*Geoip				`json:"geoip"`
}

type ElasticFileEvent struct {
	EventId						string			`json:"eventId"`
	EventType					string			`json:"eventType"`
	EventTimestamp				*time.Time		`json:"event_timestamp,omitempty"`
	InsertionTimestamp			*time.Time		`json:"insertion_timestamp,omitempty"`
	FilePath					string			`json:"file_path,omitempty"`
	FileName					string			`json:"file_name"`
	FileType					string			`json:"file_type,omitempty"`
	FileCategory				string			`json:"file_category,omitempty"`
	FileSize					*int			`json:"file_size"`
	FileOwner					[]string		`json:"file_owner,omitempty"`  //Array of owners
	Md5Checksum					string			`json:"md5_checksum,omitempty"`
	Sha256Checksum				string			`json:"sha256_checksum,omitempty"`
	CreatedTimestamp			*time.Time		`json:"created_timestamp,omitempty"`
	ModifyTimestamp				*time.Time		`json:"modify_timestamp,omitempty"`
	DeviceUsername				string			`json:"device_username,omitempty"`
	DeviceUid					string			`json:"device_uid,omitempty"`
	UserUid						string			`json:"user_uid,omitempty"`
	OsHostname					string			`json:"os_hostname,omitempty"`
	DomainName					string			`json:"domain_name,omitempty"`
	PublicIpAddress				string			`json:"public_ip_address,omitempty"`
	PrivateIpAddresses			[]string		`json:"private_ip_addresses,omitempty"` //Array of IP address strings
	Actor						string			`json:"actor,omitempty"`
	DirectoryId					[]string		`json:"directory_id,omitempty"` //An array of something, I am not sure
	Source						string			`json:"source,omitempty"`
	Url							string			`json:"url,omitempty"`
	Shared						string			`json:"shared,omitempty"`
	SharedWith					[]string		`json:"shared-with,omitempty"` //An array of strings (Mainly Email Addresses)
	SharingTypeAdded			[]string		`json:"sharing_type_added,omitempty"`
	CloudDriveId				string			`json:"cloud_drive_id,omitempty"`
	DetectionSourceAlias		string			`json:"detection_source_alias,omitempty"`
	FileId						string			`json:"file_id,omitempty"`
	Exposure					[]string		`json:"exposure,omitempty"`
	ProcessOwner				string			`json:"process_owner,omitempty"`
	ProcessName					string			`json:"process_name,omitempty"`
	RemovableMediaVendor		string			`json:"removable_media_vendor,omitempty"`
	RemovableMediaName			string			`json:"removable_media_name,omitempty"`
	RemovableMediaSerialNumber	string			`json:"removable_media_serial_number,omitempty"`
	RemovableMediaCapacity		*int			`json:"removable_media_capacity,omitempty"`
	RemovableMediaBusType		string			`json:"removable_media_bus_type,omitempty"`
	RemovableMediaMediaName		string			`json:"removable_media-media_name,omitempty"`
	RemovableMediaVolumeName	string			`json:"removable_media_volume_name,omitempty"`
	RemovableMediaPartitionId	string			`json:"removable_media_partition_id,omitempty"`
	SyncDestination				string			`json:"sync_destination,omitempty"`
}

type Geoip struct {
	Status 			string		`json:"status,omitempty"`
	Message			string		`json:"message,omitempty"`
	Continent		string		`json:"continent,omitempty"`
	ContinentCode	string		`json:"continent_code,omitempty"`
	Country			string		`json:"country_name,omitempty"`
	CountryCode		string		`json:"country_code2,omitempty"`
	Region			string		`json:"region,omitempty"`
	RegionName		string		`json:"region_name,omitempty"`
	City			string		`json:"city_name,omitempty"`
	District		string		`json:"district,omitempty"`
	ZIP				string		`json:"postal_code,omitempty"`
	Lat				float32		`json:"latitude,omitempty"`
	Lon				float32		`json:"longitude,omitempty"`
	Timezone		string		`json:"timezone,omitempty"`
	Currency		string		`json:"currency,omitempty"`
	ISP				string		`json:"isp,omitempty"`
	Org				string		`json:"org,omitempty"`
	AS				string		`json:"as,omitempty"`
	ASName			string		`json:"as_name,omitempty"`
	Reverse			string		`json:"reverse,omitempty"`
	Mobile			bool		`json:"mobile,omitempty"`
	Proxy			bool		`json:"proxy,omitempty"`
	Query			string		`json:"ip,omitempty"`
	GeoPoint		*GeoPoint	`json:"location,omitempty"`
}

type GeoPoint struct {
	Lat		float32		`json:"lat,omitempty"`
	Lon		float32		`json:"lon,omitempty"`
}

Bug: App fails to start if the inProgressQueries file is empty but exists

If the inProgressQueries files exists, the app will fail to start with the following error:

error getting old in progress queries
panic: error: parsing in progress queries from: /crashplan-ffs-puller/<query_name>inProgressQueries.json unexpected end of JSON input

goroutine 18 [running]:
github.com/BenB196/crashplan-ffs-puller/ffsEvent.FFSQuery(0xc000130600, 0x39, 0xc00005a070, 0x6b, 0xc000122e00, 0x1, 0x4, 0x1, 0x1f90, 0xc000026cf0, ...)
        /app/ffsEvent/ffsEvent.go:44 +0x10f5
created by main.main.func1
        /app/main.go:46 +0x15f

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.