thehive-project / cortex4py Goto Github PK
View Code? Open in Web Editor NEWPython API Client for Cortex
License: GNU Affero General Public License v3.0
Python API Client for Cortex
License: GNU Affero General Public License v3.0
Hi,
Can you give some clarification to the differences between job.status and report['success'] fields, please? In particular when job.status would indicate success but the report indicates unsuccessful.
TIA!
We are trying to configure our Cortex instance in your app so Splunk can talk to it, howerver we are seeing following error when trying to List Cortex jobs form within Splunk:
ERROR cortex:112 - [C26-ERROR] SERVICE UNAVAILABLE - Cortex service is unavailable, is configuration correct ?
Steps taken to troubleshoot
Splunk log
2021-12-03 12:08:10,772 DEBUG common:31 - [S2] Logging mode set to DEBUG
<KV Stored content redacted>
2021-12-03 12:08:11,090 DEBUG common:128 - [S25] Successfully recovering passwords from storage passwords
2021-12-03 12:08:11,090 DEBUG cortex:57 - [C6] Settings recovered
2021-12-03 12:08:11,090 DEBUG common:217 - [S55] Getting this parameter: 100
2021-12-03 12:08:11,091 DEBUG common:223 - [S60] Getting this parameter: -createdAt
2021-12-03 12:08:11,091 DEBUG common:177 - [S35] This instance ID (fa082ff0) returns: apiuser
2021-12-03 12:08:11,091 DEBUG common:163 - [S30] This instance ID (fa082ff0) returns: {'account_name': 'Cortex', 'authentication_type': 'api_key', 'client_cert': '-', 'host': '127.0.0.1', 'organisation': None, 'port': 9001, 'proxy_account': '-', 'proxy_url': '-', 'type': 'Cortex3', 'uri': '/', 'verify': False, '_user': 'nobody', '_key': '61aa075809cf067420127851', 'proxies': None, 'username': 'apiuser', 'password': '**********'}
2021-12-03 12:08:11,091 DEBUG common:185 - [S40] this instance id (fa082ff0) returns: authentication_type=api_key
2021-12-03 12:08:11,091 DEBUG common:185 - [S40] this instance id (fa082ff0) returns: proxies=None
2021-12-03 12:08:11,092 DEBUG common:185 - [S40] this instance id (fa082ff0) returns: client_cert=-
2021-12-03 12:08:11,092 DEBUG common:185 - [S40] this instance id (fa082ff0) returns: verify=False
2021-12-03 12:08:11,092 DEBUG common:185 - [S40] this instance id (fa082ff0) returns: organisation=None
2021-12-03 12:08:11,092 DEBUG common:185 - [S40] this instance id (fa082ff0) returns: type=Cortex3
2021-12-03 12:08:11,092 DEBUG cortex:79 - [C8] Cortex instance will be initialized with an API Key (not a password)
2021-12-03 12:08:11,093 DEBUG cortex:102 - [C20] Cortex object instanciated
2021-12-03 12:08:11,115 ERROR cortex:112 - [C26-ERROR] SERVICE UNAVAILABLE - Cortex service is unavailable, is configuration correct ?
To avoid any problems (SSL, firewall, etc) we've tried to forward the port from Cortex instance to the Splunk SH machine so Cortex actually listens on localhost (127.0.0.1) and default port (9001), so following command works:
curl -H "Authorization: Bearer [REDACTED]" http://127.0.0.1:9001/api/analyzer
Cortex Version: 3.1.1-1
Splunk version: 8.2.2.1
TheHive-Cortex version: 2.1.5
Following exception is raised when calling api.jobs.get_report_async immediately after starting job with api.analyzers.run_by_name.
cortex4py.exceptions.InvalidInputError: Invalid input exception
> requests.exceptions.HTTPError: 500 Server Error: Internal Server Error for url: http://<IP>/api/job/<job_id>/waitreport?atMost=Inf
If I insert a five-second sleep statement before calling get_report_async then it succeeds. The same error occurs if I request the waitreport API endpoint with the requests module with the atMost=Inf parameter.
If I remove the atMost=Inf parameter the request succeeds without needing the sleep statement (i.e. it waits for the analyzer to complete).
Looking at the cortex logs, the following exception is thrown for the request:
2019-04-12 15:00:27,364 [INFO] from org.thp.cortex.services.ErrorHandler in application-akka.actor.default-dispatcher-166 - GET /api/job/<job_id>/waitreport?atMost=Inf returned 500
java.lang.ClassCastException: scala.concurrent.duration.Duration$$anon$2 cannot be cast to scala.concurrent.duration.FiniteDuration
at org.thp.cortex.controllers.JobCtrl.$anonfun$waitReport$2(JobCtrl.scala:148)
at scala.concurrent.Future.$anonfun$flatMap$1(Future.scala:303)
at scala.concurrent.impl.Promise.$anonfun$transformWith$1(Promise.scala:37)
at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:60)
at akka.dispatch.BatchingExecutor$AbstractBatch.processBatch(BatchingExecutor.scala:55)
at akka.dispatch.BatchingExecutor$BlockableBatch.$anonfun$run$1(BatchingExecutor.scala:91)
at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:12)
at scala.concurrent.BlockContext$.withBlockContext(BlockContext.scala:81)
at akka.dispatch.BatchingExecutor$BlockableBatch.run(BatchingExecutor.scala:91)
at akka.dispatch.TaskInvocation.run(AbstractDispatcher.scala:40)
at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(ForkJoinExecutorConfigurator.scala:44)
at akka.dispatch.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
at akka.dispatch.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
at akka.dispatch.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
at akka.dispatch.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
I'm trying to write a little tool to keep monitored range limits.
By the way, I can't get any kind of information.
My main code is:
for analyzer in analyzer_list:
query = And(Eq('analyzer', analyzer["analyzer"]))
jobs = api.jobs.find_all(query)
total_jobs = jobs.__len__()
r = {
"analyzer": analyzer["analyzer"],
"total_jobs_performed_today": str(total_jobs)
}
Any suggestions?
Analyzers are not able to produce files as analysis output before Cortex 3. This issue aims to prepare the availability of this feature.
There is a huge difference in quality performance between using cortex4py and cortex API.
When I use cortex API to run analizer job with post requests I get valid response every time.
When I use cortex4py to do the same - the script fails at very high rate.
Consider two functions:
def run_vt(domain):
result = None
job = capi.analyzers.run_by_name('VirusTotal_GetReport_3_0', {
'data': str(domain),
'dataType': 'domain',
'tlp': 1,
'message': 'alerts verificiation'
}, force=1)
try:
result = job.json()
return result
except Exception as ex:
print(get_time_now(), status('EXPT'), 'Exception running VT analizer:', ex)
return result
return result
def run_vt_request(domain):
headers = {}
headers.update(cortex_auth)
headers.update(content_type)
url = cortex_url + '/api/analyzer/ [id] /run'
j = {
'data': domain,
'dataType': 'domain',
'tlp': 1
}
resp = None
try:
resp = requests.post(url, headers=headers, json=j, verify=False)
return resp.json()
except Exception as ex:
print(get_time_now(), status('EXPT'), 'Exception running VT analizer:', ex)
return resp
return resp
First one fails a lot (HTTP Error 500, Invalid input exception).
Second did not failed once.
I suspect force
parameter placement or interpretation in the cortex4py implementation might be at fault. I am not completely sure.
Is someone able to send me in the right direction to learn more about how the query object is used to get a subset of analyzers, please?
My use case is getting a list of analyzers for dataType ip, domain etc. with a minimum or maximum tlp value. At the moment I have the following which gets analyzers of ip type:
query = In('dataTypeList', ['ip'])
I tried something like the following to evaluate the max_tlp value within the configuration object, but that failed (I didn't expect it would work!):
query = And(In('dataTypeList', ['ip']), Child('configuration', 'max_tlp==2'))
So how would I access and evaluate fields within the configuration fields of an analyzer?
TIA!
Hi, any way to get the used quota of an analyzer?
Thanks
Just to note that I have noticed two small typing errors while going through Cortex4py documentation.
From
users = api.organizations.get_users(org.id, Eq('status', 'Active'), range='0-5', sort='-createdAt')
To
users = api.organizations.get_users(org.id, Eq('status', 'Ok'), range='0-5', sort='-createdAt')
Otherwise active users won't be shown.
From
analyzer = api.analyzers.enable('Test_1_0', {
"configuration": {
"api_key": "XXXXXXXXXXXXXx",
"proxy_http": "http://localhost:9999",
"proxy_https": "http://localhost:9999",
"auto_extract_artifacts": False,
"check_tlp": True,
"max_tlp": 2
}
To
analyzer = api.analyzers.enable('Test_1_0', {
"configuration": {
"key": "XXXXXXXXXXXXXx",
"proxy_http": "http://localhost:9999",
"proxy_https": "http://localhost:9999",
"auto_extract_artifacts": False,
"check_tlp": True,
"max_tlp": 2
}
Otherwise "Invalid input Excetion" will be returned.
Nothing big at all, just small little typing errors which might not necessarily be easy to notice for everyone as they took me quite some time to figure out! Hope this helps :)
Are those samples still valid?
When i try to connect to cortex api with a sample, cortex API response is that Authentication is invalid. I did not pass the api key. The function does not accept such parameter.
Since instead of passing the content of observable
as post
, the post
dictionary is recreated only with the dataType
and tlp
level, it is currently not possible to submit the PAP level to the job:
Cortex4py/cortex4py/controllers/analyzers.py
Lines 49 to 56 in 629069a
In a testing environment I have a Cortex in a Docker container that is still missing analysers. So the list of analysers returned by the API is empty. This causes the following exception when calling cortex4py.analyzers.run_by_name()
with any name:
'NoneType' object has no attribute 'id'
Traceback (most recent call last):
File "/home/m/PeekabooAV-Installer/PeekabooAV/peekaboo/ruleset/engine.py", line 193, in run
result = rule.evaluate(sample)
File "/home/m/PeekabooAV-Installer/PeekabooAV/peekaboo/ruleset/rules.py", line 754, in evaluate
self.submit_to_cortex(sample, cortex_analyzer)
File "/home/m/PeekabooAV-Installer/PeekabooAV/peekaboo/ruleset/rules.py", line 226, in submit_to_cortex
job_id = self.cortex.submit(sample, analyzer)
File "/home/m/PeekabooAV-Installer/PeekabooAV/peekaboo/toolbox/cortex.py", line 543, in submit
job = self.api.analyzers.run_by_name(analyzer.name, params)
File "/home/m/pd/lib/python3.8/site-packages/cortex4py/controllers/analyzers.py", line 89, in run_by_name
return self.run_by_id(analyzer.id, observable, **kwargs)
AttributeError: 'NoneType' object has no attribute 'id'
I can handle the NoneType exception in my code but wanted to report the issue here in case you think it should rather be wrapped into a module-specific exception such as cortex4py.exceptions.NotFoundError
or cortex4py.analyzers.run_by_name()
signal error by e.g. returning None
.
The code of the function:
api.py in get_analyzers(self, data_type)
153 )
154 if data_type is not None:
--> 155 return self.analyzers.find_all()
156 else:
157 return self.analyzers.get_by_type(data_type)
the line 154
shouldn't contain not
.
The goal of this feature is to add a controller that handles all the operations that can be executed agains a responder
I is it way to wait for the job to finish like in this example :
job1 = api.analyzers.run_by_name('FileInfo_6_0', {
'data': '/bin/ls',
'dataType': 'file'
}, async=True)
# The next instruction should return only Success or Failure, because we reached it after the job finished executing.
print(job1.status)
Thank you in advance for your response.
Trying to create a reporting responder using the following reporter as a base. I want the responder to fetch all analyzer job results for each observable in case. The purpose is to have all information included in the report(for a case) so the analyst can edit/remove what is required or not in the final report. I've checked the Cortex4py documentation and you can get (ex: per name or id)the analyzer and jobs details but there is no way to narrow it down to a specific case(using the case ID for example). Basically, I would like to be able to programatically get all analyzer job results for each observable in a case.
Is it something that is feasible? If not, is it something that could be implemented?
Thanks
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.