cybercentrecanada / assemblyline Goto Github PK
View Code? Open in Web Editor NEWAssemblyLine 4: File triage and malware analysis
Home Page: https://cybercentrecanada.github.io/assemblyline4_docs/
License: MIT License
AssemblyLine 4: File triage and malware analysis
Home Page: https://cybercentrecanada.github.io/assemblyline4_docs/
License: MIT License
Currently, the retention policy is set by the user in the Days to live
. We can set a default, but we can't override that setting. Our org's intention is to not delete the binaries. We would like to set a system retention policy so that we may later use those binaries for further analysis, retro-hunting, etc. If report storage / performance is a concern, perhaps the report could apply to Days to live
but the binary remains along with a record of the user's submission.
Is your feature request related to a problem? Please describe.
When submitting a domain or IP, users should be allowed to pick what country region to egress out of, maybe a general setting in the UI configurable per user would be a way to do this.
Describe the solution you'd like
In some cases we want to egress out of a specific country when submitting a domain or IP in Assemblyline, the only way to do this at the moment is to update the backend proxy settings, we should be able to do this through the UI.
Describe alternatives you've considered
Update values.yaml manually.
Additional context
N/A
When a submission bundle is downloaded, the results.json file can include user VirusTotal API Keys.
Steps to reproduce the behavior:
API Keys should not be visible in shared output.
Other potential related issues:
Debugging use of invalid API Keys may be more difficult if they are removed entirely.
Please update to the TLP version to 2.0.
TLP version 2.0 is the current version of TLP standardized by FIRST. It is authoritative from August 2022 onwards.
https://www.first.org/tlp/
None of the fetchers (Harbor, Docker, DockerHub) are setup to use a proxy. When trying to add a new service that includes $SERVICE_TAG, I get the below error.
File "/var/lib/assemblyline/.local/lib/python3.9/site-packages/flask/app.py", line 2525, in wsgi_app response = self.full_dispatch_request()
File "/var/lib/assemblyline/.local/lib/python3.9/site-packages/flask/app.py", line 1822, in full_dispatch_request rv = self.handle_user_exception(e)
File "/var/lib/assemblyline/.local/lib/python3.9/site-packages/flask/app.py", line 1820, in full_dispatch_request rv = self.dispatch_request()
File "/var/lib/assemblyline/.local/lib/python3.9/site-packages/flask/app.py", line 1796, in dispatch_request return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args)
File "/var/lib/assemblyline/.local/lib/python3.9/site-packages/assemblyline_ui/api/base.py", line 189, in base return func(*args, **kwargs)
File "/var/lib/assemblyline/.local/lib/python3.9/site-packages/assemblyline_ui/api/v4/service.py", line 256, in add_service _, tag_name, _ = get_latest_tag_for_service(tmp_service, config, LOGGER)
File "/var/lib/assemblyline/.local/lib/python3.9/site-packages/assemblyline_core/updater/helper.py", line 138, in get_latest_tag_for_service tags = _get_dockerhub_tags(image_name, update_channel)
File "/var/lib/assemblyline/.local/lib/python3.9/site-packages/assemblyline_core/updater/helper.py", line 184, in _get_dockerhub_tags resp = requests.get(url)
File "/var/lib/assemblyline/.local/lib/python3.9/site-packages/requests/api.py", line 73, in get return request("get", url, params=params, **kwargs)
File "/var/lib/assemblyline/.local/lib/python3.9/site-packages/requests/api.py", line 59, in request return session.request(method=method, url=url, **kwargs)
File "/var/lib/assemblyline/.local/lib/python3.9/site-packages/requests/sessions.py", line 587, in request resp = self.send(prep, **send_kwargs)
File "/var/lib/assemblyline/.local/lib/python3.9/site-packages/requests/sessions.py", line 701, in send r = adapter.send(request, **kwargs)
File "/var/lib/assemblyline/.local/lib/python3.9/site-packages/requests/adapters.py", line 565, in send raise ConnectionError(e, request=request)
ConnectionError: HTTPSConnectionPool(host='registry.hub.docker.com', port=443): Max retries exceeded with url: /v2/repositories/cccs/assemblyline-service-avclass/tags?page_size=5&page=1&name=stable (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7fc9cfa3bcd0>: Failed to establish a new connection: [Errno 111] Connection refused'))
If the boundary of the matched base64 string does not align with the original offset, the expected strings will not be found.
For example, .exePK\x01\x02
can be extracted from each of these strings, but this service will only find that on the first string because the matched offset aligns original offset.
LmV4ZVBLAQI
0LmV4ZVBLAQI
J0LmV4ZVBLAQI
3J0LmV4ZVBLAQI
The ability to search for UTF16 strings like "http" could be simpler if there was a "wide" option.
Most noticeable when core.scaler.service_defaults.min_instances: 0
.
This can lead to service preemption errors on appliances or environments that want to minimize resource utilization.
Is your feature request related to a problem? Please describe.
Assemblyline should have the ability to add tags to a submissions, this can be used to tag campaigns, groups, and more.
Describe the solution you'd like
Each submission should have the ability to be tagged by the analyst, such that the list would appear somewhere along here:
users should than be able to search based on these tags, and also query the API based on these tags.
These tags are not the IOC tags extracted from the submission, here tags refers to a free form text which would allow us add extra information to the submission.
One use case would be tagging a submission as "production" this means that automation can pick the IOCs up from this submission and push them to production.
Another user case would be tagging a submission with a particular APT name, users can than query or search on these tags to find all the malware samples belonging to that APT, similar use case for campaigns.
There should not be a limit to how many tags a submission can have, or maybe controlled through values.yaml
Describe alternatives you've considered
n/a
Additional context
n/a
Is your feature request related to a problem? Please describe.
When setting users to "custom" and manually adding roles back in based on groups, using the "type: role" we should be able to specify an array of values instead of copy pasting the field and changing the role value, this should also be an option for removing roles. As it stands if we want to add or remove roles we need to create a new field group with patterns, type and value, as it only accepts a keyword value.
eg:
- field: groups
pattern: ^pattern1$
type: role
value: alert_manage
- field: groups
pattern: ^pattern1$
type: role
value: alert_view
- field: groups
pattern: ^pattern1$
type: role
value: apikey_access
- field: groups
pattern: ^pattern1$
type: role
value: file_detail
Describe the solution you'd like
- field: groups
pattern: ^pattern1$
type: role
value: [file_detail, apikey_access, alert_view, alert_manage]
Describe alternatives you've considered
When trying the above exceptions is thrown;
raise ValueError(f"[{self.name or self.parent_name}] {value} not in the possible values: { self.values}")\nValueError: [roles] ['alert_manage', 'alert_view', 'apikey_access', 'file_detail', 'heuristic_view', 'obo_access', 'replay_trigger', 'safelist_view', 'safelist_manage', 'signature_view', 'signature_download', 'submission_create', 'submission_delete', 'submission_manage', 'submission_view', 'workflow_manage', 'workflow_view', 'replay_system', 'archive_view', 'archive_manage', 'archive_trigger', 'self_manage'] not in the possible values: {'bundle_download', 'signature_ import', 'signature_download', 'file_download', 'submission_view', 'self_manage', 'alert_manage', 'signature_manage', 'administration', 'archive_download', 'replay_system', 'replay_trigger', 'alert_view', 'apikey_access', 'workflow_manage', 'heuristic_view', 'submission_manage', 'safelist_view', 'archive_view', 'workflow_view', 'submission_delete', 'submission_create', 'safelist_manage', 'signature_view', 'archive_trigger', 'obo_access', 'file_detail', 'archive_manage'}\n"}
The same is true if we try:
- field: groups
pattern: ^pattern1$
type: role
value: file_detail, apikey_access, alert_view, alert_manage
or
- field: groups
pattern: ^pattern1$
type: role
value: ["file_detail", "apikey_access", "alert_view", "alert_manage"]
or
- field: groups
pattern: ^pattern1$
type: role
value:
- "file_detail"
- "apikey_access"
- "alert_view"
- "alert_manage"
Additional context
unless I'm doing something wrong here I think collections of roles are not allowed.
Similar issue on the documentation as seen below:
Full XLSB code deobfuscation is probably out of scope for this service, but extracting and/or combining related subfiles for submission to another service may be appropriate.
https://isc.sans.edu/forums/diary/XLSB+Files+Because+Binary+is+Stealthier+Than+XML/28476/
https://www.virustotal.com/gui/file/156796008e389996f37f423c3f4d0be316a1b7effe14fbe09e2bb242c92bd3ef
https://www.virustotal.com/gui/file/5a6157eefc8d0b1089a5bfdee351379b27baff4c40b432fd22e0cbe1f6102fab
Hi, we would like to know if you have available Diagrams showing the Containers and DBs of Assembly Line, and maybe other component if any, and their interactions.
That would help in understanding the platform Assembly Line.
Assembly Line is split in different repositories: core, base, ui, etc. If it makes sense, can the diagram gather the Components coming from these different parts?
The scaling-manager k8s service account has overly permissive cluster level permissions that could be narrowed to only what is necessary.
e.g. These are the permissions the clusterrole edit has.
$ kubectl describe clusterrole edit
Name: edit
Labels: kubernetes.io/bootstrapping=rbac-defaults
rbac.authorization.k8s.io/aggregate-to-admin=true
Annotations: rbac.authorization.kubernetes.io/autoupdate: true
PolicyRule:
Resources Non-Resource URLs Resource Names Verbs
--------- ----------------- -------------- -----
leases.coordination.k8s.io [] [] [create delete deletecollection get list patch update watch]
configmaps [] [] [create delete deletecollection patch update get list watch]
events [] [] [create delete deletecollection patch update get list watch]
persistentvolumeclaims [] [] [create delete deletecollection patch update get list watch]
pods [] [] [create delete deletecollection patch update get list watch]
replicationcontrollers/scale [] [] [create delete deletecollection patch update get list watch]
replicationcontrollers [] [] [create delete deletecollection patch update get list watch]
services [] [] [create delete deletecollection patch update get list watch]
daemonsets.apps [] [] [create delete deletecollection patch update get list watch]
deployments.apps/scale [] [] [create delete deletecollection patch update get list watch]
deployments.apps [] [] [create delete deletecollection patch update get list watch]
replicasets.apps/scale [] [] [create delete deletecollection patch update get list watch]
replicasets.apps [] [] [create delete deletecollection patch update get list watch]
statefulsets.apps/scale [] [] [create delete deletecollection patch update get list watch]
statefulsets.apps [] [] [create delete deletecollection patch update get list watch]
horizontalpodautoscalers.autoscaling [] [] [create delete deletecollection patch update get list watch]
cronjobs.batch [] [] [create delete deletecollection patch update get list watch]
jobs.batch [] [] [create delete deletecollection patch update get list watch]
daemonsets.extensions [] [] [create delete deletecollection patch update get list watch]
deployments.extensions/scale [] [] [create delete deletecollection patch update get list watch]
deployments.extensions [] [] [create delete deletecollection patch update get list watch]
ingresses.extensions [] [] [create delete deletecollection patch update get list watch]
networkpolicies.extensions [] [] [create delete deletecollection patch update get list watch]
replicasets.extensions/scale [] [] [create delete deletecollection patch update get list watch]
replicasets.extensions [] [] [create delete deletecollection patch update get list watch]
replicationcontrollers.extensions/scale [] [] [create delete deletecollection patch update get list watch]
ingresses.networking.k8s.io [] [] [create delete deletecollection patch update get list watch]
networkpolicies.networking.k8s.io [] [] [create delete deletecollection patch update get list watch]
poddisruptionbudgets.policy [] [] [create delete deletecollection patch update get list watch]
deployments.apps/rollback [] [] [create delete deletecollection patch update]
deployments.extensions/rollback [] [] [create delete deletecollection patch update]
pods/eviction [] [] [create]
serviceaccounts/token [] [] [create]
pods/attach [] [] [get list watch create delete deletecollection patch update]
pods/exec [] [] [get list watch create delete deletecollection patch update]
pods/portforward [] [] [get list watch create delete deletecollection patch update]
pods/proxy [] [] [get list watch create delete deletecollection patch update]
secrets [] [] [get list watch create delete deletecollection patch update]
services/proxy [] [] [get list watch create delete deletecollection patch update]
bindings [] [] [get list watch]
endpoints [] [] [get list watch]
limitranges [] [] [get list watch]
namespaces/status [] [] [get list watch]
namespaces [] [] [get list watch]
persistentvolumeclaims/status [] [] [get list watch]
pods/log [] [] [get list watch]
pods/status [] [] [get list watch]
replicationcontrollers/status [] [] [get list watch]
resourcequotas/status [] [] [get list watch]
resourcequotas [] [] [get list watch]
services/status [] [] [get list watch]
controllerrevisions.apps [] [] [get list watch]
daemonsets.apps/status [] [] [get list watch]
deployments.apps/status [] [] [get list watch]
replicasets.apps/status [] [] [get list watch]
statefulsets.apps/status [] [] [get list watch]
horizontalpodautoscalers.autoscaling/status [] [] [get list watch]
cronjobs.batch/status [] [] [get list watch]
jobs.batch/status [] [] [get list watch]
endpointslices.discovery.k8s.io [] [] [get list watch]
daemonsets.extensions/status [] [] [get list watch]
deployments.extensions/status [] [] [get list watch]
ingresses.extensions/status [] [] [get list watch]
replicasets.extensions/status [] [] [get list watch]
nodes.metrics.k8s.io [] [] [get list watch]
pods.metrics.k8s.io [] [] [get list watch]
ingresses.networking.k8s.io/status [] [] [get list watch]
poddisruptionbudgets.policy/status [] [] [get list watch]
serviceaccounts [] [] [impersonate create delete deletecollection patch update get list watch]
I did some analysis on this a while ago and narrowed the permissions to only what is needed. I confirmed on an EKS deployment over multiple months and more recently on a microk8s cluster with RBAC enabled.
Please consider this PR: CybercentreCanada/assemblyline-helm-chart#63
Users report that notification queue objects had a field "original_selected" set in the submission parameters which is now missing.
This was probably supplementary information being added by ingester for resubmits. Check if that was the case and if so, look into making sure the information is still available in some way when the notification is published by the new post-processing module.
Describe the bug
Using the assemblyline-client, version 4.5.0 to submit a file for analysis, the client 'submit()' is returning an error: "Error: init() missing 1 required positional argument: 'status_code'".
In the assemblyline-client code, the 'ClientError' class takes a 'status_code' as a required parameter, and I think whatever is throwing an exception is trying to instantiate a 'ClientError', but failing to supply a 'status_code'. So, instead of seeing the actual error that originally threw the exception, we are getting an error message from the failed 'ClientError' instantiation.
To Reproduce
I don't think you could reproduce this without using the same file we are trying to submit.
Expected behavior
Something is going wrong with the file submission, and we would expect to get an informative error message showing the actual cause of the problem.
Environment (please complete the following information if pertinent):
Describe the bug
After updating from 4.3.0.stable62 to 4.3.1.stable23 user's are not able to sign-in to AL using oauth. The non-oauth sign-in flow still works as expected.
To Reproduce
The flow is: you get the initial sign-in page, you sign-in with your oauth provider, then it takes you back to AL where you hit sign in one more time. When you click the sign in button, it takes you to a page that says There's definitely a 'bug' here!
The message on Chrome is: Cannot read properties of undefined (reading 'toUpperCase').
The message on firefox is: e[0] is undefined.
Environment (please complete the following information if pertinent):
Discord: https://discord.com/channels/908084610158714900/908717441771794472/1019296110239567963
Does Extract (or any other service) have any processing for NSIS installers? Was hoping to extract [NSIS].nsi if it's available in the exe file
Possible decompilers: https://nsis.sourceforge.io/Can_I_decompile_an_existing_installer%3F
Magic for identify: Nullsoft Installer self-extracting archive
Building on retro hunting, it would be great to add an option to keep the hunt running (for a period of time requiring renewal), which would add the user's yara rule to the Yara scanning service for incoming binaries or alternatively a batch process that runs at a regular interval. Thus if someone else submits a binary that matches another user's rule, they both get a notification of such.
The EmlParser does not perform case insensitive filtering of email headers for submissions with the type "document/office/email".
To Reproduce
Steps to reproduce the behavior:
Expected behavior
The header_filter should be consistent across submission file types
Additional context
The msg parser does not make case-insensitive comparisons
https://github.com/CybercentreCanada/assemblyline-service-emlparser/blob/d1035125f6222255c2e7846dd3d1ef214e1947f1/emlparser/emlparser.py#L63
The eml parser makes case-insensitive comparisons:
https://github.com/CybercentreCanada/assemblyline-service-emlparser/blob/d1035125f6222255c2e7846dd3d1ef214e1947f1/emlparser/emlparser.py#L412
Since the release of RELEASE.2022-10-29T06-21-33Z minio does not support being backed by the host file system. If an admin would chose to change the compose file to bind a host FS to the minio "/data", the following error is triggered when trying to run AL when the latest version of MINIO container is used.
ERROR Unable to use the drive /data: Drive /data: found backend type fs, expected xl or xl-single
Documentation Reference
As a workaround one could modify the compose file to use the last image that supports FS. Like this
image: ${REGISTRY}minio/minio:RELEASE.2022-10-24T18-35-07Z
It might be a good idea to add a note in the documentation and have a way to support host FS, specially since the filestore would usually run on a slower disk.
The basic proposal is to allow analytic services to be updated via the helm chart.
You can already install the analytic service via the helm chart when specifying the REGISTER_ONLY
env var for a given service as the current AL4 helm chart already does. This would take it a step further and allow for service updates. It would allow you to specify an updated service_manifest.yml
along with a new version tag. This can already be done in the existing system if you simply re-run the service install k8s job and specify an updated service_manifest
. The change requested, is when a service is updated (via a k8s job for example), the new service version merges and replaces the existing version. The expected behavior can be replicated by calling the assemblyline python client service.add()
followed by service.update()
.
This is a git ops focused approach allowing the current config of the system to be managed in git.
It would be nice if you could add a column for Internal / External services. That would be great to see at a glance since it involves less network isolation and sending info about your malware to others (even if it's just a hash).
Example hash:
c7dd490adb297b7f529950778b5a426e8068ea2df58be5d8fd49fe55b5331e28
hachoir-subfile output:
# hachoir-subfile c7dd490adb297b7f529950778b5a426e8068ea2df58be5d8fd49fe55b5331e28.doc
[+] Start search on 1563136 bytes (1.5 MB)
[+] File at 0 size=1563136 (1.5 MB): Microsoft Office document
[+] File at 512: Microsoft Office Word document
[+] File at 8775 size=1371772 (1.3 MB): PNG picture: 2480x3508x24
When running this same file through AL, the PNG file is not extracted. An extracted ole object contains the png file, but the image itself does not appear anywhere in the AL output.
Perhaps this could be a deep scan feature, since it may add a lot of artifacts that people may not care much about.
Describe the bug
If kernel FIPS mode is enabled, the UI and services fails because they cannot import crypt.
To Reproduce
Steps to reproduce the behavior:
Expected behavior
Import crypt statement succeeds
Additional context
This issue is fixed in Python 3.10 and higher.
python/cpython@2fa03b1
python/cpython#95231
Describe the bug
NSRL has changed their format in RDSv3, original RDSv2 links don't work and require conversion or adoption of new format.
See: https://www.nist.gov/itl/ssd/software-quality-group/national-software-reference-library-nsrl/nsrl-download/current-rds
As an analyst I want to be able to extract all IOCs related to a submission in the Post-Processing webhook without making additional API calls. Tags related to a submission should be present on the webhook call, this feature would be enabled using a flag in the post processing, and if enabled, the action would enrich the submission results with all the IOCs for that submission.
Is your feature request related to a problem? Please describe.
This release [4.3.0] introduces backward incompatible changes to the yara-python API.
In particular, it will impact: https://github.com/CybercentreCanada/assemblyline-service-yara/blob/3061ac084004db4cb514d192ab88fd142ad9d09e/yara_/yara_.py#L225
Describe the solution you'd like
Update service code to account for this breaking change.
It appears that announcements / notifications from CCCS are presented to all users. If you're on a Kubernetes cluster, this might be great if the updates are instantly deployed. However, if you're on docker or attempting to remain behind the latest stable build, it would seem the user's would get notification of updates to AssemblyLine that the admin hasn't actually implemented. Such notices might also be undesired for the average user, hiding other messages by the system or the admin. It might make sense to distinguish between CCCS build notifications and others. As an admin, it's great to see the changelog. I suggested filtering these messages to the admins or provide a configuration option that does so.
Per conversation with cccs-sgaron on the Discord server.
For default services, add the capability to adjust the scores associated with heuristics while allowing that service to run normally.
Is your feature request related to a problem? Please describe.
Has been requested over the years but there is no official service because of the license requirements & lack of API documentation:
VMRay integration (google.com)
Describe the solution you'd like
Community-written service to leverage VMRay analysis
Describe the bug
The file.parent
element in the ontology result points to the root file of the submission rather than the actual parent.
To Reproduce
Take a file that has child files extracted, and further children extracted from those files so you have a hierarchy 3 or more deep. When retrieving the ontology results for that submission via /api/v4/ontology/submission/<sid>
, the file.parent
element of a level 2 or more child file is specifying the root file hash of the submission rather than the actual parent of the extracted file.
Expected behavior
The file.parent
element of an ontology result should point to the direct parent of that file.
If a user is submitting a file at TLP:R or TLP:R//[context] we should be able to blacklist services from running on such submissions, even if a user has requested to run that service. This will allow us to introduce riskier services without increasing risk.
Is your feature request related to a problem? Please describe.
For every file submitted to Intezer, all extracted files that are downloaded from that submission count against your quota.
https://support.intezer.com/hc/en-us/articles/360021366619-How-is-Your-Analysis-Quota-Calculated-
Describe the solution you'd like
Please provide a service config parameter that lets you disable the download sub-files option as this eats into the quota really quickly.
Describe alternatives you've considered
Given that a file download counts against your quota, I can't think of an alternative solution.
Additional context
Add any other context or screenshots about the feature request here.
If Yara and ConfigExtractor signatures are not present in the system, the Yara and ConfigExtractor service will never terminate and submissions get stuck waiting for these two services to finish.
If no signatures are present in the system, services which require signatures wont run.
We have an ever increasing usecase to run windows services, as it stands we are communicating with external environments to run windows binaries.
Describe the solution you'd like
Assemblyline, should allow us to orchestrate windows containers as well as Linux containers, this will centralise the management of resources to just the k8s cluster instead of setting up and managing auxiliary infrastructure, which does not necessarily scale as well.
Describe alternatives you've considered
Created VMs with an API server to receive process and respond to Assemblyline
Additional context
n/a
TLSH is a fuzzy matching program and library. Given a file (min 50 bytes), TLSH generates a hash value which can be used for similarity comparisons. Similar files will have similar hash values which allows for the detection of similar objects by comparing their hash values TLSH has been adopted by a range of bodies and malware repositories including:
More information about TLSH, along with diff compare examples and reading in via buffer can be found at: https://tlsh.org/
Below is what we currently have for including TLSH as a Service, but we think TLSH would be a nice addition to the AssemblyLine Core to include with other hashes like SSDEEP. Installed via py-tlsh
import shutil
import tempfile
from pathlib import Path
import tlsh
from assemblyline_v4_service.common.base import ServiceBase
from assemblyline_v4_service.common.result import Result, ResultJSONSection
class TlshService(ServiceBase):
def __init__(self, config=None):
super(TlshService, self).__init__(config)
def start(self):
self.log.info(f"start() from {self.service_attributes.name} service called")
def execute(self, request):
self.log.info(f"execute() from {self.service_attributes.name} service called for '{request.file_name}'")
try:
with open(request.file_path, "rb") as sample_file_object:
working_dir = Path(tempfile.mkdtemp())
try:
sample_bytes = sample_file_object.read()
tlsh_digest = tlsh.hash(sample_bytes).lower()
result = Result()
json_section = ResultJSONSection("TLSH Response")
json_section.set_json({'tlsh': tlsh_digest})
result.add_section(json_section)
request.result = result
finally:
shutil.rmtree(working_dir, ignore_errors=True)
result = Result()
except Exception as ex:
self.log.error(str(ex))
raise
Would like to request SAML Authentication support. I'm currently using LDAP in our dev, but production will need to be able to use SAML to authenticate users. Wanted to try and get this on the roadmap. Thanks
When you do a Resubmit using Use the same parameters
, I think you should also add Ignore result cache
. Otherwise you're not really resubmitting it. You're just pulling up the cache report of the prior submission, which you were already viewing. In the original submission, there likely wasn't a cache to ignore. So the system's state has changed and I think the resubmit needs to account for that. I think the likely user intent here would be to actually run the submission again against those services.
Add a limited user role, which would only be able to view its own profile and submissions. It could still be part of a classification group, so that the limited user submissions could still be viewed by an analyst assigned to the same group.
With auto-registration, this could increase the ability for users to submit samples or URLs for analysis when they should not see other submissions.
My first thought was to overload the TLP:RED classification, but I don't think that would work. It still may be appropriate to only allow the submitter to view TLP:RED (since it should be "exchanged verbally or in person"), but that is not simple either.
My user's most requested feature for AssemblyLine is the ability to do retro hunting. We would like the ability for a user to be able to submit a yara rule that scans our filestore.
Describe the bug
Consider this scenario:
User 1 has permissions to pull samples from a restricted data sources because they're in a group that allows them to do so using a SHA256, the file is submitted into Assemblyline as normal.
User 2 does not have permissions to pull the sample from the restricted data source, and does not have the same group/context as user 1 so user 2 cannot see the submission made by user 1. However, if User 2 submits the same SHA256 in Assemblyline, because Assemblyline is not pulling the sample from external sources but from Assemblyline itself (user 1) has already submitted it, User 2 should not be able to do this, as it can be used to bypass access controls on data sources.
To Reproduce
Steps to reproduce the behaviour:
Expected behaviour
TLP permissions should be retained even if the file is being pulled from AL itself, User 2 who does not have permissions to that external data source should not be able to use Assemblyline as a data source to get access to files they don't have the proper permissions to get access too.
Screenshots
n/a
Environment (please complete the following information if pertinent):
Additional context
n/a
It would be nice if the Signature Search also searched on Source
. So if I search bartblaze or reversinglabs, I'd get some Yara hits for Signatures. I couldn't find the Yara (perhaps due to a 10,000 limit on Signatures) so I was searching on the Source and kept getting 0 hits. I didn't realize I needed to search Yara. I did end up finding them by clicking on the source fingerprint.
Some of my yara source updates don't appear to be processing the yara rules. Here are couple public repositories that I've added that say they've downloaded, but show no signatures.
I didn't see a link in the UI to download a submission result (json) as I might get if I used the API. Would be nice to add that as an option.
The ability to search for UTF16 strings like "http" could be simpler if there was a "wide" option.
If a service imports bbcrack and passes it a level not equal to 1, 2 or small_string, then the Level 3 transforms will be attempted.
https://github.com/CybercentreCanada/assemblyline-v4-service/blob/8aa77e6f1ee0c943ba9ff8710edf5ff19c130d80/assemblyline_v4_service/common/balbuzard/bbcrack.py#L730
Slices of bytestrings returns either the ordinal representation of the slice in Python3, or the bytestring in Python2. Since we are still trying to call ord()
in these level 3 transforms, despite using Python3 in AL services this will throw an error.
Not sure if any services are using this level, but if they did the error would be raised.
Describe the bug
We noticed that the extract service will enter into crashing on loop if privileged mode is enabled. This seems to be related to heartbeat, as we can see a heartbeat error when you describe the pod.
To Reproduce
Steps to reproduce the behaviour:
Expected behaviour
Should work in both privileged and normal mode.
Screenshots
Can replicate if required
Environment (please complete the following information if pertinent):
Additional context
Happy to provide any additional information as needed.
Hi,
when inspecting a submission, it would be helpful to have the list of related submissions.
And for each file of the submission, add the list of related submissions as well.
For example :
Submission_A : file1, file2, file3
Submission_B : file2
Submission_C : file3
related submissions of Submission_A : [Submission_B, Submission_C]
related submissions of file2 : [Submission_A, Submission_B]
This way an analyst can easily see if a file has already been seen elsewhere and quickly map the relationships.
This feature would impact the endpoint /api/v4/submission/report/<sid>/
and the front-end I believe :
GET /api/v4/submission/report/<submission_id_A>/
{
"api_response": {
...,
"related_submissions": ["<sid_B>", "<sid_C>"],
"file_tree": {
"sha256_file2": {
"related_submissions": ["<sid_B>"],
...
}
}
}
}
or maybe another endpoint should be created.
Thank's !
Possibility to add Service Aliases for the API, so we can call for alternate names/casings in addition to the real name as the API is case sensetive.
Context:
ELJeffe: When submitting a task via the api, we have "services": {"selected": ["MetaDefender", "...", "..."]}. Seems the services are case sensitive (at least on my initial tests). Would be nice if they weren't as instead of passing various tool names, I now have to specify if "metadefender" then "MetaDefender" for any particular service. I expect the same is true of others, like VirusTotal.
cccs-sgaron: It is case sensitive, you can have two service with the same name with different cases...
cccs-sgaron: Was that a good design choice, probably not, but that prevents us from removing the case sensitiveness in the API
markus-lassfolk: Would it maybe be possible for you to add Service Aliases in the API so we can call for alternate names/casings in addition to the real name?
cccs-sgaron: It's possible. Create a ticket in the Assemblyline ui repo. This might not make it as an high priority item though.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.