microsoft / msticpy Goto Github PK
View Code? Open in Web Editor NEWMicrosoft Threat Intelligence Security Tools
License: Other
Microsoft Threat Intelligence Security Tools
License: Other
Is your feature request related to a problem? Please describe.
For most companies, there are likely multiple instances of logs due to data residency or other architecture constraint.
Analyst may want to execute queries and playbook on those multiple instances in one play instead of one by one, to have a global view, to quicken analysis or else.
Describe the solution you'd like
Describe alternatives you've considered
The normal current alternative is to execute playbook once per instance or loop by instances with manual data merging.
Describe the bug
Showed error message: AttributeError: 'list' object has no attribute 'items', when run init_notebook(namespace=globals());
To Reproduce
Successfully installed msticpy-1.3.1
Steps to reproduce the behavior:
❯ python
Python 3.9.6 (default, Jun 29 2021, 05:25:02)
[Clang 12.0.5 (clang-1205.0.22.9)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
from msticpy.nbtools.nbinit import init_notebook
init_notebook(namespace=globals());
Desktop (please complete the following information):
Additional context
<IPython.core.display.HTML object>
Processing imports....
Imported: pd (pandas), IPython.get_ipython, IPython.display.display, IPython.display.HTML, IPython.display.Markdown, widgets (ipywidgets), pathlib.Path, plt (matplotlib.pyplot), matplotlib.MatplotlibDeprecationWarning, np (numpy), msticpy, msticpy.data.QueryProvider, msticpy.nbtools.foliummap.FoliumMap, msticpy.common.utility.md, msticpy.common.utility.md_warn, msticpy.common.wsconfig.WorkspaceConfig, msticpy.datamodel.pivot.Pivot, msticpy.datamodel.entities
Checking configuration....
Setting notebook options....
Traceback (most recent call last):
File "", line 1, in
File "/Users/User/notebook/.venv/lib/python3.9/site-packages/msticpy/nbtools/nbinit.py", line 293, in init_notebook
prov_dict = load_user_defaults()
File "/Users/User/notebook/.venv/lib/python3.9/site-packages/msticpy/nbtools/user_config.py", line 74, in load_user_defaults
prov_dict = _load_query_providers(user_defaults)
File "/Users/User/notebook/.venv/lib/python3.9/site-packages/msticpy/nbtools/user_config.py", line 83, in _load_query_providers
for prov_name, qry_prov_entry in user_defaults.get("QueryProviders").items():
AttributeError: 'list' object has no attribute 'items'
Requests is in maintenance mode and has requirements on outdated packages. It also does not support async
We should replace requests with httpx 0.18 throughout the code base.
Is your feature request related to a problem? Please describe.
AzureSentinel doesn't currently support all the Sentinel REST API features.
Describe the solution you'd like
Expand AzureSentinel to support additional features such as the creating and updating of Bookmarks and Watchlists.
We currently have most of the DP documentation crammed into a single doc.
This mixes provider-specific docs with generic items like creating queries.
As we expand the number of providers this makes the page difficult to navigate and find the relevant information.
We should restructure something like this:
Describe the bug
Currently MSTICPy is mostly ignorant of Azure cloud environment. A number of components require authentication to cloud-specific AAD endpoints.
To Reproduce
Steps to reproduce the behavior:
Expected behavior
We need a single point to configure Azure cloud setting. All relevant components should take this config item when authenticating and performing other operations.
Screenshots
If applicable, add screenshots to help explain your problem.
Hi,
I would like to display an horizontal bar (start - end) for each event in a timeline.
Example: in this case, I would like to visualize how long a domain has been seen in passive DNS data:
import pandas as pd
import json
from msticpy.nbtools import nbdisplay
data = [{"rrtype": "A", "time_last": 1445413846, "time_first": 1444238290,
"count": 15, "rrname": "raybaneyeglasses.us.com.", "rdata":
"151.237.189.86"},
{"rrtype": "A", "time_last": 1445478986, "time_first": 1443590863,
"count": 56, "rrname": "abercrombieandfitchoutletsonline.com.", "rdata":
"151.237.189.86"},
{"rrtype": "A", "time_last": 1445061327, "time_first": 1444404624,
"count": 13, "rrname": "furlaoutletsonline.in.net.", "rdata":
"151.237.189.86"},
{"rrtype": "A", "time_last": 1444735169, "time_first": 1444302797,
"count": 42, "rrname": "burberryoutlet.top.", "rdata": "151.237.189.86"},
{"rrtype": "A", "time_last": 1444948864, "time_first": 1444948864,
"count": 2, "rrname": "discountnfljerseys.top.", "rdata": "151.237.189.86"}]
df = pd.DataFrame.from_records(data)
df['time_first'] = pd.to_datetime(df['time_first'], unit="s")
df['time_last'] = pd.to_datetime(df['time_last'], unit="s")
nbdisplay.display_timeline(data=df, time_column="time_first", source_columns=["rrname", "rrtype", "rdata", "time_last"], group_by="rrname")
Describe the solution you'd like
Add a parameter time_column_end
to msticpy.nbtools.timeline.display_timeline
:
nbdisplay.display_timeline(data=df, time_column="time_first", time_column_end="time_last", source_columns=["rrname", "rrtype", "rdata", "time_last"], group_by="rrname")
Needed for better perf and user experience - esp with large sets of queries
Hi mstic
team,
I wrote a custom driver/data environment that leverages the Azure Python SDK to hit the Azure Resource Graph. This lets us query for resources using the power of the YAML templates. I know about azure_data
, but I think the Resource Graph has access to a much wider body of security relevant information.
Is this something that folks would be interested in if I opened a PR? If so, is there anyone in particular I could work with to get that over the line?
If this isn't the right channel to ask, feel free to nuke this issue.
All the best,
Ryan
Describe the solution you'd like
Support for ElasticSearch data sources in the same way other data providers are supported in MSTICPy
This has been discussed separately to this issue and will be expanded on as work starts.
Describe the bug
When installing msticpy
's latest version v0.8.0
, the module azure-cli-core
has an incompatibility issue with the azure-mgmt-resource
dependency.
ERROR: azure-cli-core 2.12.0 has requirement azure-mgmt-resource==10.2.0, but you'll have azure-mgmt-resource 15.0.0 which is incompatible.
To Reproduce
In a new Python virtual environment, install the latest release from GitHub.
pip install git+https://github.com/microsoft/[email protected]
Expected behavior
I expect the installation to occur without errors.
Output
[...]
Building wheels for collected packages: msticpy
Building wheel for msticpy (setup.py) ... done
Created wheel for msticpy: filename=msticpy-0.8.0-py3-none-any.whl size=315577 sha256=ecfc2754cd6f3f3dc22b135ae938fc7ac98a8059b15f6f0e1d6ad60e6d867728
Stored in directory: /tmp/pip-ephem-wheel-cache-bfkkevlx/wheels/f0/85/c4/d76217db44f43c45c8fe70ea7ab54b5223f2367fe7eae80a57
Successfully built msticpy
ERROR: azure-cli-core 2.12.0 has requirement azure-mgmt-resource==10.2.0, but you'll have azure-mgmt-resource 15.0.0 which is incompatible.
Installing collected packages: Pygments, backcall, ipython-genutils, traitlets, ptyprocess, pexpect, parso, jedi, wcwidth, prompt-toolkit, decorator, pickleshare, ipython, tornado, pyzmq, six, python-dateutil, jupyter-core, jupyter-client, ipykernel, retrying, plotly, psutil, pyjwt, numpy, pytz, pandas, Markdown, urllib3, idna, chardet, certifi, requests, scipy, pillow, pyparsing, cycler, kiwisolver, matplotlib, seaborn, tabulate, colorama, jmespath, pyyaml, argcomplete, knack, pkginfo, portalocker, msal, msal-extensions, oauthlib, requests-oauthlib, isodate, msrest, pycparser, cffi, cryptography, adal, azure-core, azure-mgmt-core, pyopenssl, msrestazure, applicationinsights, azure-cli-telemetry, azure-common, azure-mgmt-resource, pynacl, bcrypt, paramiko, humanfriendly, azure-cli-core, prettytable, lxml, soupsieve, beautifulsoup4, MarkupSafe, Jinja2, Werkzeug, click, itsdangerous, flask, pyperclip, Kqlmagic, attrs, azure-identity, azure-keyvault-secrets, azure-mgmt-compute, azure-mgmt-keyvault, azure-mgmt-monitor, azure-mgmt-network, azure-mgmt-subscription, typing-extensions, packaging, bokeh, wrapt, deprecated, dnspython, branca, folium, maxminddb, async-timeout, multidict, yarl, aiohttp, geoip2, ipwhois, prometheus-client, terminado, pyrsistent, jsonschema, nbformat, Send2Trash, argon2-cffi, jupyterlab-pygments, mistune, pandocfilters, nest-asyncio, async-generator, nbclient, defusedxml, testpath, webencodings, bleach, entrypoints, nbconvert, notebook, widgetsnbextension, ipywidgets, jeepney, SecretStorage, keyring, networkx, threadpoolctl, joblib, scikit-learn, splunk-sdk, patsy, statsmodels, requests-file, tldextract, tqdm, msticpy
Successfully installed Jinja2-2.11.2 Kqlmagic-0.1.113.post1 Markdown-3.2.2 MarkupSafe-1.1.1 Pygments-2.7.1 SecretStorage-3.1.2 Send2Trash-1.5.0 Werkzeug-1.0.1 adal-1.2.4 aiohttp-3.6.2 applicationinsights-0.11.9 argcomplete-1.12.0 argon2-cffi-20.1.0 async-generator-1.10 async-timeout-3.0.1 attrs-20.2.0 azure-cli-core-2.12.0 azure-cli-telemetry-1.0.6 azure-common-1.1.25 azure-core-1.8.1 azure-identity-1.3.1 azure-keyvault-secrets-4.2.0 azure-mgmt-compute-17.0.0 azure-mgmt-core-1.2.0 azure-mgmt-keyvault-7.0.0 azure-mgmt-monitor-1.0.1 azure-mgmt-network-16.0.0 azure-mgmt-resource-15.0.0 azure-mgmt-subscription-0.6.0 backcall-0.2.0 bcrypt-3.2.0 beautifulsoup4-4.9.1 bleach-3.2.1 bokeh-2.2.1 branca-0.4.1 certifi-2020.6.20 cffi-1.14.3 chardet-3.0.4 click-7.1.2 colorama-0.4.3 cryptography-3.1 cycler-0.10.0 decorator-4.4.2 defusedxml-0.6.0 deprecated-1.2.10 dnspython-2.0.0 entrypoints-0.3 flask-1.1.2 folium-0.11.0 geoip2-4.0.2 humanfriendly-8.2 idna-2.10 ipwhois-1.2.0 ipykernel-5.3.4 ipython-7.18.1 ipython-genutils-0.2.0 ipywidgets-7.5.1 isodate-0.6.0 itsdangerous-1.1.0 jedi-0.17.2 jeepney-0.4.3 jmespath-0.10.0 joblib-0.16.0 jsonschema-3.2.0 jupyter-client-6.1.7 jupyter-core-4.6.3 jupyterlab-pygments-0.1.1 keyring-21.4.0 kiwisolver-1.2.0 knack-0.7.2 lxml-4.5.2 matplotlib-3.3.2 maxminddb-2.0.2 mistune-0.8.4 msal-1.0.0 msal-extensions-0.1.3 msrest-0.6.19 msrestazure-0.6.4 msticpy-0.8.0 multidict-4.7.6 nbclient-0.5.0 nbconvert-6.0.5 nbformat-5.0.7 nest-asyncio-1.4.0 networkx-2.5 notebook-6.1.4 numpy-1.19.2 oauthlib-3.1.0 packaging-20.4 pandas-1.1.2 pandocfilters-1.4.2 paramiko-2.7.2 parso-0.7.1 patsy-0.5.1 pexpect-4.8.0 pickleshare-0.7.5 pillow-7.2.0 pkginfo-1.5.0.1 plotly-4.10.0 portalocker-1.7.1 prettytable-0.7.2 prometheus-client-0.8.0 prompt-toolkit-3.0.7 psutil-5.7.2 ptyprocess-0.6.0 pycparser-2.20 pyjwt-1.7.1 pynacl-1.4.0 pyopenssl-19.1.0 pyparsing-2.4.7 pyperclip-1.8.0 pyrsistent-0.17.3 python-dateutil-2.8.1 pytz-2020.1 pyyaml-5.3.1 pyzmq-19.0.2 requests-2.24.0 requests-file-1.5.1 requests-oauthlib-1.3.0 retrying-1.3.3 scikit-learn-0.23.2 scipy-1.5.2 seaborn-0.11.0 six-1.15.0 soupsieve-2.0.1 splunk-sdk-1.6.14 statsmodels-0.12.0 tabulate-0.8.7 terminado-0.9.1 testpath-0.4.4 threadpoolctl-2.1.0 tldextract-2.2.3 tornado-6.0.4 tqdm-4.49.0 traitlets-5.0.4 typing-extensions-3.7.4.3 urllib3-1.25.10 wcwidth-0.2.5 webencodings-0.5.1 widgetsnbextension-3.5.1 wrapt-1.12.1 yarl-1.5.1
Desktop (please complete the following information):
Is your feature request related to a problem? Please describe.
Python 3.6 will reach end of life at the end of 2021. There will be limited support for updates and security fixes.
There are also features of later releases that we would like to use in development.
Describe the solution you'd like
We are proposing to make the minimum supported version of Python 3.8
Describe alternatives you've considered
Python 3.7 should still work but there would be no guarantees
Describe the bug
An error is thrown when AzureSentinel config is not present in the msticpyconfig.yaml file.
To Reproduce
Steps to reproduce the behavior:
# AzureSentinel:
# # Note if you do not specify any settings in the Args key for the AzureSentinel
# # provider, it will default to using your default Azure Sentinel workspace.
# Args:
# WorkspaceID: "your-azure-sentinel-workspace-id"
# TenantID: "your-azure-sentinel-tenant-id"
# Primary: False
# Provider: "AzSTI"
# iplocation = GeoLiteLookup()
# loc_result, ip_entity = iplocation.lookup_ip(ip_address='90.156.201.97')
Expected behavior
If AzureSentinel config is commented out or not included in the config file, silence the warning error. Only provide warning error if AzureSentinel config is present but incorrectly populated.
Follow-on commands to show config file was successfully parsed:
Desktop (please complete the following information):
Is your feature request related to a problem? Please describe.
Currently you have to specify the Azure Sentinel workspace you want to interact with for each and every call you make to Azure Sentinel which is cumbersome.
Describe the solution you'd like
Allow a set of Workspace details to be passed in at _init_
that are then used for all future calls.
MSTICPy fails to load when Azure key exists in msticpyconfig.yaml but has no settings.
Reported by Sreedar Ande @ microsoft
To Reproduce
Steps to reproduce the behavior:
Azure: {}
AzureSentinel:
Workspaces:
...
/anaconda/envs/azureml_py38/lib/python3.8/site-packages/msticpy/common/azure_auth_core.py in <module>
248
249 # externally callable function using the class above
--> 250 _AZ_CACHED_CONNECT = _AzCachedConnect()
global _AZ_CACHED_CONNECT = undefined
global _AzCachedConnect = <class 'msticpy.common.azure_auth_core._AzCachedConnect'>
251 az_connect_core = _AZ_CACHED_CONNECT.connect
252
/anaconda/envs/azureml_py38/lib/python3.8/site-packages/msticpy/common/azure_auth_core.py in __init__(self=<msticpy.common.azure_auth_core._AzCachedConnect object>)
217 """Initialize the class."""
218 self.az_credentials: Optional[AzCredentials] = None
--> 219 self.cred_cloud: str = self.current_cloud
self.cred_cloud = undefined
global str = undefined
self.current_cloud = undefined
220
221 @property
/anaconda/envs/azureml_py38/lib/python3.8/site-packages/msticpy/common/azure_auth_core.py in current_cloud(self=<msticpy.common.azure_auth_core._AzCachedConnect object>)
222 def current_cloud(self) -> str:
223 """Return current cloud."""
--> 224 return AzureCloudConfig().cloud
global AzureCloudConfig.cloud = undefined
225
226 def connect(self, *args, **kwargs):
AttributeError: 'AzureCloudConfig' object has no attribute 'cloud'
Expected behavior
No error
When using pivot functions with a dataframe errors caused by incorrect column names are unclear.
If a column name that isn't present in the dataframe is specified the pivot falls back to using the default value rather than telling the user that the columns isn't in the dataframe. This then leads to an error stating the default value isn't in the dataframe instead (if that is the case).
For the function _check_and_update_db, I am troubleshooting the following:
iplocation = GeoLiteLookup(api_key=value, db_folder = Path('./max_mind_db/'), force_update=True)
elif force_update and auto_update: print( "force_update is set to True.", f"Attempting to download new database to {db_folder}", ) try: db_is_current = self._download_and_extract_archive(url, db_folder) except MsticpyConfigException as no_key_err: warnings.warn(" ".join(no_key_err.args))
Any help would be greatly appreciated! Been fun test driving this python tooling, thanks infosec Jupyterthon :)
Current it really supports Windows security events from Azure Sentinel (that have a couple of custom fields) and Linux Auditd.
It should at least support:
MDE provider currently uses these API definitions
self.req_body = {
"client_id": None,
"client_secret": None,
"grant_type": "client_credentials",
"resource": "https://api.securitycenter.microsoft.com",
}
self.oauth_url = "https://login.windows.net/{tenantId}/oauth2/token"
self.api_root = "https://api.securitycenter.microsoft.com/"
self.api_ver = "api"
We need to move these to use cloud-specific endpoints.
https://docs.microsoft.com/en-us/microsoft-365/security/defender-endpoint/gov?view=o365-worldwide#api
It would be ideal to support regional endpoints as well.
https://docs.microsoft.com/en-us/microsoft-365/security/defender-endpoint/exposed-apis-list?view=o365-worldwide
Hello,
Does this tool support Sentinel in AzureGov?
Describe the solution you'd like
The current MDE integration only support MDE data, by changing the API calls to reach the M365 Hunting API we can support all the M365D data types.
Is your feature request related to a problem? Please describe.
A picture is worth a thousand words. From official documentation example :
From top to bottom:
Describe the solution you'd like
Have both the legend and display be consistent, it would help a lot when visualizing more events.
I can work on a PR this weekend.
Currently VT3 module only supports file, ip, domain, url simple APIs,
It also a generic get_object API
We'd like to be able support some of the richer APIs to retrieve file behaviors, contacted IPs and domains.
Should also add support for outputting detonation/behavior process tree in a format consumable by the Process Tree visualization
Timestamps associated with entities should have this implemented in the base entity in order to help ensure DRP philosophy and maintain v3 compatability.
In addition timestamps should be added to entity edges to better represent temporal connection of entities.
Create UI/container to allow user to build timeline visualization from multiple data sets:
We currently submit queries to KQL magic with time ranges in the query. It is more efficient to have the time range as API parameters.
As well as including the start/end time in the string we should use Kqlmagic time bounds parameters as well.
Kqlmagic syntax - https://github.com/microsoft/jupyter-Kqlmagic/blob/e483b34c605a2d332e078ea74e6015dad5f89d77/HISTORY.md#new-feature-timespan-query-property
For time period spec see https://en.wikipedia.org/wiki/ISO_8601
I have received a tenant_id, client_id, client_scret, workspace_id and workspace_name.
When I use Kqlmagic to connect like the following:
%kql loganalytics://tenant='tenant_id';clientid='client_id';clientsecret='client_secret';workspace='workspace_id';alias='workspace_name'
It works perfectly.
But when I use msticpy to connect using the following:
qry_prov = QueryProvider(data_environment='LogAnalytics')
qry_prov.connect(connection_str="loganalytics://tenant='tenant_id';clientid='client_id';clientsecret='client_secret';workspace='workspace_id';alias='workspace_name'")
It fails to connect.
I am assuming it is because using this data environment I can only supply a tenant_id and workspace_id.
When I try
qry_prov = QueryProvider('MDATP')
qry_prov.connect(tenant_id=tenant_id, client_id=client_id, client_secret=client_secret)
It connects succesfully, but then I can only access qry_prov.MDATP.* and not the qry_prov.SecurityAlert.list_alerts() that I require.
Describe the bug
vt_lookup.lookup_iocs
from vtlookupv3To Reproduce
import pandas as pd
from msticpy import init_notebook
from msticpy.sectools.vtlookupv3 import VTLookupV3, VTEntityType
from msticpy.common.provider_settings import get_provider_settings
import nest_asyncio
nest_asyncio.apply()
init_notebook(namespace=globals());
vt_key = get_provider_settings("TIProviders")["VirusTotal"].args["AuthKey"]
vt_lookup = VTLookupV3(vt_key)
samples_raw = """018ac8f95d5e14b92011cdbfc8c48056ca4891161ed6bdd268770a5b56bb327f
DONOTEXISTDONOTEXISTDONOTEXIST"""
samples = pd.DataFrame({"target":samples_raw.split(),"target_type":"file"})
vt_lookup.lookup_iocs(observables_df=samples, observable_column="target")
Minimized stacktrace
APIError Traceback (most recent call last)
APIError: ('NotFoundError', 'Resource not found.')
The above exception was the direct cause of the following exception:
MsticpyVTNoDataError Traceback (most recent call last)
MsticpyVTNoDataError: An error occurred requesting data from VirusTotal
Expected behavior
A dataframe with 2 rows
018ac8f95d5e14b92011cdbfc8c48056ca4891161ed6bdd268770a5b56bb327f
NaN
or None
value for DONOTEXISTDONOTEXISTDONOTEXIST
(or an equivalent to inform the analyst that the observable was not found on VT)Not finding an observable on VT is fundamentally not an error, it is an event that is potentially expected and this should not prevent retrieving the results for the other observables.
Additional context
msticpy version installed: 1.3.1
The SecurityAlert feature should be wrapped up into the datamodel
Created from issues raised on DL:
Currently working with some custom queries in msticpy. However using the datetime parameter seems to not be working. The query runs (gives no errors), but it says that there are no query results. If I copy the query and run it in Log Analytics or MDE, it gives results. Example is with MDE but same applies with LA.
Describe the bug
I am trying to create a schema from some macOS logs created by the Elastic agent. I have attempted to implement a custom schema as described here. I am getting an error
UnboundLocalError: local variable 'extr_proc_tree' referenced before assignment
that does not appear to be any variable I have control over and am not sure if this is a bug or an error in how I am using this.
To Reproduce
Steps to reproduce the behavior:
import json
import pandas as pd
import yaml
import glob
from msticpy.sectools import ptree
from msticpy.sectools.proc_tree_builder import build_process_tree
from msticpy.nbtools import process_tree
# Assumes this path is correct, may need to adjust depending on where you cloned the repo
files = glob.glob("./**/*.json")
attacks = []
for file in files:
with open(file, 'r') as f:
attack = {}
data = json.loads(f.read())
attack[file] = data
attacks.append(attack)
# picked a single file with 10 events to build a sample tree
for attack in attacks:
for k, v in attack.items():
if k == "./Persistence/persistence_emond_process_execution.json":
df = pd.json_normalize(v['documents'])
from msticpy.sectools.proc_tree_builder import build_process_tree, LX_EVENT_SCH
from msticpy.sectools import proc_tree_build_winlx as winlx
from copy import copy
cust_lx_schema = copy(LX_EVENT_SCH)
cust_lx_schema.time_stamp = "_source.@timestamp"
cust_lx_schema.host_name_column = "_source.host.hostname"
cust_lx_schema.process_id = "_source.process.pid"
cust_lx_schema.parent_id = "_source.process.parent.pid"
cust_lx_schema.cmd_line = "_source.process.command_line"
cust_lx_schema.event_id_column = "_source.data_stream.dataset"
cust_lx_schema.cmd_line = "_source.process.command_line"
cust_lx_schema.event_id_identifier = "endpoint.events.process"
cust_lx_schema.path_separator = "/"
process_tree.plot_process_tree(df, schema=cust_lx_schema)
error output
---------------------------------------------------------------------------
UnboundLocalError Traceback (most recent call last)
/var/folders/6j/kg1frr2j61l2_xx9m01j_x3w0000gn/T/ipykernel_64665/961226578.py in <module>
13 cust_lx_schema.path_separator = "/"
14
---> 15 process_tree.plot_process_tree(df, schema=cust_lx_schema)
~/.local/share/virtualenvs/macOS-ATTACK-DATASET-JrS6VCr6/lib/python3.8/site-packages/msticpy/nbtools/process_tree.py in plot_process_tree(data, schema, output_var, legend_col, show_table, **kwargs)
187 pid_fmt = kwargs.pop("pid_fmt", "hex")
188
--> 189 data, schema, levels, n_rows = _pre_process_tree(data, schema, pid_fmt=pid_fmt)
190 if schema is None:
191 raise ProcessTreeSchemaException("Could not infer data schema from data set.")
~/.local/share/virtualenvs/macOS-ATTACK-DATASET-JrS6VCr6/lib/python3.8/site-packages/msticpy/nbtools/process_tree.py in _pre_process_tree(proc_tree, schema, pid_fmt)
298 missing_cols = _check_proc_tree_schema(proc_tree)
299 if missing_cols:
--> 300 proc_tree = build_process_tree(procs=proc_tree, schema=schema)
301
302 if schema is None:
~/.local/share/virtualenvs/macOS-ATTACK-DATASET-JrS6VCr6/lib/python3.8/site-packages/msticpy/sectools/proc_tree_builder.py in build_process_tree(procs, schema, show_summary, debug)
254 elif schema == MDE_EVENT_SCH:
255 extr_proc_tree = mde.extract_process_tree(procs, debug=debug)
--> 256 merged_procs_keys = _add_tree_properties(extr_proc_tree)
257
258 # Build process paths
UnboundLocalError: local variable 'extr_proc_tree' referenced before assignment
Expected behavior
Expect a plot of processes with parent and child relationships according to example images
Screenshots
If applicable, add screenshots to help explain your problem.
Desktop (please complete the following information):
Smartphone (please complete the following information):
Additional context
Add any other context about the problem here.
When creating a proc tree with proc_tree_build_winlx, some of the dataframes that get created do not create a 'TimeGenerated_orig_par' column. This causes the column to hold the type Object, generating an exception when it's treated as a datetime object later in the code.
Data is pulled in from Splunk windows event logs and used as follows
import pandas as pd
from msticpy.sectools import proc_tree_build_winlx as ptree
from msticpy.sectools import proc_tree_schema
SPLWIN_EVENT_SCH = proc_tree_schema.ProcSchema(
time_stamp="date",
process_name="New_Process_Name",
process_id="New_Process_ID",
parent_name="Creator_Process_Name",
parent_id="Creator_Process_ID",
logon_id="Subject_Logon_ID",
target_logon_id="Target_Logon_ID",
cmd_line="Process_Command_Line",
user_name="Subject_Account_Name",
path_separator="\\",
user_id="Subject_User_Sid",
event_id_column="EventCode",
event_id_identifier="4688",
host_name_column="host",
)
p_tree_win = ptree.extract_process_tree(procs, SPLWIN_EVENT_SCH)
Generating: AttributeError: Can only use .dt accessor with datetimelike values
At line: proc_data[timestamp_col].dt.round("10us").dt.strftime(TS_FMT_STRING)
I believe it becomes a problem on this line:
merged_procs.loc[root_procs_crit, "TimeGenerated_orig_par"] = time_zero
Because not all rows will contain datetime objects as a result
I also had to make the following changes to get it as far as I did I believe because of my custom schema
# Extract synthentic rows for the parents of root processes
inferred_parents = (
merged_procs[root_procs_crit][
[
schema.event_id_column,
schema.host_name_column,
schema.parent_id,
"EffectiveLogonId_par",
"Creator_Process_Name", # Changed from "ParentProcessName"
"parent_proc_lc",
]
]
.rename(
columns={
schema.parent_id: schema.process_id,
"Creator_Process_Name": schema.process_name, # Changed key from "ParentProcessName"
"parent_proc_lc": "new_process_lc",
"EffectiveLogonId_par": schema.logon_id,
}
)
.assign(TimeGenerated=time_zero, EffectiveLogonId=merged_procs[schema.logon_id])
.drop_duplicates()
)
I included my sample DF as json output since it was easy to redact versus pickle.
evtlogs.zip
In the Source Dist on PyPI the setup.py refers to requirements.txt and other files that are not present in the package causing the setup to fail.
Traceback (most recent call last):
File "<string>", line 1, in <module>
File "/tmp/pip-download-4kdqlqg5/msticpy/setup.py", line 39, in <module>
with open("requirements.txt", "r") as fh:
FileNotFoundError: [Errno 2] No such file or directory: 'requirements.txt'
Missing files probably need to be added to https://github.com/microsoft/msticpy/blob/master/MANIFEST.in
Current the widgets have a variety of parameters and properties that don't quite match and don't follow the ipywidgets property names.
This mainly applies to the selection widgets.
To Dos:
Describe the bug
Running https://github.com/Azure/Azure-Sentinel-Notebooks/blob/master/TroubleShootingNotebooks.ipynb in Azure ML got following error during installation of msticpy (full installation log):
Collecting iniconfig
Using cached iniconfig-1.1.1.tar.gz (8.1 kB)
Installing build dependencies ... -� �error
ERROR: Command errored out with exit status 1:
command: /anaconda/envs/azureml_py36/bin/python /anaconda/envs/azureml_py36/lib/python3.6/site-packages/pip install --ignore-installed --no-user --prefix /tmp/pip-build-env-k0pbth3e/overlay --no-warn-script-location --no-binary :none: --only-binary :none: -i https://pypi.org/simple -- 'setuptools>=41.2.0' wheel 'setuptools_scm>3'
cwd: None
Python version: 3.6.9
To Reproduce
Steps to reproduce the behavior:
Describe the bug
I tried to run the build_process_tree on mordor data. It seems to went into a loop. I did some check and seems like "max_depth" in _build_proc_tree() is always -1?
To Reproduce
df2 = mdr_data.small.windows.credential_access.host.empire_shell_rubeus_asktgt_createnetonly()
df2.rename(columns={'Hostname': 'Computer', 'ActivityID': 'TenantId'}, inplace=True)
df_new = df2[['TenantId', 'Computer', 'EventID', '@timestamp', 'NewProcessName', 'NewProcessId', 'ParentProcessName', 'ProcessId', 'SubjectLogonId', 'TargetLogonId', 'CommandLine', 'SubjectUserName', 'SubjectUserSid']].drop_duplicates()
df_new['@timestamp'] = pd.to_datetime(df_new['@timestamp'])
p_tree_win = ptree.build_process_tree(df_new, schema=schema, show_progress=True)
Screenshots
MSTICPy should expand its supported security data sources to include OSQuery - https://www.osquery.io/
The ideal solution would include OSQuery as a data connector in the same pattern as existing data connectors.
The JS callback for the plot_process_tree doesn't have the var
for the variables out_str and py_str, causing the execution to fail, telling the variables were not found.
Need some getting started/tutorial docs for getting started with
Current intro doc is here
https://msticpy.readthedocs.io/en/latest/getting_started/QuickStart.html
Describe the bug
I was following the directions from the document - https://msticpy.readthedocs.io/en/latest/data_acquisition/TIProviders.html#notebook - and received an error when performing the IOC lookup.
To Reproduce
Steps to reproduce the behavior:
Then try to execute the following code from the directions:
Expected behavior
I had expected to see detail.raw_result_fmt get displayed after the print statement "Raw Results". instead there was an error message
Desktop (please complete the following information):
In Azure ML, using pip in a subprocess or using "!pip" from the notebook causes cross-kernel problems for the conda environment.
Specifically if you !pip install a package in py36 conda environment, then switch to py38 environment you cannot either import the package, nor can you !pip install it. Pip finds the package in the py36 environment and claims it is already installed.
The package is however, not available to the py38 kernel.
Change the code in check_and_install_missing_packages to:
Is your feature request related to a problem? Please describe.
I really like the idea behind Access Reviews, but it is very manual in that during the review process, the Reviewer has to manually go an check all the permissions and the access that a user has before they can Approve/Deny. This process is different when doing access reviews for applications, as it lists all the users with access to a particular application.
Describe the solution you'd like
It would be time-saving and efficient if the user's permissions would be listed from within the review portal and the reviewer would not have to click around trying to establish the user's access and if its approved or not. This way, it would then be possible to have corresponding Approve/Deny buttons for each permission.
Describe alternatives you've considered
A corresponding feature that uses AI and a correlation of sort to flag out suspicious permissions based off on various metrics like results from previous Access Reviews, current role, other users with similar roles and sign-in activity.
Additional context
N/A
Describe the bug
Using the MDE provider the DeviceProcess data does not correctly work with the MP ProcessTree. There are two issues:
When trying to display the process tree you get the error:
AttributeError: Can only use .dt accessor with datetimelike values
However, even if you manually update the columns to datetimes, the process still fails with schema mismatch.
Returned schema from MDE provider is
Timestamp: object
DeviceId: object
DeviceName: object
ActionType: object
FileName: object
FolderPath: object
SHA1: object
SHA256: object
MD5: object
FileSize: int64
ProcessVersionInfoCompanyName: object
ProcessVersionInfoProductName: object
ProcessVersionInfoProductVersion: object
ProcessVersionInfoInternalFileName: object
ProcessVersionInfoOriginalFileName: object
ProcessVersionInfoFileDescription: object
ProcessId: int64
ProcessCommandLine: object
ProcessIntegrityLevel: object
ProcessTokenElevation: object
ProcessCreationTime: object
AccountDomain: object
AccountName: object
AccountSid: object
AccountUpn: object
AccountObjectId: object
LogonId: int64
InitiatingProcessAccountDomain: object
InitiatingProcessAccountName: object
InitiatingProcessAccountSid: object
InitiatingProcessAccountUpn: object
InitiatingProcessAccountObjectId: object
InitiatingProcessLogonId: int64
InitiatingProcessIntegrityLevel: object
InitiatingProcessTokenElevation: object
InitiatingProcessSHA1: object
InitiatingProcessSHA256: object
InitiatingProcessMD5: object
InitiatingProcessFileName: object
InitiatingProcessFileSize: int64
InitiatingProcessVersionInfoCompanyName: object
InitiatingProcessVersionInfoProductName: object
InitiatingProcessVersionInfoProductVersion: object
InitiatingProcessVersionInfoInternalFileName: object
InitiatingProcessVersionInfoOriginalFileName: object
InitiatingProcessVersionInfoFileDescription: object
InitiatingProcessId: int64
InitiatingProcessCommandLine: object
InitiatingProcessCreationTime: object
InitiatingProcessFolderPath: object
InitiatingProcessParentId: int64
InitiatingProcessParentFileName: object
InitiatingProcessParentCreationTime: object
InitiatingProcessSignerType: object
InitiatingProcessSignatureStatus: object
ReportId: int64
MSTICPy should support GRR as a data source - https://grr-doc.readthedocs.io/en/latest/
This should follow the same pattern as other MSTICPy data connectors.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.