forensicartifacts / artifacts

Digital Forensics artifact repository

Digital Forensics Artifacts Repository

A free, community-sourced, machine-readable knowledge base of digital forensic artifacts that the world can use both as an information source and within other tools.

If you'd like to use the artifacts in your own tools, all you need to be able to do is read YAML. That is it, no other dependencies. The Python code in this project is just used to validate all the artifacts to make sure they follow the specification.

For more information see:

• Project documentation: https://artifacts.readthedocs.io/en/latest

Contributing

Please send us your contribution! See the developers guide for instructions.

Contact

• [email protected]
• Artifacts channel of Open Source DFIR Slack

artifacts's Issues

Artifact package pollutes the environment.

The artifacts package installs files in the top level of site-packages which breaks any other packages installed in the same virtualenvs. It should create a single directory and install itself into it. Instead this causes a generic name like "utils" to be at the top of the import path thus masking any other such package.

⟫ virtualenv Test
New python executable in Test/bin/python
Installing setuptools, pip...done.
⟫ source  Test/bin/activate
(Test)⟫ pip install --upgrade pip 
Downloading/unpacking pip from https://pypi.python.org/packages/b6/ac/7015eb97dc749283ffdec1c3a88ddb8ae03b8fad0f0e611408f196358da3/pip-9.0.1-py2.py3-none-any.whl#md5=297dbd16ef53bcef0447d245815f5144
  Downloading pip-9.0.1-py2.py3-none-any.whl (1.3MB): 1.3MB downloaded
Installing collected packages: pip
  Found existing installation: pip 1.5.4
    Uninstalling pip:
      Successfully uninstalled pip
Successfully installed pip
Cleaning up...
(Test)⟫ pip install artifacts
Collecting artifacts
Collecting PyYAML>=3.11 (from artifacts)
Installing collected packages: PyYAML, artifacts
Successfully installed PyYAML-3.12 artifacts-20170522
(Test)⟫ python
Python 2.7.6 (default, Oct 26 2016, 20:30:19) 
[GCC 4.8.4] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import utils
>>> utils.__file__
'/tmp/Test/local/lib/python2.7/site-packages/utils/__init__.pyc'

Remove explicit format type indicators

Replace {0:s} by {0}

Also see: http://stackoverflow.com/questions/24170519/python-typeerror-non-empty-format-string-passed-to-object-format

Check Registry key for value definitions

Make sure that the Registry key definitions do not implicitly want the default value to be returned.

update documentation

per: #197

Empty value is what we decided on for default key value? We should put that in the style guide.

add test_data to to sdist package

add test_data to to sdist package

rename definitions to data

Improve documentation

• describe Parameter expansion and globs
  • describe use of ** notation and **[0-9], also why not use **{2} which is closer to regexp notation
• describe provides
• add section about naming conventions #25
• merge style and definitions guide: #42

Deprecate provides

Deprecate provides

Per GRR team, provides are used in interpolation dependency resolving, but provides make GRR more fragile.

• remove provides from GRR
• undo changes in #506 seeing this currently breaks GRR tests
  • #612

Deprecate LinuxWtmp

Determine if LinuxWtmp is used by GRR deprecate this in favor more generic LinuxUtmpFiles

validator add support for COMMAND source type

Artifact collector agent

Hi, how are these usually utilised? What tools consume these (apart from GRR).
Are there any stand alone agents?
TIA

Windows Path and File type sources missing path segment separator

Many artifact defintions in windows.yaml are missing a separator attribute for the FILE type, this means that a / is returned by default from artifacts when it should be a \.

set up travis testing

Duplicate artifacts: WindowsLoadAndRunValues and WindowsProgManRunKeys

Those 2 artifacts in windows.yaml seem to be duplicate (same registry key / value, same url):
WindowsLoadAndRunValues and WindowsProgManRunKeys

Check if WindowsEnvironmentVariableAllUsersProfile should be a REGISTRY_VALUE

Check if WindowsEnvironmentVariableAllUsersProfile should be a REGISTRY_VALUE

https://support.microsoft.com/en-us/kb/214653

Add Windows ETWL files definitions

Also see: http://www.hecfblog.com/2018/06/etw-event-tracing-for-windows-and-etl.html

Improve artifact definition specification

Done

• Change request: #11
  • Rename collectors to sources
  • Split Registry key and value types
  • add file and path separator
• update documentation: #36

To do

• add tags/labels e.g. for persistence method
  • this is not going to be very useful as filter method since there are many persistence methods
  • Labels will be removed in #465
  • separate "trait" definitions might be more useful, especially if they can be pro-grammatically validated
• Make path relative to the file system root (absolute?)
• Change provides so that it has clear type indicators like sources
• Define a way to specify data streams
• have artifact names contain type information e.g "files" in "browser history files"
  • add source type to artifact name e.g. instead of ChromeHistory use ChromeHistoryFiles
• define environment variables (path expanders)

Based on #12 there is a need to specify:

• NTFS ADS names
• HFS resource fork
• extended attribute names

Add Application label

• Add Application label

Improve validator

• check for key paths starting with %current_control_set%
  • #120
  • currently non-fatal fix GRR to support CurrentControlSet
• Warn that '%%users.homedir%%\AppData\Roaming can be replaced by %%users.appdata%%
• Warn that '%%users.homedir%%\AppData\Local can be replaced by %%users.localappdata%%
• enforce naming conversions
  • #25
  • add legacy indicator for GRR compatibility in legacy.yaml
• warn about use of **
• check for trailing whitespace
• add check to see if GRR and plaso dependencies are broken
  • https://github.com/ForensicArtifacts/artifacts/wiki/Dependencies
• #181
• move HKEY_CURRENT_USERS check(

  artifacts/artifacts/source_type.py

  Line 279 in 8f526d1

  if key_path.startswith(u'HKEY_CURRENT_USER\\'):

  ) to validator
• change validation test not to use validator tool and remove __init__.py from tools
• check path expansion variable (#301)

https://en.wikipedia.org/wiki/Special_folder

Broken link in Readme

The "definitions directory" link in the readme looks broken.

Use yapf to unify style

• Use yapf to unify style @joachimmetz
• add travis test to check style using yapf @destijl
• Once yapf 0.11.0 is available add COALESCE_BRACKETS to config

Add email address to AUTHORS file

Matt can you add a public share-able email address to: https://github.com/ForensicArtifacts/artifacts/blob/master/AUTHORS

Update ACKNOWLEDGEMENTS file

File is currently empty, we need to add the contributors.

Improve cloud storage client artifacts

• Add One Drive/Sky Drive
  • add information to https://forensicswiki.xyz/wiki/index.php?title=One_Drive
• Google Drive add relevant paths on non-Windows operating systems

We should rename ARTIFACT to ARTIFACT_GROUP or similar

Here's something I've seen a couple of times now, it's not obvious that the REGISTRY_VALUE is the artifact. People think they need to create a separate ARTIFACT object to hold it. In reality you only need to do this if you are grouping artifacts into a meta artifact.

ARTIFACT_GROUP
ARTIFACT_LIST
META_ARTIFACT

would probably be more intuitive names.

name: ConfickerIndicators
doc: Conficker Indicators.
sources:
- type: ARTIFACT
  attributes:
    names:
      - ConfickerWindowsRegKeys
supported_os: [Windows]
urls: ['https://www.iocbucket.com/iocs/e2903921f8b21d71dc88f34bee8b2f3491ea4656']

---
name: ConfickerWindowsRegKeys
doc: Windows Conficker IOCs.
sources:
- type: REGISTRY_VALUE
  attributes:
    key_value_pairs:
    - {key: 'HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\[RANDOM CHARACTERS]\"ImagePath"', value: '"%System%\svchost.exe -k netsvcs"'}
    - {key: 'HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\[WORM GENERATED SERVICE NAME]\"ImagePath"', value: '%SystemRoot%\system32\svchost.exe -k netsvcs'}
    - {key: 'HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows\CurrentVersion\explorer\Advanced\Folder\Hidden\SHOWALL\"CheckedValue"', value: '0'}
    - {key: 'HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\Tcpip\Parameters\"TcpNumConnections"', value: '00FFFFFE'}
supported_os: [Windows]
urls: ['https://www.iocbucket.com/iocs/e2903921f8b21d71dc88f34bee8b2f3491ea4656']

check if update version updates config/dpkg/changelog

check if update version updates config/dpkg/changelog

Add SCCM logs definition(s)

https://blogs.msdn.microsoft.com/lxchen/2009/04/03/a-list-of-sccm-log-files/

Fix Python 3 tests failing on Travis

Fix Python 3 tests failing on Travis. Feel free to grab this one otherwise I'll have a look when time permits.

Consider adding a FindSpec artifact definition

Though most artifacts (so far) are path/file based it would be useful to be able to specify paths in a more flexible way similar to find specs (as provided in GRR and dfVFS).

tests on AppVeyor failing

• Fix yapf missing from l2tbinaries
• Fix TLS issue - https://ci.appveyor.com/project/joachimmetz/artifacts/build/103#L11

Windows version conditions do not apply for Windows 10

Some artifacts have conditions in the form of os_major_version >= X AND os_minor_version >= Y.
This fails starting with Windows 10, which can have version 10.0. A condition like os_major_version >= 6 AND os_minor_version >= 1 will fail here although the artifact applies.

One example is

artifacts/data/windows.yaml

Line 1335 in 015b375

conditions: [os_major_version >= 6 AND os_minor_version >= 1]

I can think of multiple solutions to this:

• More complicated conditions like (os_major_version >= 6 AND os_minor_version >= 1) OR os_major version >= 7
• A semantic version compare, something like os_version >= "6.1"

Add functionality to read, edit, write artifacts

Per request by @pidydx

Add functionality to read, edit, write artifacts

Add artifact for RecentFileCache.bcf

Add artifact for RecentFileCache.bcf

History files marked with Configuration Files category

Per #119

The UsersShellHistory and RootUserShellHistory artifact labels are marked with the Configuration Files category. Change this into History Files or equiv.

code clean up

• Update codebase to make return values consistent

Add artifacts for Sophos logs

https://community.sophos.com/kb/en-us/43391
https://community.sophos.com/kb/en-us/36207

Remove unused test_lib GetTestFilePath function

Remove unused test_lib GetTestFilePath function

document artifact definition naming convention

I have noticed there is a lot of variance between the names and labels for similar things. What is the best way to align all of these things?

fix duplicate TYPE_INDICATOR_COMMAND

https://github.com/ForensicArtifacts/artifacts/blob/master/artifacts/artifact.py#L60

Originally reported by @jdavid

Change behavior %%current_control_set%%

%%current_control_set%% now maps to HKEY_LOCAL_MACHINE\SYSTEM\ControlSet001 however this should be ControlSet001. The expansion should be used as HKEY_LOCAL_MACHINE\SYSTEM\%%current_control_set%% to match the behavior of the OS more closely.

Clarification on Wildcards in REGISTRY_KEY

Hi everyone,

this is probably covered by #23, but since I'm working on an extractor for artifacts, I'd like to understand how wildcards are used in the artifacts:

Many of the REGISTRY_KEY artifacts have a \* at the end. My understanding is that this means a subkey. This is strange, because some of the defined registry paths (e.g. the Autorun "Run"-Keys in WindowsRunKeys) clearly do not have subkeys, but still are given with a \* at the end. Although not all of them do, for example WindowsControlPanelFilePaths.

Can you clarify if a wildcard at the end of a REGISTRY_KEY path means "All subkeys" or "All values in this key" or both?

Thanks!
Demian

Merge style guide and definitions

Move this into a single asciidoc:

• https://github.com/ForensicArtifacts/artifacts/blob/master/docs/style_guide.adoc
• https://github.com/ForensicArtifacts/artifacts/wiki/Artifact-definitions

Flaky yapf unit test with newer versions of yapf

I believe google/yapf#228 changing to exit code 0 has caused https://github.com/ForensicArtifacts/artifacts/blob/master/tests/style_test.py#L17 to not properly detect style violations.

fix failing yapf test

https://travis-ci.org/ForensicArtifacts/artifacts/jobs/328929298

likely cause, changes due to new version of yapf

• update yapf in GIFT PPA
• update yapf in l2tbinaries for win32, win64
• update yapf in l2tbinaries for macos
• updated source https://codereview.appspot.com/339380043/

exception.message is not portable to Python 3

artifacts/tools/validator.py

Line 44 in 6b789d6

except errors.FormatError as exception:

Artifact Sources deprecated fields

Should supported_os, conditions, and returned_types be getting removed from sources?
Comments in the code indicate they should be removed.
Documentation on style indicates they are correct.

I think ideally ArtifactDefinition.AppendSource() would create a source with all of its valid fields rather than attaching supported_os, conditions, and returned_types in the ArtifactsReader.
Also Source.CopyToDict() should return the entire Source as dict (type, supported_os if defined, conditions if defined, returned_types if defined, attributes) rather than constructing it in ArtifactWriter.

I didn't want to refactor any of this without knowing what direction supported_os, conditions, and returned_types should be going in.

add tests for stats script

add tests for stats script

Add GRR artifact definitions

It would be nice to use this to check which GRR artifact definitions exist.

Improve validation/linting

Improve validation/linting to:

• detect multi line documentation string that does not have long docs with a white line after the first line. Also see Style Guide
• detect duplicate paths
  • Registry keys
  • Registry values
  • etc.
• non existent artifact names defines in meta artifacts
• check for WOW64 missing Registry key aliases
• detect cyclic artifact groups
• check naming
  • artifact definitions in windows.yaml should always start with Windows
• detect duplicate sources section?

sources:
sources:
- type: FILE

Add means to register custom artifact types

• Add means to register custom artifact types.

Improve documentation string of Macos Doc plist

artifacts/data/macos.yaml

Line 107 in dc59c8b

doc: Dock database

And rename artifact to MacOSDockPlistFile