shakemap-amp-tools's Introduction

Status

Note: any general purpose ground motion processing functionality that was in this repository is being moved to the groundmotion-processing repository so that this repository will only conctain shakemap-specific methods.

shakemap-amp-tools

Fetch, read, and process ground motion amplitude data.

Introduction

shakemap-amp-tools is a project designed to provide a number of functions ancillary to the ShakeMap project, mostly those having to do with finding and parsing strong ground motion data and converting it into the XML data format used by ShakeMap. This repository includes the following tools:

amps2xml Convert peak ground motions in Excel format to ShakeMap XML data structure.
sm2xml Convert a directory of strong motion data files into ShakeMap xml.

Installing

If you already have a miniconda or anaconda Python 3.5 environment:

For conda package:

conda install amptools

Source code:

git clone https://github.com/usgs/shakemap-amp-tools.git
cd shakemap-amp-tools
bash intall.sh
conda activate amptools

Tools

sm2xml

This tool converts a directory of ground motion time series data into an xml file formatted for ShakeMap. The input file must be in a format supported by thegroundmotion-processing.

amps2xml

This tool converts an excel file of peak ground motion data into an xml file formatted for use by ShakeMap. Excel spreadsheets must contain only tabular (rows/columns) data.

A "complete" peak ground motion spreadsheet would look like this:

reference	Mexican National Seismic Service report dated 2010-01-01
station	NAME	NETID	SOURCE	LAT	LON	ELEV	DISTANCE	INTENSITY	H1					H2					Z
									PGA	PGV	SA(0.3)	SA(1.0)	SA(3.0)	PGA	PGV	SA(0.3)	SA(1.0)	SA(3.0)	PGA	PGV	SA(0.3)	SA(1.0)	SA(3.0)
ACAM	Guanajuato	MX	National Seismic Service	20.043186	-100.716777	5.1	913.1	0.72	4.48	1.31	5.12	7.74	2.93	4.928	1.44	5.632	8.514	3.223	4.256	1.24	4.864	7.353	2.784

A minimum intensity only peak ground motion table would look like this:

reference	Jones and Smith, 2007. Journal of Seismology.
station	lat	lon	network	intensity
ABC	20.123	-118.123	MX	4.5

A minimum pga only peak ground motion table would look like this:

reference	Jones and Smith, 2007. Journal of Seismology.
station	lat	lon	network	unk
				pga
ABC	20.123	-118.123	MX	0.5

Spreadsheet Requirements

The reference section in the first row is mandatory, and should consist of a cell in the first column with "reference" in it, and the cell in the second column containing the reference string.

The required columns are:

station: Usually a short abbreviated code.
lat: Latitude of the station where data was collected.
lon: Longitude of the station where data was collected.
network: Usually a short code indicating the seismic network who collected the data.

In addition to these required columns, there must be additionally at least:

intensity: Modified Mercalli Intensity (MMI) at the station location.

AND/OR:

channel (more on this below), containing at least one of the following:
- pga: Peak Ground Acceleration, in units of %g.
- pgv: Peak Ground Velocity, in units of cm/s.
- psa03: Peak Spectral Acceleration at 0.3 seconds, in units of %g.
- psa10: Peak Spectral Acceleration at 1.0 seconds, in units of %g.
- psa30: Peak Spectral Acceleration at 3.0 seconds, in units of %g.

The channel columns can be one of three different naming schemes:

XXE, XXN, XXZ: Three channels named according to the scheme found at http://www.fdsn.org/seed_manual/SEEDManual_V2.4_Appendix-A.pdf where XXE/XXN are horizontal channels, and XXZ is the vertical channels.
H1, H2, Z: Three channels where the direction of the channel is not known, where H1/H2 are horizontal channels and Z is the vertical channel.
UNK: The "channel" to use when only summary peak ground motion values are known.

For the "XXE/XXN/XXZ" or "H1/H2/Z" naming schemes, at least one of the horizontal channels must be included.

Links to example spreadsheets:

shakemap-amp-tools's People

Contributors

Watchers

shakemap-amp-tools's Issues

Modify streams_to_dataframe to be able to create PGA/PSA from velocity data

https://github.com/usgs/shakemap-amp-tools/blob/master/amptools/stream.py#L296

Add script to test results agains NGA flatfile

Make sure xml output is correct

This seems right?...

https://github.com/usgs/shakemap-amp-tools/blob/master/amptools/table.py#L301

Add plotting

Choose a configuration format for amptools

Options:

yaml: https://pyyaml.org/wiki/PyYAMLDocumentation
configObj:

http://configobj.readthedocs.io/en/latest/configobj.html

Modify table.dataframe_to_xml to handle de-normalized dataframe structure

That dataframe will have the following columns:

station usu. station code
channel something like HHN, HHE, BHN, etc.
imt (pga,pgv,psa03, etc.)
value Float value of IMT
lat
lon
network Contributing network
location Location string
distance Distance of station to origin

Implement smconvert

The purpose of this routine is supposed to be something that allows a user to convert data from one of the amptools supported formats (KNET, GeoNet, etc.) into anything ObsPy supports. We may also write (some) writers for these formats, in particular Cosmos V2, since that seems to be the format used by PRISM.

Define PGM base class

blocks #29

Break processing into a separate module

Add init.py to cosmos (and other) io directories

Why? https://docs.python.org/3/tutorial/modules.html#packages

Implement standard PGM child classes

One class each for:

PGV
PGA
Spectral amplitudes

Make a read method that dynamically picks the correct parser

Similar to obspy, allow for the user to call read(filename), rather than calling the specific format functions.

blocks #59

Standardize network names in readers:

Change CWB to TW
Change all unknown to ZZ
Change geonet to NZ
Change SMC to ZZ or US

Remove extraneous if statements from streams.py

In streams_to_dataframe, there are now a bunch of extraneous checks for old metadata - these should now be safe to remove.

Look into necessity for zero padding (processing)

Change channels in cwb reader to be H1, H2, Z

This will make it consistent with other readers. Check other readers while we're at it.

Add config.py module to handle reading/defaulting configs

smdb can use the .amptools/config.yml file (if present) for getpgm, but requires .smdb/config.yml for putpgm.

Add constants.py module

Containing:

time format for datetime strings in JSON (%Y-%m-%dT%H:%M:%S.%f')
time format for output miniseed/json files (%Y%m%d%H%M%S')
default config dictionary

and any other constants we can think of.

Modify all readers to use seed channel naming conventions

http://www.fdsn.org/seed_manual/SEEDManual_V2.4_Appendix-A.pdf

Basically, our channels should almost always be:
HN1, HN2, HNZ (accelerometers with sample rates 80 Hz or greater)

See the above link for details on other kinds of data/sample rates.

See #74 to see why this is an OK thing to do.

Look into getting data from European ORFEUS site

https://www.orfeus-eu.org/data/strong/

Check readers against binary files

KNET checker breaks when encountering a non-ascii file. Other checkers are likely to have the same problem.

Review strong motion header data - geonet, knet, smc

Find common metadata
Find unique metadata
Turn "HHE" to "H1" unless HHE is the explicit name of the channel

Modify sm2xml output to include orientation="h/z" in "comp" element.

Horizontal channels should be marked as "h", Vertical as "z". Data that does not distinguish btw horizontal and vertical channels will not be parsed.

Use the more conservative corners when filtering two horizontal channels

Brought up in the meeting with Menlo Park. Horizontal channels should be filtered in the same way to prevent errors with rotation.

Write test for PGM base class

Refactor "data processing" into it's own method

Change vertical IMC to be a generic IMC for any channel

Also write test for sm2xml

Create a class for putting and getting from CWB

Similar to the RawInputClient in geomag-algorithms.

Make default values configurable

This applies to amptools.stream. The IMTs returned (specifically the periods for spectral amplitudes) should be configurable.

Create function to format metadata and timeseries data for COMCAT and CWB

Format metadata as QuakeML or JSON
Use obspy to format timeseries to be used by #25

Create empty constructor in StationSummary

Then have current constructor become a class method that calls the new one and then does it's stuff after that.

Add override for 'problem flagged' smc data

Signal-to-noise ratio

Be sure that the code is aware that the what we sometimes call the "signal" window is actually "signal and noise" window or the "total" window. So the signal to noise ratio is actually calculated as

(total - noise)/noise

Change name of filter_detrend to something more general

Discuss list of IMCs

If I run sm2xml on a station with channels H1,H2, and Z, I get Z and H2 back as components. For that tool, we actually don't want greater of two horizontals, as ShakeMap figures that out for itself.

Write tests for store/retrieve methods such that they can at least be tested locally.

We cannot put configuration information into the repository, so set up a system in test scripts where it looks for a file with remote host/key type information (config files?) and if not found, exits. Developers will be able to configure systems so that py.test will exercise these functions locally at least.

Add multisensor grouping capability

Example scenario. There is a building with multiple sensors. The station name, start time, and orientations of the sensors are the same. At this time, these sensor traces will be marked as duplicates.
891018_1.cra-1.0254i.txt
891018_1.cra-1.0254a.txt

Create a global read() method

Create a read() or read_data() method that imports all reader modules and calls the is_X() method until it finds a match. Raise an exception on failure.