ukdataservice / qamd

QAMyData, a data quality assurance tool for SPSS, STATA, SAS and CSV files.

QAMyData

QAMyData offers a free easy-to-use tool that automatically detects some of the most common problems in survey and other numeric data and creates a ‘data health check’, assisting with the clean up of data and providing an assurance that data is of a high quality.

Getting Started

See the Wiki for more information!

Authors

• Myles Offord

See also the list of contributors who participated in this project.

License

QAMyData is licensed under the MIT license Copyright (c) 2019 University of Essex (except where otherwise noted)

See the file LICENCE for the full license.

Acknowledgments

• WizardMac / Evan Miller for the amazing ReadStat C library

qamd's Issues

Update wiki Output-Specification

https://github.com/ukdataservice/qamd/wiki/Output-Specification
The command:
qamd path/to/file.dta --output-format html --output out.html
seems to be replaced with
qamd run path/to/file.dta --output-format html --output out.html

New Check: Defined variable Ranges

Given a list of variables with defined ranges:

Var1=1-7
Var2=1-99
Var3=-3-66

Report on values outside this range.

Typo on Wiki - output specification

https://github.com/Raymanns/qamd/wiki/Output-Specification

should be include-locators, not included-locators

New Check for duplicate IDs

User would specifiy ID variable - ala primary_id.

Report any duplicate values for that variable.

foo

HTML Output Default

Make the HTML output deafult. Output format option will still accept HTML & JSON

Cannot compile qamd

ns@bach ~/q/qamd> cargo build --release
Compiling qamd v1.0.0 (/home/ns/qamd/qamd)
error[E0502]: cannot borrow *context as mutable because context.variables is also borrowed as immutable
--> src/check/post.rs:188:16
|
174 | for variable in context.variables.iter()
| ----------------- immutable borrow occurs here
...
188 | dictionary(context, ValueLabelSpellcheck, &words, spellcheck_predicate);
| ^^^^^^^ mutable borrow occurs here
189 | }
| - immutable borrow ends here

error: aborting due to previous error

Sampling of cases prior to checking

Checks such as Regex can take some time to execute with larger files,especially on older or more resource restricted machines.To assuage this issue a subset of the data could be checked by 'slicing'the file prior to running the checks. The user would be required to configurethe size of the sample as a percentage.

Update wiki installation instructions

Seems to be depricated installation information in the wiki regarding installation:
https://github.com/ukdataservice/qamd/wiki/Installation
For example curl https://raw.githubusercontent.com/Raymanns/qamd/master/setup.sh -sSf | sh is not the url for the repository and the file setup.sh does no longer exist (should be setup-debian.sh?)

CSV support

Hello! I believe I first heard about QAMyData from @janetm in April and I just took another look after seeing QAMyData and Dataverse mentioned in the same slide at https://twitter.com/alina_danciu_/status/1177496046181634048

As a developer, I came to this GitHub repo first and saw from the description that QAMyData has support for three proprietary formats (SPSS, STATA and SAS):

I was a little sad that CSV isn't supported (I also checked the README and the wiki) but from looking at https://www.ukdataservice.ac.uk/about-us/our-rd/qamydata.aspx I see that CSV is supported! 🎉 🎉

I think the fix here in GitHub is pretty easy. I would suggest mentioning CSV (and all formats you support) in the description and the README.