Comments (8)
Testing "Sample Collection Date Unit"
Branch: data-bucket
Testing Date: 2021-02-10
Have just done some testing on sample collection date unit
and haven't found any issues with importing, copy-pasting, using the picklist to enter and validate values. All work fine. No matter how I add the date unit, it appears to automatically reformat the sample collection date
with 01
pseudo values (before the validation step). E.g. if I import 2020
it becomes 2020-01-01
when I select year
, or if I paste 2020
it becomes 2020-01-01
even if no unit has been selected.
The only usability concern I have is that if someone is adding sample collection date unit
within the DataHarmonizer, after already having sample collection dates
, they could accidently overwrite values in their sample collection date. E.g. I have 2020-02-18
and then accidently select month
instead of year
the date changes to 2020-02-01
.
from dataharmonizer.
I tested importing with all eligible file types using modified (and updated) versions of the validTestData
as well as the test file provided by damion. However, when I did some tests on the modified (and updated) version of the invalidTestData
I noticed 2020
wasn't automatically converting to 2020-01-01
. I tried seeing what would happen if I paired 2020
with year
, month
, and day
and the result was the following:
Not certain why this is happening, but fortunately the validation process will always catch and draw attention to these occurences.
Edit:
Input: DH1311p_collection-date-unit_test-05 (invalid data - 2020 testing).csv
Output: DH1311p_collection-date-unit_test-05-output (invalid data - 2020 testing).csv
from dataharmonizer.
Test Files:
DH Test_2021-02-10 (sample collection date unit).zip
- DH1311p_collection-date-unit_test-01 (valid data).csv
- DH1311p_collection-date-unit_test-02 (damion's file).xlsx
- DH1311p_collection-date-unit_test-03 (improper date unit pairs).csv
- DH1311p_collection-date-unit_test-04 (invalid data).csv
- DH1311p_collection-date-unit_test-05 (invalid data - 2020 testing).csv
- DH1311p_collection-date-unit_test-05-output (invalid data - 2020 testing).csv
- DataHarmonizer-exampleInput_0.13.11 (pre-release)
- invalidTestData.csv
- validTestData.csv
- validTestData.tsv
- validTestData.xls
- validTestData.xlsx
I only saved the output when there were unexpected results, in the future I will include the output regardless of the results.
Edit:
"DH" stands for "DataHarmonizer"
"DH1311p" stands for "DataHarmonizer version 0.13.11 pre-release"
from dataharmonizer.
So I've made a change that when a spreadsheet is loaded, the program will stop trying to automatically correct dates into a yyyy-mm-dd format, e.g. "2020" in a date field was getting converted into 2020-01-01 on load, but now it remains 2020. That way a user will be able to manually adjust any date rather than program making assumptions about what it should be converted to. The values will trigger validation error to highlight ones that need correction.
The reason a "day" setting kept 2020 as-is is I didn't want to make assumptions about setting day and month component of what was only a year.
Similarly for month, its prompting user for month when only a year is given. In that case it assumes day is 01.
from dataharmonizer.
Also, we have it that no changes are automatically made any more to month/year/day granularity (did this by renaming the "sample collection date unit" field to "sample collection date precision", since the program still involkes the auto-update on any date + unit field. Instead, any given date is converted to the given date granularity only on export to a particular target database.
from dataharmonizer.
Export Testing
I doubled checked this (while testing the CanCOGen-vocabulary-fix branch) and sample collection date precision
combined with sample received date
behaved as you described when imported and exported.
Example
Import:
Export (CNPHI):
Attachments:
DH-Test_2021-02-21 (CNPHI Export - date precision).zip
from dataharmonizer.
Date Auto-Update Concern
The program is invoking the auto-update on any date field, not just the date + unit field pairs.
The following fields have the auto-date formatting to ensure there are value for year, month, and day - but they don't have a paired precision date column to clarify that these are not actually dated "YYYY-01-01".
- symptom onset date
- vaccination date
- most recent travel departure date
- most recent travel return date
- sequencing date
Attachments:
DH-Test_2021-02-21 (CanCOGeN vocabulary fix).zip
from dataharmonizer.
The date auto-format function (which would be applied to all dates in a loaded spreadsheet on load) has been removed from date fields across the board, so malformed dates remain as is and are only highlighted when one presses "Validate".
from dataharmonizer.
Related Issues (20)
- Loading a template modifies root slot definitions HOT 1
- Enumerations with a large number of elements are cut off when displaying the entire list HOT 3
- DH should materialize LinkML dynamic enumerations and display the permissible values with hierarchy preserved HOT 1
- When DH creates menus from LinkML enums, it requires the presence of the optional `text` property HOT 3
- Transform URLs in help popup into clickable links HOT 5
- "Dates" collected by DataHarmonizer are datetime objects HOT 7
- Document the fact that DH JSON is a bare list and not compatible with LinkML tools as is HOT 2
- Implement time and datetime datatypes HOT 2
- Add column help sidebar to core library HOT 2
- Can't install HOT 5
- `linkml.py` does not generate a `schema.js` as expected but a `schema.json` HOT 2
- Schema not loading in DataHarmonizer (only Chrome and Edge; Firefox working) HOT 3
- Adding term normalization / ontology lookup feature to DataHarmonizer HOT 1
- parse() function appears to have repeated regular expression compilation?
- Validation(), specifically setDataAtCell(), can be made much more efficient by eliminating re-render on each column. HOT 2
- Handsontable 13.0.1 and Flatpickreditor.js date column cut/paste challenge HOT 2
- Compress image assets HOT 6
- Customizable Help Menu with Linkage to LinkML generated data model schema documentation HOT 1
- Dropdown picklists including print tags in latest release HOT 3
- Proposal: UI for working with non-enum multivalued slots HOT 2
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from dataharmonizer.