Repository of issues for Dryad project boards
Home Page: https://github.com/orgs/datadryad/projects
This repository is used to support the product development roadmap.
Work will be defined via the issue tracker and tracked via Github projects
(changed description) We need to get Travis.ci working with both the dryad and the stash repositories in github to run tests on those repos.
After logging in and if no datasets we'd have a notification like in this wireframe (forget about the GDPR on that screen for now).
https://jxosix.axshare.com/#g=1&p=my_datasets_-_v1
Then they 'd go through this validation routine.
https://jxosix.axshare.com/#g=1&p=migrate_data_-_v1
I believe what this would switch the current_user to the matching email record and update any information in that user record and remove discarded user record.
It's great that we can commit and see results on Travis.ci, but for writing tests and being sure they work, that long lag time is really unacceptable.
Tests are working locally on my machine, but I need good instructions for getting them set up for others so they can run them easily, also.
I requested ORCID add these login callbacks to our current dash (sandbox login) application
https://dryad-dev.cdlib.org/stash/auth/orcid/callback
https://dryad-stg.cdlib.org/stash/auth/orcid/callback
http://daisiedash.datadryad.org/stash/auth/orcid/callback
We need servers set up and they may involve a number of things. Development environment will be most time-critical to get into a basic useful state.
The development environment has running UI code, but it needs some additional things done to fully function.
See https://github.com/CDL-Dryad/dryad-product-roadmap/wiki/Migration
Perform the database-side user migration.
https://github.com/CDL-Dryad/dryad-product-roadmap/wiki/Migration
We want to keep some Dryad data that may or may not be actively used in the new system
It is useful to be notified of GitHub activity through Slack.
https://github.com/CDL-Dryad/dryad-product-roadmap/wiki/Migration
Some Dryad data has readme files on the file level. It sounds like we want to munge a related readme filename and add it to the file list and send it somehow.
Configuration of shibboleth to work with dryad-dev.cdlib.org end point.
Remove "connect your ORCID" icon from the metadata entry page since they will already by logged in with orcid.
Also remove anything in the callbacks for this.
Note that we still need a callback for co-author orcids and people that receive an invitation to connect an orcid.
Points: 1
I believe this might be how we take care of this.
https://github.com/CDL-Dryad/dryad-product-roadmap/wiki/Migration
We need to be able to validate metadata besides through the review screen or attempting to submit a dataset
Currently it is at https://github.com/CDL-Dryad/stash/blob/master/stash_api/basic_submission.md but it should go with the rest.
We also have OpenAPI documentation at https://dash.ucop.edu/api/docs/ .
https://github.com/CDL-Dryad/dryad-product-roadmap/wiki/Migration
I don't know exactly what this will look like but it can start with a small subset of datasets to test the process, most likely.
We'll probably need a fuller process after that.
https://github.com/CDL-Dryad/dryad-product-roadmap/wiki/Migration
The API needs to have some additional things for the Dryad migration.
I may need to talk through more about what the needs are here.
What would it take for old Dryad to crunch against CoP so that we can transfer these stats to new Dryad system? Scott to talk with Ryan.
(description copied from CDLUC3/stash#2 by @dmolesUC3)
When I first set up RuboCop, I assumed # rubocop:disable was block scoped, but it turns out it's meant to be paired with a matching # rubocop:enable. Without that, it means "disable till further notice".
We've put # rubocop:enable in a few places (e.g. Resource::duplicate_filenames), but there are still a lot of orphan disables out there. At some point we should clean these up, and fix any style issues we discover in code where checks were unintentionally disabled.
Login is more of a milestone, probably, but putting some things in here where we can find them for background.
Jane has created this flow and these wireframes for us. I believe the email flow is essentially OK, though the login flow wasn't complete for all the things we talked about. I'm sure we were both confused about this and could've used more definition.
We talked through a simplified login flow on Friday among our team: login_v2.pdf The main point of this flow is that everyone will log in with ORCID. The possible additional steps on an initial login will then be 1) associate that you're at a partner institution by logging in by Shibboleth and 2) claim/migrate your account if you were a previous Dryad user by way of something similar to a 2-factor authentication flow by getting a code to validate your email.
Seems like there is also a feature I'd forgotten about in there which is a GPDR acceptance? Also "missing data" thing for users on the list of datasets page.
When logging in for the first time, if the user does not have a previous Dryad account, they should be presented with information about the Dryad's Privacy Policy and Terms of Service, and asked to accept the Terms of Service.
The location within the login process should be after the Orcid login (and after institutional login). It could be included within the data migration page, but it may be better to present it as a separate page.
The current wording from old Dryad can be re-used:
The links should go to:
We'd like to be on a rails that is getting more than just security fixes before we're even more obsolete.
Unit tests are going to start breaking in a number of areas
It sounds like we should get the account from them and configure it to be used by the new Dryad code.
Also we still need to support old EZID stuff for previously submitted datasets?
Jane has https://jxosix.axshare.com/
There are two main options. Non-partner just sets the tenant to be Dryad.
Choosing a partner does a secondary (shibboleth) login and then comes back to set the tenant to be more specific after login is validated.
The Dryad option is already done.
Once shibboleth is working it should call back to the sessions controller as shown at https://github.com/CDL-Dryad/stash/blob/master/stash_engine/app/controllers/stash_engine/sessions_controller.rb .
It will be the callback method that it will go back to. A tenant_id parameter should be pass along in the URL.
Once the user has successfully validated their shibboleth then their tenant_id should be written to that field in their user record. (StashEngine::User.tenant_id)
So that could be done before redirecting them to the dashboard path.
Once a user's tenant has been set after logging it, it does not need to be set again on subsequent logins.
We have an almost working deploy on uc3-dryad-dev.cdlib.org but are running into issues.
IMPORTANT, there is a branch for this stuff that I don't want to merge into the master branch until the deploy works. It has Capistrano changes and other stuff (and I set up the server like at https://confluence.ucop.edu/display/UC3/dryad_server_setup_steps ).
The branch you want is here: https://github.com/CDL-Dryad/dryad/tree/deploy_test and it's called deploy test. You might need a working application to deploy with Capistrano (maybe). I think this gives the basic steps. https://github.com/CDL-Dryad/dryad/blob/master/documentation/dryad_install.md Our private config repo is at https://github.com/cdlib/dryad-config.
After tall that fun, you should be able to type 'cap development deploy' and it will deploy to our new dryad-dev.cdlib.org server.
Right now it fails because of missing libraries on the server when it tries to compile a native-code gem. Jim needs to install it.
We hope after that then it will deploy and run and you could log in or try to log in to the application. Or maybe we'll have to fix shibboleth problems. Yay.
How to test:
After new dataset version is being edited in the UI, go in and find the version in the database and set the 'skip_datacite_update' flag to 1.
Submit the dataset (or new version of it).
The dataset will either not update the metadata in EZID (or else will not exist for the first version).
A user should be able to choose a partner institution (or choose none) if they are new and don't have something set. We have this example at https://jxosix.axshare.com/#g=1&p=affiliated_with_a_partner_ .
If they choose a partner and log in with Shibboleth then we need to set the tenant to the correct ID in the database for that tenant. If they choose not a partner then they'd be a generic Dryad tenant for submission.
https://github.com/CDL-Dryad/dryad-product-roadmap/wiki/Migration
The current dryad has 24-hour public urls. It sounds like these will be used to ingest things into Merritt. It sounds like they need to be longer than that.
Gauge how much work it would entail to standardize old Dryad usage metrics before migrating
We want to move code over so that we have a separate space for it rather than just branching in the current repositories. I believe we can fork our current code to make this happen.
The current three repositories we've been using are below and development currently happens in the "development" branch. This could all be changed in the future.
https://github.com/CDLUC3/dashv2
https://github.com/CDLUC3/stash
https://github.com/cdlib/dash2-config (much of the info in here needs to stay private)
Our new space that Marisa created is at https://github.com/CDL-Dryad
Marisa created the repository at https://github.com/CDL-Dryad/dryad . This could be for the main app (currently dashv2) or if it doesn't make sense, please create a new repo.
The current stash repo could be called differently. It contains engines and gems where the heavier functionality lives. Maybe "dryad-engines" or whatever makes most sense to you.
The config repository has sensitive information so needs to stay private. I'm not sure we can create a private repo under CDL-Dryad because a paid account may be needed for private repos. CDL has a paid account under the /cdlib space, so if needed we can create a new private repository there.
I've asked Melissanne and Elizabeth to figure out what their needs are for Stripe invoicing and create an account. Once the account credentials are created we can hook it up to our system.
Stripe account set up by Elizabeth.
Our need for Stripe:
I need Stripe to be triggered to invoice a submitter when a data submission is approved by the curation team, if there is no fee waiver or sponsorship for the DPC. If a payment fails, I need Stripe to email the submitter with instructions for updating their payment information. When a payment is made, I need Stripe to send the researcher a customized receipt.
https://github.com/CDL-Dryad/dryad-product-roadmap/wiki/Migration
I think this will be a Dryad-paid bucket that Merritt will use for a collection? Maybe Ryan/David know more.
It looks like these fields are no longer needed in the Dryad system with a main ORCID login. The table containing these is stash_engine_users.
I believe for now we'll want to keep tenant_id (dryad for a generic login and some other value for more specifics for partner), though the way tenants will be used might be a little different soon.
To remove these fields, you'll want to read over http://guides.rubyonrails.org/v4.2/active_record_migrations.html to get an overview of how migrations work to track state of the database and keep it in sync with the code. If you want to get an overview of ActiveRecord you might end up using it sometime soon, also (but isn't necessary yet for migrations).
http://railscasts.com/ is also a really good resource for tutorials, though some of them are out of date for earlier versions of rails because they've been abandoned and not updated in a while. They still are good for some general concepts or an outline of how something works, though. Anything Rails 3 onward has had fewer changes than the earlier versions which had some fairly big changes from one major version to the next. So something talking about Rails 3 is often similar to Rails 4 (or maybe 5).
You can use a generator to create the basic file as outlined in that article. We would be creating the migration within the stash/stash_engine directory. This will put a file inside that engine for the migration.
The way we are using migrations is a bit complicated because it is set up so the main application will take care of the migrations for all engine sub-components. This means to run the migration itself you'd run "rake db:migrate" from the main dryad application.
Running the migration there will not only modify whatever development database you're using, but it will change the schema.rb in the dryad application.
Because of that you'll need to add parallel changes to both the stash repo and the dryad repo.
When we pull back citations from the crossref eventdata API we are excluding certain citation types.
It looks as though the DataCite folks have added one more that we don't have. We would add has_part to our list of exclusions.
Error like below:
Cloning https://github.com/CDLUC3/stash:
+git clone https://github.com/CDLUC3/stash
Cloning into 'stash'...
remote: Counting objects: 38426, done.
remote: Compressing objects: 100% (294/294), done.
remote: Total 38426 (delta 215), reused 256 (delta 113), pack-reused 38003
Receiving objects: 100% (38426/38426), 27.25 MiB | 14.60 MiB/s, done.
Resolving deltas: 100% (23869/23869), done.
+echo 'Checking out stash branch doc-update'
Checking out stash branch doc-update
+cd stash
+git checkout doc-update
error: pathspec 'doc-update' did not match any file(s) known to git.
Detect Dash EZID DOIs and use CDL EZID credentials to update
When this branch is created, change Capistrano deploys to point toward new server names so we don't accidentally wipe out existing dev/stage/demo/production until we are ready.
Following the instructions at https://github.com/CDLUC3/dashv2/blob/development/documentation/dash2_install.md and https://github.com/CDLUC3/dashv2, set up a vagrant/ansible build that spins up and provisions this machine on an EC2 instance.
We need to evaluate how these will change, specifically these issues/questions.
Dryad-classic DOIs currently go to URLs like https://datadryad.org/resource/doi:10.5061/dryad.9g . Our normally set URLs are in the format /stash/dataset/<doi>
Problem description, and links to relevant helpdesk tickets:
We need to update the target URLs for non-Dash, Dryad-classic DOIs.
Describe the solution you'd like:
In a rake task, but some of the code may be in other places, also.
Oh, some other random notes about checking targets manually by hand for validation. Looks like urls like https://doi.org/10.5061/dryad.9gf10?action=showurls will show the target urls easily without special headers.
More info than I ever wanted about DOIs at https://www.doi.org/doi_handbook/TOC.html .
And handy api for viewing like https://api.datacite.org/dois/10.5061/dryad.9gf10 . See docs at https://support.datacite.org/docs/api-get-doi .
This was filling in an email address if it was exposed from orcid into the user record, but now it no longer seems to work.
https://github.com/CDL-Dryad/dryad-product-roadmap/wiki/Migration
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
Django
The Web framework for perfectionists with deadlines.
Laravel
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
Microsoft
Open source projects and samples from Microsoft.
Google
Google ❤️ Open Source for everyone.
Alibaba
Alibaba Open Source for everyone
D3
Data-Driven Documents codes.
Tencent
China tencent open source team.