GithubHelp home page GithubHelp logo

Comments (2)

cboettig avatar cboettig commented on August 23, 2024

Perhaps not much of a concern for us at this time, but this does assume that the data file is committed to git, which is often not ideal and will fail for data > 50Mb.

I've been exploring various ways around this (e.g. see unconf issue ropensci/unconf18#51). My current strategy is to take a clue from Rich FitzJohn and upload the data as assets attached to a release. piggyback will let you do something like:

library(piggyback)
pb_upload("user/repo", tag= "data", file = "mydata.csv.gz")
url <- pb_download_url("user/repo", tag= "data", file = "mydata.csv.gz")

to construct a download url for the asset.

This should work for any individual data files up to 2GB in size. I know this mid-size range of > 50mb but < 2 GB isn't huge, so may not be particularly useful for most people, but it is an easy way to avoid cluttering up a git repo.

Of course ideally the data would eventually be uploaded to a DOI-providing repository and contentURL would be amended to that anyway.

from dataspice.

amoeba avatar amoeba commented on August 23, 2024

Whoa, piggyback is cool. This looks like a nice way to get a file shareable fast considering the user already has a repo on GitHub. Does Zenono archive release assets like the ones piggyback creates?

Part of me likes the idea of scoping this package to "data checked into git" but that might be just to simplify things for me rather than a user.

Stepping back, I had thought about how we'd support users making use of non-local files in their scripts. There's nothing in our metadata generation process that prohibits a user from filling in more rows in the access.csv but it'd be nice to automate this. Can you think of any other patterns we could leverage to automatically fill in rows in access.csv (and attributes.csv too for that matter) when the user wants to document more than files checked in under ./data?

from dataspice.

Related Issues (20)

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.