download with datedown about smap_io

[sebhahn]

Would you have to copy functionality from datedown?

[wpreimes]

You mean if we don't use datedown? Basically the parts that create dates between--startand --end and replace the datetimes in the URLs (but thats mostly what datedown does, as far as I see it) we would have to copy (or import) and then call wget outside of datedown.

[wpreimes]

Also eg parallelization is in datedown

[sebhahn]

And do you think an extension of datedown makes sense, or is this download too special to be implemented in datedown?

[wpreimes]

I haven't worked enough with wget to assess this ;) But maybe it would make sense to add an option to datedown to force some parameters that are not explicitly used in the download yet (like -e in our case), but I don't know how restrictive datedown should be. As far as I have read now, robots=off may go around restrictions that the server provider has set (?) (which is bad, and I would not use it if it wasn't written in the official guide)

[wpreimes]

@sebhahn: ok I looked at it again, I think I can just implement it without using wget from datedown and defining my own download function (for smap only) ;) but we could still update datedown if it makes sense for the future...

[sebhahn]

Ok create a PR and we can have a look at it

[wpreimes]

Download module is part of the master now