Comments (10)
I could take a shot at this. One point on which I am a bit confused is that when should the astroquery.utils.progressbar.chunk_read()
be preferred over the generic astropy.utils.data.get_readable_fileobj()
?
from astroquery.
Unfortunately, that's not entirely clear right now. I think we should rebuild astroquery.utils.progressbar
to use requests, then it will be preferred over get_readable_fileobj
from astroquery.
I should clarify: I think we should prefer astroquery.utils.get_file_from_url
over aud.get_readable_fileobj
in all astroquery tools, but astroquery.utils.download_file
should probably use get_readable_fileobj
internally for now, with a goal to shift to a requests
-based cached download tool at a later date.
from astroquery.
get_file_from_url
should be relatively easy: I think it should be a thin wrapper around get_readable_fileobj
, ideally with an option to save the file to disk somewhere (but adding this feature would require changes upstream in get_readable_fileobj
so that the tempfile that is created is not deleted).
from astroquery.
@jdnc - please do take a shot at this! I'm not sure we'll understand all of the possible issues until there's some code in place.
from astroquery.
Right I am working on this right now - i.e. a generic get_file_from_url
. Should we leave it to the caller to decide whether the returned object is a fitsfile, votable, etc and parse/validate it in the caller (and not here)? This is because for instance in the ned.core
, an additional workaround needs to be used before the votable can be parsed correctly by astropy.io.votable
. Also what about services like ukidss in which we can retrieve the url via an authenticated session request only?.
from astroquery.
We'll make separate functions to parse downloaded data into table/fits form, so individual functions can work around it. As for authentication, we'll need to look at that more closely... maybe the current solution is best, but perhaps there's a better way. Don't worry about that case for now.
from astroquery.
so I have had a first shot at righting this function. It can be found here https://gist.github.com/jdnc/1b3b41657766a5ba0cc1. Is this what was required?
from astroquery.
Yes, pretty much, though line 88 should probably be unindented. I think it may also be wise to have some permissions checking - perhaps wrap the whole "save" block in a try/except so that if writing to disk fails, the data is still returned and accessible
from astroquery.
oh! yes of course - so updated the gist anyways
from astroquery.
Related Issues (20)
- Switch to dict instead of ordered dict in Vizier find_catalogs HOT 3
- Splatalogue.query_lines() seems to be broken? HOT 2
- Slow Simbad.query_objects & IRSA.query_region searches HOT 6
- Use plot directive to generate figures
- modulate verbosity of astroquery.mast.Observations.download_products() HOT 1
- Add support for upload tables for queries in ALMA
- BUG: CI -- no space left on device HOT 2
- ESA Herschel download_data connection error HOT 2
- MAINT: retire/rename exceptions.TimeoutError in favour of requests
- TST: esasky tests take too long time and donwload too much data HOT 8
- BUG: alma test downloads 850+Mb of data HOT 8
- How to download Level 1 ancillary (*cbcd.fits and *cbunc.fits) from IRSA?
- Vizier.query_region() returns incorrect index for some catalog HOT 3
- CI: rerun failing tests at the end
- Bug in rows limitation in module vizier HOT 2
- Unexpected behavior when passing `obj_names` to `SDSS.query_region()`
- Handling No astronomical object found response HOT 5
- IRSA: adding SSA method HOT 1
- Astroquery.gaia API Error: RA cannot be converted to hour or hourangle. HOT 2
- MAST cloud dataset doesn't show level 3 data products
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from astroquery.