ckanapi dump datasets --all doesn't work for catalog.data.gov about ckanapi

[wardi]

It's possible (and more efficient) to dump all datasets with package_search, but you have to be careful about duplicate or missing records that may appear in the results if datasets are being modified while you're dumping them.

Are you interested in helping to implement this? It shouldn't be too difficult.

[ekzhu]

I looked at the code in dump.py. It seems that it doesn't support making multiple paginated requests. catalog.data.gov has 193483 metadata so a single ckan.call_action with package_search is probably not going to return the complete result.

http://catalog.data.gov/api/action/package_search?rows=200&start=0

Does the current code support paginated requests somewhere?

[wardi]

No, using package_search is very different than the *_list, *_show that dump.py uses, so it won't be much help. getting dataset results from package_search should be very fast though, so you don't need dump.py's multiple-worker support. How about just writing it as a separate sub-command?

Also note that older versions of CKAN don't return the same data from package_search that they do from package_show, so it's useful to keep the current dump datasets implementation around for a little while.

[ekzhu]

Okay. Since it's convenient I will ask another question here. How do I know which version of the API am I calling. Is this

http://catalog.data.gov/api/action/package_search?rows=200&start=0

the newest API or do I have to explicitly add /3/:

http://catalog.data.gov/api/3/action/package_search?rows=200&start=0

Using /3/ may not work for some endpoints which don't have version 3.

[wardi]

in ckanapi so far I've omitted the api version when calling ckan because I've always wanted the latest version of the API.

[wardi]

note that we now have ckanapi search datasets for getting packages with package_search, closing this issue