CartoDB Import API does not replace old table it adds a new one about farmview

[stlim0730]

How agile!

So, this issue will be about using Sync table API of CartoDB. I labeled it as enhancement.

I found this: Force a synchronization action on a sync table, which means sync now! feature is still possible with a Sync table. In fact, with this feature, Sync table is more advanced use of existing tech than standard table.

Thanks, Adam & Irene.

[calo1]

Irene and I agreed that sync feature was desired. However, we need a place to host the survey data that cartodb can read. Currently Dropbox doesn't work. The shareable link doesn't render the data- in question.

[sohyeonlee]

I actually did some google search and found out that replacing
www.dropbox.com to dl.dropboxusercontent.com will render the actual json. I
think I can go ahead and continue with dropbox. So, I guess I am
implementing the sync feature (specifically sync now from the link Taek
provided) if I have understood it correctly?

On Tue, Jun 28, 2016 at 8:37 AM, calo1 [email protected] wrote:

Irene and I agreed that sync feature was desired. However, we need a place
to host the survey data that cartodb can read. Currently Dropbox doesn't
work. The shareable link doesn't render the data- in question.

—
You are receiving this because you were assigned.
Reply to this email directly, view it on GitHub
#35 (comment),
or mute the thread
https://github.com/notifications/unsubscribe/AFoiB5u4cgYmIbIPPxLTwsHLdV44SOIOks5qQT-1gaJpZM4I_qsg
.

[calo1]

Good work Irene! Test to see if your edit will work with cURL. I thin the goal is to set the sync interval at a long interval: How about every 2 weeks. Then we will need the control of sync now to push important data to cartodb.

If this works we will need to do this:

1. Delete the old survey data from cartodb
2. Re-upload the survey data with the correct table names while initiating the sync feature. (data_point and data_polygon)

That should keep all of the visualizations and table references intact

[sohyeonlee]

So what I think should happen is:

1. convert ona survey data into geojson files and uploaded to dropbox
2. create new data sets in cartodb with API and create a map based on that new data sets (sync_now feature needs import_id which can only be retrieved by uploading data set via curl/API calls I believe)
3. create sync_now (requires at least 15 min in between syncs)

I have tested these using curl and I am converting these to python script.

My question so far is: are we going to connect this to the admin panel or run it separately on terminal? Because each function in my script should know which data set is being synced and such, so I am wondering if I should hard code the information for now or expect those info as parameters. I am not sure if my question is clear enough so let me know!

[stlim0730]

Irene, thanks for bringing up this issue! It's very clear and relevant.

As you already know, Django admin panel is a model-centric interface, but the survey data is not a part of the models for now. It's a really good idea that we model the data flow, rather than the data itself. This will enable admin actions on the panel that invoke the data processing procedures you made (https://docs.djangoproject.com/en/1.9/ref/contrib/admin/actions/).

For now, I'd like to encourage you to parameterize your functions, even though you call those functions with parameters of constant string literals. Also, it'll be nice if you share the general specification of the functions (i.e., what parameters each function will have, what each function does, etc) with us before complete implementation, so that I can get ideas on how to model the abstract object.

[sohyeonlee]

I have pushed a new model with three functions (I tried refactoring a little bit).
def create_sync_table(ona_id) # converts all ona survey data sets into geoJSON and creates new model with response retrieved.
def force_sync(import_id) # uses import_id, which was saved into model by calling create_sync_table function, to make PUT request to cartoDB
def download_ona_data(data_id) # called inside create_sync_table to download ona survey data sets
def upload_dropbox(*args) # uploads files (multiple) to dropbox account, inside farmview folder

I still have some minor issues in the code and might bring up some questions to the meeting!