wpgp / wopr Goto Github PK

@bondarenkom The NGA v1.2 requests are quite a bit slower because v1.2 has 10,000 posterior samples compared to NGA v1.1 which has 1,000 samples. A user may want to have a thinned sample of the posteriors to speed up the calculation. (the cell id switch may help this issue too)

We might want to consider adding a "thin" argument that can be 0 to 1. If thin=1, then all 10,000 posterior samples are returned. If thin=0.5, then only 5,000 posterior samples are returned. Also, it would need to be thinned in a particular way. For example, if thin=0.5, we would want to keep every other sample (e.g. rather than the first 5,000 samples).

How hard would this be?

We need an attribute 'filesize' in the API return for each file from http://wopr.worldpop.org/api/v1.0/data

I added code to downloadData() that will allow users to set a threshold when downloading files from WOPR so they can skip large files (default=50MB)

We need to prevent non-public data from being accessed via API. Should we block these requests completely or should we make a key to provide restricted access to all files for a given non-public release (e.g. /COD/Population/v1.0/*)?

I get an error when I submit a point request to dev_server/v1/grid3/sample .

Here is the request:
$iso3
[1] "NGA"
$ver
[1] 1.2
$lat
[1] 6.22337
$lon
[1] 4.771174
$key
[1] "..."

Here is the error when I try to retrieve the result from dev_server/v1/tasks:
$status
[1] "finished"
$error
[1] TRUE
$error_message
[1] "No User Description for this type of error: ValueError"
$taskid
[1] "d59007a0-e6e6-525d-a120-a6a905c115a3"
$startTime
[1] "2019-11-19 19:00:20.811556"
$endTime
[1] "2019-11-19 19:00:21.044907"
$executionTime
[1] 1

An empty query to:
https://api.worldpop.org/v1/wopr/pointtotal

Returns this result:
{
"error": true,
"message": "Required parameters [ iso3], [ ver], [ lat], [ lon] are missing or empty",
"example_api_url": "api.worldpop.org/v1/services/sample?dataset={dataset_name}&year={year}&lat={lat}&lon={lon}"
}

The "example_api_url" is formatted for the WorldPop Global endpoint, not the WOPR endpoint.

We need to sync the following folder on the WorldPop drive to the gridFree server:
Z:\Projects\WP517763_GRID3_DataRelease\DataFinal\*

This will ensure that the data catalogue is kept up to date for API download requests and that the correct version number is assigned to SQL databases being queried by API requests for specific locations.

Currently API requests to the following endpoints are returning results from NGA v1.1 when I request results from NGA v1.2:
https://api.worldpop.org/v1/grid3/stats
http://10.19.100.66/v1/grid3/stats
http://10.19.100.66/v1/grid3/popag

FYI - There is no SQL database for v1.2 yet. I am working on that now.

We don't make model predictions for pixels that are mapped as unsettled so those cellids are not in the SQL database. When I submit a polygon that includes no settled pixels, it returns this error: "No User Description for this type of error: AttributeError".

A more informative message might be: "Polygon unsettled"

Take a look at the last two files in the API return from http://wopr.worldpop.org/api/v1.0/data

They both have the same hash: f11531ca5cdacb40846182bee54778d6.

The md5sum for the file "NGA_population_v1_1_uncertainty.tif" is different than the hash provided in the API return.

I am getting intermittent errors when clilcking the "Submit" button for GIN v1.0:

simpleWarning: No User Description for this type of error: OperationalError

The error only occurs for some locations and it sometimes returns the population estimate, but othertimes it doesn't.

Hi,

I was trying to query only a single age-sex group using a local SQL database. In getPopSql() here: https://github.com/wpgp/wopr/blob/master/R/getPopSql.R#L73 the code collapses to a vector when agesex_select has a length of 1 (e.g. 'm0'). If you add the option drop = FALSE to the data frame that will preserve the 2 dimensional structure that apply expects. Happy to submit a quick pull request with the change, if you'd like.

The population tiles are not showing for any countries in woprVision. I have tested from Brave Browser and Firefox on Ubuntu (20.04) workstation and from my phone (Brave browser).

The dev API now includes the functionality to specify the column from the SQLite database ("Nhat" table) that you would like to query ("Pop" is default). We can store estimated parameters other than total population in these additional columns (vector of posterior predictions or single numbers). We have immediate applications in SSD to add counts of people displaced from/to each pixel, and also for GHA to add counts of housing units.

End-user should be able to upload shapefile for getting area-wide estimates

I am trying to get population estimates for areas within Guinea.

And I get an error message back, plus a couple of warnings.

The warnings either refer to an unknown column 'geometry' or 'geom', depending on whether I've renamed geometry to geom or not.

I presume I'm just doing something wrong.

G <- woprize(features = focidg, country = 'GIN', version = '1.0', confidence = 0.95)
Submitting 4 feature(s) to:
  https://api.worldpop.org/v1/wopr/polytotal
Checking status of 36 tasks:
  Error in results[[j]] : subscript out of bounds
In addition: Warning messages:
1: Unknown or uninitialised column: `geometry`.
2: Unknown or uninitialised column: `geometry`.

Hi all.

I'm having a problem with the wopr::getPop() function. It throws an error about the geometries class, even though my input is in the class require. Please see the reprex below.

ps. Congrats on such a fantastic project and set of packages!

reprex

library(sf)
library(geobr)
library(wopr)

# download area of interest
my_area <- geobr::read_state(code_state = 'DF')

class( st_geometry(my_area) )
> [1] "sfc_MULTIPOLYGON" "sfc" 

# get population estimate
pop <- wopr::getPop(feature = my_area,
                    country = 'BRA')

> Error in endpoint(features = feature, agesex = length(agesex_select) <  : 
                      Input feature geometries must be of class "sfc_POLYGON", "sfc_MULTIPOLYGON", "sfc_POINT", or "sfc_MULTIPOINT"

Let's try to make cellid an option attribute to return in the API requests to the /polytotal and /polyagesex endpoints. This could really speed up transfer time for large requests.

It would be great to share woprVision to external reviewers of newly produced gridded estimates.
@doug-leasure how feasible it is would you reckon?

The title at wopr.worldpop.org should be "WorldPop Open Population Repository" rather than "WorldPop Open Data Repository"

WOPR not WODR :)

Hey @edarin ,
Can you run this code to test the API speed from Lagos this week?

@bondarenkom When I try to use wopr::getPop() to query the COD database using dev_server/v1/wopr/polytotal , I get this error:

$status
[1] "finished"

$error
[1] TRUE

$error_message
[1] "Internal File Error."

$taskid
[1] "1fef3886-3817-550c-991d-11587a62558c"

$startTime
[1] "2019-12-05 21:52:34.275152"

$endTime
[1] "2019-12-05 21:52:34.523026"

$executionTime
[1] 0

Any ideas? Can you tell me what "Internal File Error" means so maybe it will help me figure out the issue on my end?

I am using the same geojson that works with this link:

http://10.19.100.66/v1/wopr/polytotal?iso3=COD&ver=1.0&&geojson={%22type%22:%22FeatureCollection%22,%22features%22:[{%22type%22:%22Feature%22,%22properties%22:{},%22geometry%22:{%22type%22:%22Polygon%22,%22coordinates%22:[[[16.94091796875,-4.127285323245357],[17.5341796875,-4.127285323245357],[17.5341796875,-3.5572827265412794],[16.94091796875,-3.5572827265412794],[16.94091796875,-4.127285323245357]]]}}]}

When a polygon is queried that contains no settled area, the API currently returns:
status=finished, error=true, error_message=No User Description for this type of error: ValueError.

It needs to return:
status=finished, error=true, error_message=No settlements mapped within this area.

I may also be beneficial to add an attribute "error_code" so that the font-end doesn't need to parse the actual error_message to know how to process different results.

Hi,
I'm trying to install wopr, but this fails because it takes a dependency on geojsonlint, which has been retired from CRAN.
Would it be possible to migrate the particular functionality to a different package?
Thank you.

The catalogue lists only 51 countries for which data is available. The population dataset is available only for eight countries. Can this be updated, please? Could you please help me out if I'm missing something?

I know other datasets are available (for example, here's a list of datasets for India).

> catalouge = getCatalogue() %>% as_tibble()

> catalouge %>% filter(category == "Population") %>% distinct(country)
# A tibble: 8 x 1
  country
  <chr>  
1 BFA    
2 COD    
3 GHA    
4 MOZ    
5 NGA    
6 SLE    
7 SSD    
8 ZMB


> catalouge %>% distinct(country)
# A tibble: 51 x 1
   country
   <chr>  
 1 AGO    
 2 BDI    
 3 BEN    
 4 BFA    
 5 BWA    
 6 CAF    
 7 CIV    
 8 CMR    
 9 COD    
10 COG    
# … with 41 more rows