Comments (20)
Reprex:
library(tidyverse)
library(cyclestreets)
# devtools::load_all()
# od = readRDS("~/nptscot/npt/inputdata/od_commute_jittered.Rds")
od_raw = pct::get_od()
#> No region provided. Returning national OD data.
#> Rows: 2402201 Columns: 14
#> ── Column specification ────────────────────────────────────────────────────────
#> Delimiter: ","
#> chr (2): Area of residence, Area of workplace
#> dbl (12): All categories: Method of travel to work, Work mainly at or from h...
#>
#> ℹ Use `spec()` to retrieve the full column specification for this data.
#> ℹ Specify the column types or set `show_col_types = FALSE` to quiet this message.
#> Rows: 7201 Columns: 6
#> ── Column specification ────────────────────────────────────────────────────────
#> Delimiter: ","
#> chr (2): MSOA11CD, MSOA11NM
#> dbl (4): BNGEAST, BNGNORTH, LONGITUDE, LATITUDE
#>
#> ℹ Use `spec()` to retrieve the full column specification for this data.
#> ℹ Specify the column types or set `show_col_types = FALSE` to quiet this message.
lsoas = pct::get_pct(layer = "z", national = TRUE, geography = "msoa")
#> Loading required package: sp
#> Warning in CPL_crs_from_input(x): GDAL Message 1: +init=epsg:XXXX syntax is
#> deprecated. It might return a CRS with a non-EPSG compliant axis order.
od = od_raw %>%
slice(seq(20000))
summary(od$geo_code1 %in% lsoas)
#> Mode FALSE
#> logical 20000
od = od::od_to_sf(x = od, z = lsoas)
#> 0 origins with no match in zone ids
#> 184 destinations with no match in zone ids
#> points not in od data removed.
nrow(od) # 19k
#> [1] 19816
od_100 = od %>%
slice(seq(100))
od_10k = od %>%
slice(seq(10000))
od_15k = od %>%
slice(seq(15000))
batch(desire_lines = od_100, wait = FALSE, silent = FALSE, username = "robinlovelace")
#> POSTing the request to create and start the job
#> Posting to: https://api.cyclestreets.net/v2/batchroutes.createjob?key=xxx
#> Job id: 3886
#> Sending data, wait...
#> Routing not complete
#> Routing job sent, check back in around 1 minutes if you've just sent this
#> Check at www.cyclestreets.net/journey/batch/ for route id: 3886
#> [1] 3886
batch(desire_lines = od_10k, wait = FALSE, silent = FALSE, username = "robinlovelace")
#> POSTing the request to create and start the job
#> Posting to: https://api.cyclestreets.net/v2/batchroutes.createjob?key=xxx
#> Job id: 3887
#> Sending data, wait...
#> Routing not complete
#> Routing job sent, check back in around 6 minutes if you've just sent this
#> Check at www.cyclestreets.net/journey/batch/ for route id: 3887
#> [1] 3887
batch(desire_lines = od_15k, wait = FALSE, silent = FALSE, username = "robinlovelace")
#> POSTing the request to create and start the job
#> Posting to: https://api.cyclestreets.net/v2/batchroutes.createjob?key=xxx
#> Job id:
#> Error in batch(desire_lines = od_15k, wait = FALSE, silent = FALSE, username = "robinlovelace"): Check your credentials, try again, and maybe contact CycleStreets
Created on 2023-02-25 with reprex v2.0.2
from cyclestreets-r.
It seems that the POST request silently fails:
res = httr::POST(url = batch_url, body = body, httr::timeout(60))
from cyclestreets-r.
Could you run this and let us know an exact time it was run so we can trace this in the logs?
I suspect the API is hitting a POST size limit, which is easy to fix. The original batch API was designed for smaller amounts of data, but since the addition of the A-B mode I guess this will probably be hitting a limit.
If it's possible to supply a data file of what is posted that would help.
PS I recommend adding error handling logging in your code so that res
tells you back what HTTP status code or error message you are getting.
from cyclestreets-r.
Sure. Here is the error I'm seeing, seems it's CycleStreets side:
"error": "The number of route calculations that would be required (112,492,500) is...
from cyclestreets-r.
Full details in debug mode. I'll aim to send a file with the body of the request sent:
Browse[2]> res = httr::POST(url = batch_url, body = body, httr::timeout(60))
Browse[2]> res
Response [https://api.cyclestreets.net/v2/batchroutes.createjob?key=xxx]
Date: 2023-02-25 14:56
Status: 200
Content-Type: application/json; charset=UTF-8
Size: 241 B
{
"error": "The number of route calculations that would be required (112,492,500) is...
from cyclestreets-r.
Heads-up @mvl22 I have emailed you a link to the JSON uploaded. It seems that the checking mechanism is calculating the number of route calculations incorrectly on your side.
from cyclestreets-r.
Ah, it thinks you are sending 112 million calculations, which looks to me like 15k squared in one direction only...
You are intending 15k pre-defined lines, yes?
from cyclestreets-r.
Yes, seems it does not realise that these are OD pairs, not points for which every combo is needed. Should be a quick fix.
from cyclestreets-r.
You are intending 15k pre-defined lines, yes?
Correct. Same data structure and POST request works for 10k lines as shown in the reproducible example above.
from cyclestreets-r.
In the meantime, I'll try to pull out and print the error message if there is one. Good shout on that Martin, would have made the root cause of this quicker to identify.
from cyclestreets-r.
Could you also e-mail the 10k lines version?
from cyclestreets-r.
Sure.
from cyclestreets-r.
Thanks.
Sorry you've ran into this problem. We did some server upgrades last week. It's possible that a deployment default that we were silently relying on has changed.
from cyclestreets-r.
See the GeoJSONs here: https://github.com/cyclestreets/cyclestreets-r/releases/tag/v0.6.0
from cyclestreets-r.
See the GeoJSONs here: https://github.com/cyclestreets/cyclestreets-r/releases/tag/v0.6.0
The 15k example at least seems to be very long distance lines, e.g. from London to Newcastle. Just checking these are the right files? That seems to be different to the gist you sent.
Please ensure that the specific dataset you give is actually failing. Otherwise I don't have a reliable reproduce-case.
from cyclestreets-r.
I've confirmed we are not seeing POST size maxouts in the logs, i.e. the sent data is being correctly received, as indeed the error you are getting implies.
So there is something about the data or the processing code that is making the system this is a set of points rather than linestrings and therefore triggering combinations mode.
from cyclestreets-r.
See the GeoJSONs here: https://github.com/cyclestreets/cyclestreets-r/releases/tag/v0.6.0
The 15k example at least seems to be very long distance lines, e.g. from London to Newcastle. Just checking these are the right files? That seems to be different to the gist you sent.
Please ensure that the specific dataset you give is actually failing. Otherwise I don't have a reliable reproduce-case.
That was just an example dataset to show the error. Happy to provide a more realistic one.
from cyclestreets-r.
See updated GeoJSON here: https://github.com/cyclestreets/cyclestreets-r/releases/download/v0.6.0/od_15k.geojson
from cyclestreets-r.
This should now be working.
I found a bug, whereby the number of calculations in LineString mode was not done correctly. The 10k file was doing the incorrect calculation but was not triggering the calculation limit. This is now fixed.
from cyclestreets-r.
Great to hear @mvl22, thanks for quickly identifying and fixing this issue your side.
from cyclestreets-r.
Related Issues (20)
- Cannot upload GeoJSON files with more than ~30k lines HOT 26
- Create batch_multi function to send many batches before getting data
- Cannot delete jobs that are 'finalising' HOT 2
- Missing sf:: HOT 1
- Batch function misaligns outputs'
- Left join fails with no id column
- Replace jsonlite with Rcppsimdjson HOT 2
- Reduce waiting time in batch() HOT 2
- Add progress bars to batch functions
- Speed-up json2sf_cs HOT 9
- Test single segment route HOT 1
- Extra column name associated with short routes HOT 3
- Reduce size of GeoJSON files uploaded for batch routing HOT 1
- Repeated message on batch routing HOT 1
- Refactor
- Export `batch_deletejob()`
- Reduce size of GeoJSON objects sent for routing HOT 1
- Quietness and other variables have character type with journey()
- Batch routes: Implement new successThreshold parameter
- Parameter emailOnCompletion should not e-mail a junk address by default
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from cyclestreets-r.