ipeagit / flightsbr Goto Github PK

View Code? Open in Web Editor NEW

40.0 40.0 5.0 11.5 MB

R Package to Download Flight and Airport Data from Brazil

Home Page: https://ipeagit.github.io/flightsbr/

License: Other

R 100.00%

aviation-data brazil data r rstats rstats-package

flightsbr's People

Contributors

Stargazers

Watchers

Forkers

mathiasfls angelorodrigues gutovellozo wandrys-dev zsz-hst

flightsbr's Issues

instability in ANAC data links

ANAC data links seem to get unstable every now and then. This causes some read_ functions to fail. When this happens, the user may call the function again it will likely work normally. This is a silly issue, but can be a burder for test checks.

I'm changing the read_ functions to try downloading the data a 2nd time if the 1st attempt didn't work

update v0.2.1 on CRAN

side effects of `month` and `year` on global envirornment

currently, both functions read_flights() and read_airport_movements() have a side effect, creating the objects month and year on the global environment. This can create problems if the user already has objects with these names loaded on their global environment.

new vignette read_airports()

Error when read_airfares(date = 202203, domestic = T)

f <- flightsbr::read_airfares(date = 202203, domestic = T)

> Error in check_date(date = date, all_dates) : 
> The data is currently only available for dates between 200201 and 202210.`

fail gracefully in case of problems with internet connection

single progress bar when downloading multiple months

new function to check the date of the latest flight data available

Function read_airfares() not working

The function read_airfares() currently not working because ANAC has changed the url address where the data files are stored.

In the recent past, fares data on domestic and international flights were stored in:

The data is now only available other webinterfaces which work with point and click:

domestic: https://sas.anac.gov.br/sas/downloads/view/frmDownload.aspx?tema=14
international: https://sas.anac.gov.br/sas/downloads/view/frmDownload.aspx?tema=17

However, haven't been able to extract from the new site the url to the data files:

base_url = 'https://sas.anac.gov.br/sas/downloads/view/frmDownload.aspx?tema=14'

h <- try(rvest::read_html(base_url), silent = TRUE)

# filter elements of basic data
elements <- rvest::html_elements(h, "a")

any help here would be much appreciated.

allow for read_airports( type = 'all')

error in read_airport_movements() : Error in data.table::rbindlist(dt_list) : Class attribute

Downloading the airport movment data for the year of 2019 is throwing the following error:

> m2019 <- read_airport_movements(date = 2019)
  |==================================================| 100%
Error in data.table::rbindlist(dt_list) : 
  Class attribute on column 12 of item 11 does not match with column 12 of item 1.

Download aircrafts data from different dates

Add date parameter to read_aircrafts()

Function to download airfares

It would be good to create a read_airfares() function to download microdata of airfares.

Problem: the problem is that these data sets need to be downloaded though this system here https://sistemas.anac.gov.br/sas/downloads/. I haven't found a website or ftp link where the data sets are stored.

Error when using read_flights

Hello,
I was trying to download the data from 2015, with type being either 'basica' or 'combinada' and for both types I get error as well as waning messages.
For type = 'basica':

For type = 'combinada':

In this last case, the data ends up loading, but the encoding appears like this:

Similar things happened for year = 2001, for example.
I am working on R version 4.2.3 on windows.
Thank you so much for your attention, as well as for the package!

check input of dates

Error in rbindlist of read_airfares()

fares <- read_airfares(date = 2022)

>   |==================================================| 100%
> Error in data.table::rbindlist(dt_list) : 
>   Item 4 has 8 columns, inconsistent with item 1 which has 7 columns. To fill missing columns use fill=TRUE.
>

Input vector of dates to read_flights()

suggestion from @baarthur mentioned in issue #28

Erro ao utilizar `read_flights()`

Utilizando a versão de desenvolvimento.

library(flightsbr)

df_2015 <- read_flights(date=2015, showProgress = FALSE)
#> Warning in (if (.Platform$OS.type == "unix") system
#> else shell)(paste0("(", : execução de '(unzip -p
#> C:\Users\Bruno\AppData\Local\Temp\Rtmpq25KdB/basica2015-01.zip) >
#> C:\Users\Bruno\AppData\Local\Temp\Rtmpq25KdB\file3538c3c6362' falhou com código
#> de erro 1
#> Warning in data.table::fread(cmd = temp_local_file_zip, select = select, :
#> File 'C:\Users\Bruno\AppData\Local\Temp\Rtmpq25KdB\file3538c3c6362' has size 0.
#> Returning a NULL data.table.
#> Warning in (if (.Platform$OS.type == "unix") system
#> else shell)(paste0("(", : execução de '(unzip -p
#> C:\Users\Bruno\AppData\Local\Temp\Rtmpq25KdB/basica2015-02.zip) >
#> C:\Users\Bruno\AppData\Local\Temp\Rtmpq25KdB\file353813ff3a49' falhou com código
#> de erro 1
#> Warning in data.table::fread(cmd = temp_local_file_zip, select = select, : File
#> 'C:\Users\Bruno\AppData\Local\Temp\Rtmpq25KdB\file353813ff3a49' has size 0.
#> Returning a NULL data.table.
#> Warning in (if (.Platform$OS.type == "unix") system
#> else shell)(paste0("(", : execução de '(unzip -p
#> C:\Users\Bruno\AppData\Local\Temp\Rtmpq25KdB/basica2015-03.zip) >
#> C:\Users\Bruno\AppData\Local\Temp\Rtmpq25KdB\file353810b6236' falhou com código
#> de erro 1
#> Warning in data.table::fread(cmd = temp_local_file_zip, select = select, :
#> File 'C:\Users\Bruno\AppData\Local\Temp\Rtmpq25KdB\file353810b6236' has size 0.
#> Returning a NULL data.table.
#> Warning in (if (.Platform$OS.type == "unix") system
#> else shell)(paste0("(", : execução de '(unzip -p
#> C:\Users\Bruno\AppData\Local\Temp\Rtmpq25KdB/basica2015-04.zip) >
#> C:\Users\Bruno\AppData\Local\Temp\Rtmpq25KdB\file35386d603838' falhou com código
#> de erro 1
#> Warning in data.table::fread(cmd = temp_local_file_zip, select = select, : File
#> 'C:\Users\Bruno\AppData\Local\Temp\Rtmpq25KdB\file35386d603838' has size 0.
#> Returning a NULL data.table.
#> Warning in (if (.Platform$OS.type == "unix") system
#> else shell)(paste0("(", : execução de '(unzip -p
#> C:\Users\Bruno\AppData\Local\Temp\Rtmpq25KdB/basica2015-05.zip) >
#> C:\Users\Bruno\AppData\Local\Temp\Rtmpq25KdB\file35386be21cd4' falhou com código
#> de erro 1
#> Warning in data.table::fread(cmd = temp_local_file_zip, select = select, : File
#> 'C:\Users\Bruno\AppData\Local\Temp\Rtmpq25KdB\file35386be21cd4' has size 0.
#> Returning a NULL data.table.
#> Warning in (if (.Platform$OS.type == "unix") system
#> else shell)(paste0("(", : execução de '(unzip -p
#> C:\Users\Bruno\AppData\Local\Temp\Rtmpq25KdB/basica2015-06.zip) >
#> C:\Users\Bruno\AppData\Local\Temp\Rtmpq25KdB\file353873e947ea' falhou com código
#> de erro 1
#> Warning in data.table::fread(cmd = temp_local_file_zip, select = select, : File
#> 'C:\Users\Bruno\AppData\Local\Temp\Rtmpq25KdB\file353873e947ea' has size 0.
#> Returning a NULL data.table.
#> Warning in (if (.Platform$OS.type == "unix") system
#> else shell)(paste0("(", : execução de '(unzip -p
#> C:\Users\Bruno\AppData\Local\Temp\Rtmpq25KdB/basica2015-07.zip) >
#> C:\Users\Bruno\AppData\Local\Temp\Rtmpq25KdB\file353815b61c98' falhou com código
#> de erro 1
#> Warning in data.table::fread(cmd = temp_local_file_zip, select = select, : File
#> 'C:\Users\Bruno\AppData\Local\Temp\Rtmpq25KdB\file353815b61c98' has size 0.
#> Returning a NULL data.table.
#> Warning in (if (.Platform$OS.type == "unix") system
#> else shell)(paste0("(", : execução de '(unzip -p
#> C:\Users\Bruno\AppData\Local\Temp\Rtmpq25KdB/basica2015-08.zip) >
#> C:\Users\Bruno\AppData\Local\Temp\Rtmpq25KdB\file35384867254c' falhou com código
#> de erro 1
#> Warning in data.table::fread(cmd = temp_local_file_zip, select = select, : File
#> 'C:\Users\Bruno\AppData\Local\Temp\Rtmpq25KdB\file35384867254c' has size 0.
#> Returning a NULL data.table.
#> Warning in (if (.Platform$OS.type == "unix") system
#> else shell)(paste0("(", : execução de '(unzip -p
#> C:\Users\Bruno\AppData\Local\Temp\Rtmpq25KdB/basica2015-09.zip) >
#> C:\Users\Bruno\AppData\Local\Temp\Rtmpq25KdB\file35386781616e' falhou com código
#> de erro 1
#> Warning in data.table::fread(cmd = temp_local_file_zip, select = select, : File
#> 'C:\Users\Bruno\AppData\Local\Temp\Rtmpq25KdB\file35386781616e' has size 0.
#> Returning a NULL data.table.
#> Warning in (if (.Platform$OS.type == "unix") system
#> else shell)(paste0("(", : execução de '(unzip -p
#> C:\Users\Bruno\AppData\Local\Temp\Rtmpq25KdB/basica2015-10.zip) >
#> C:\Users\Bruno\AppData\Local\Temp\Rtmpq25KdB\file353851c11942' falhou com código
#> de erro 1
#> Warning in data.table::fread(cmd = temp_local_file_zip, select = select, : File
#> 'C:\Users\Bruno\AppData\Local\Temp\Rtmpq25KdB\file353851c11942' has size 0.
#> Returning a NULL data.table.
#> Warning in (if (.Platform$OS.type == "unix") system
#> else shell)(paste0("(", : execução de '(unzip -p
#> C:\Users\Bruno\AppData\Local\Temp\Rtmpq25KdB/basica2015-11.zip) >
#> C:\Users\Bruno\AppData\Local\Temp\Rtmpq25KdB\file35385b7f31f2' falhou com código
#> de erro 1
#> Warning in data.table::fread(cmd = temp_local_file_zip, select = select, : File
#> 'C:\Users\Bruno\AppData\Local\Temp\Rtmpq25KdB\file35385b7f31f2' has size 0.
#> Returning a NULL data.table.
#> Warning in (if (.Platform$OS.type == "unix") system
#> else shell)(paste0("(", : execução de '(unzip -p
#> C:\Users\Bruno\AppData\Local\Temp\Rtmpq25KdB/basica2015-12.zip) >
#> C:\Users\Bruno\AppData\Local\Temp\Rtmpq25KdB\file3538192cae8' falhou com código
#> de erro 1
#> Warning in data.table::fread(cmd = temp_local_file_zip, select = select, :
#> File 'C:\Users\Bruno\AppData\Local\Temp\Rtmpq25KdB\file3538192cae8' has size 0.
#> Returning a NULL data.table.


sessionInfo()
#> R version 4.1.1 (2021-08-10)
#> Platform: x86_64-w64-mingw32/x64 (64-bit)
#> Running under: Windows 10 x64 (build 19045)
#> 
#> Matrix products: default
#> 
#> locale:
#> [1] LC_COLLATE=Portuguese_Brazil.1252  LC_CTYPE=Portuguese_Brazil.1252   
#> [3] LC_MONETARY=Portuguese_Brazil.1252 LC_NUMERIC=C                      
#> [5] LC_TIME=Portuguese_Brazil.1252    
#> 
#> attached base packages:
#> [1] stats     graphics  grDevices utils     datasets  methods   base     
#> 
#> loaded via a namespace (and not attached):
#>  [1] rstudioapi_0.13   knitr_1.38        magrittr_2.0.3    R.cache_0.15.0   
#>  [5] rlang_1.1.1       fastmap_1.1.0     fansi_1.0.3       stringr_1.5.0    
#>  [9] styler_1.7.0      highr_0.9         tools_4.1.1       xfun_0.30        
#> [13] R.oo_1.24.0       utf8_1.2.2        cli_3.4.1         withr_2.5.0      
#> [17] htmltools_0.5.5   yaml_2.3.5        digest_0.6.29     tibble_3.2.1     
#> [21] lifecycle_1.0.3   purrr_1.0.0.9000  R.utils_2.11.0    vctrs_0.6.2      
#> [25] fs_1.5.2          glue_1.6.2        evaluate_0.15     rmarkdown_2.18.1 
#> [29] reprex_2.0.2      stringi_1.7.6     compiler_4.1.1    pillar_1.9.0     
#> [33] R.methodsS3_1.8.1 pkgconfig_2.0.3

^{Created on 2023-07-03 with reprex v2.0.2}

Parece estar relacionado a este trecho, ou algo mais.

flightsbr/R/utils.R

Lines 442 to 446 in 915cb7e

 # address of zipped file stored locally 

 temp_local_file_zip <- paste0('unzip -p ', temp_local_file) 

 # read zipped file stored locally 

 dt <- data.table::fread( cmd = temp_local_file_zip, select=select, colClasses = 'character', sep = ';')

pkgdown not working properly

Trouble selecting variables

Hi there,

First of all, thanks for the great work on this package! I am not sure if this is a bug or more of a feature request, but here it goes:

When selecting variables in read_flights(), I can't select columns "sg_equipamento_icao" and "ds_matricula", which are available on Anac's data according to the link in the function's manual. I think there may be a bug while fetching those columns... Or they aren't available in the package's dataset?

In addition: it would be great if we could pass a vector of dates in the function!

Here's my code:

df_flights <- read_flights(
  202303, 
  type = "combinada", 
  select = c(
    "id_combinada", 
    "sg_empresa_icao", "nr_voo", "dt_referencia", 
    "sg_icao_origem", "sg_icao_destino", 
    "ds_natureza_etapa", "nr_etapa", "cd_di", "ds_di", 
    "dt_partida_real", "dt_chegada_real", 
    "sg_equipamento_icao", "ds_matricula", # these two throw a warning message
    "nr_passag_pagos", "nr_passag_gratis", 
    "kg_bagagem_livre", "kg_bagagem_excesso", "kg_carga_paga", "kg_carga_gratis", "kg_correio"
  )
) %>% 
  mutate(
    across(
      nr_passag_pagos:kg_correio,
      as.numeric
    )
  )

cache downloaded files in temp dir

read_flights()
read_airport_movements()

ps. The other functions dowload very small data sets, so there would not be a big benefit in changing them to chache the data

download all months in a year

update vignette of read_flights

check code coverage and add shield

tolower invalid multibyte string

add aerodromes

https://www.anac.gov.br/acesso-a-informacao/dados-abertos/areas-de-atuacao/aerodromos

new function to read aiport operations data

test read_flights

read_flights() not working

ANAC has changed the url address of where they store the data.

Automatically detect latest flight data available

ANAC adds new flight data sets to their website almost every month. Ideally, we would like to automatically detect the latest flight data available . There should be a way to list the files on ANAC url using curl or httr. Any help / contributions / suggestions would be much appreciated!

	# address of zipped file stored locally
	temp_local_file_zip <- paste0('unzip -p ', temp_local_file)

	# read zipped file stored locally
	dt <- data.table::fread( cmd = temp_local_file_zip, select=select, colClasses = 'character', sep = ';')

ipeagit / flightsbr Goto Github PK

flightsbr's People

Contributors

Stargazers

Watchers

Forkers

flightsbr's Issues

Recommend Projects

Recommend Topics

Recommend Org

Jobs