cuducos / calculadora-do-cidadao Goto Github PK
View Code? Open in Web Editor NEW💵 Tool for Brazilian Reais monetary adjustment/correction
Home Page: https://calculadora-do-cidadao.readthedocs.io
License: GNU General Public License v3.0
💵 Tool for Brazilian Reais monetary adjustment/correction
Home Page: https://calculadora-do-cidadao.readthedocs.io
License: GNU General Public License v3.0
As instruções para rodar os testes, atualmente, passam pela instalação manual de alguns pacotes (tox
, pytest
e mais plugins).
Acredito que utilizando o Poetry a gente consiga simplificar essas instruções (e o processo todo) com:
A suíte de testes roda com diversas versões do Python via
tox
:$ poetry run tox
Se quiser rodar os testes mais rapidamente apenas para a versão em uso, é possível utilizar o
pytest
:$ poetry run pyetst
Ainda, simplificamos a parte de gerar documentação para apenas:
Para visualizar alterações na documentação, é preciso instalar alguns pacotes e utilizar a o Sphinx:
$ poetry run sphinx-build docs docs/_build
Depois, é só acessar
docs/_build/index.html
.
Vale notar que no #41 já é introduzido o requirements-development.txt
(obrigado, @turicas!) para contornar esse mesmo problema, mas talvez o Poetry seja uma solução mais robusta.
Implementar adaptador para o IPCA-E.
Comecei com um rascunhão, mas uma hora a API vai ficar mais clara e podemos escrever testes!
Antes de me dedicar aos outros índices vou atualizar o código inserindo docstrings
para ficar no padrão da PEP 257. Tudo bem?
Creio que essa verificação pode ser feita em dois momentos:
self.ÍNDICES
)é sempre bom colocar um arquivo "requirements.txt" com as bibliotecas que precisam baixar ou mostrar as que importou e/ou versão das mesmas
Exemplo com pip install rows[xls]
para IPCA:
from urllib.request import urlretrieve
from tempfile import TemporaryDirectory
from zipfile import ZipFile
from rows.plugins.xls import import_from_xls
MONTHS = ("JAN", "FEV", "MAR", "ABR", "MAI", "JUN", "JUL", "AGO", "SET", "OUT", "NOV", "DEZ")
URL = "ftp://ftp.ibge.gov.br/Precos_Indices_de_Precos_ao_Consumidor/IPCA/Serie_Historica/ipca_SerieHist.zip"
def ipca():
# download & unarchive & load
filename, _ = urlretrieve(URL)
with ZipFile(filename) as archive, TemporaryDirectory() as tmp:
excel_file, *_ = archive.infolist()
path = archive.extract(excel_file, path=tmp)
table = import_from_xls(path, end_column=2)
# read & deserialize
last_year = None
for year, month, value in table:
if month not in MONTHS:
continue
year = int(year or last_year)
month = MONTHS.index(month) + 1
value = float(value)
yield year, month, value
last_year = year
for row in ipca():
print(row)
Saída:
(1994, 1, 141.31)
(1994, 2, 198.22)
(1994, 3, 282.96)
(1994, 4, 403.73)
(1994, 5, 581.49)
(1994, 6, 857.29)
(1994, 7, 915.93)
(1994, 8, 932.97)
(1994, 9, 947.24)
(1994, 10, 972.06)
(1994, 11, 999.37)
(1994, 12, 1016.46)
(1995, 1, 1033.74)
(1995, 2, 1044.28)
(1995, 3, 1060.47)
(1995, 4, 1086.24)
(1995, 5, 1115.24)
(1995, 6, 1140.44)
(1995, 7, 1167.35)
(1995, 8, 1178.91)
(1995, 9, 1190.58)
(1995, 10, 1207.37)
(1995, 11, 1225.12)
(1995, 12, 1244.23)
(1996, 1, 1260.9)
(1996, 2, 1273.89)
(1996, 3, 1278.35)
(1996, 4, 1294.46)
(1996, 5, 1310.25)
(1996, 6, 1325.84)
(1996, 7, 1340.56)
(1996, 8, 1346.46)
(1996, 9, 1348.48)
(1996, 10, 1352.53)
(1996, 11, 1356.86)
(1996, 12, 1363.24)
(1997, 1, 1379.33)
(1997, 2, 1386.23)
(1997, 3, 1393.3)
(1997, 4, 1405.56)
(1997, 5, 1411.32)
(1997, 6, 1418.94)
(1997, 7, 1422.06)
(1997, 8, 1421.78)
(1997, 9, 1422.63)
(1997, 10, 1425.9)
(1997, 11, 1428.32)
(1997, 12, 1434.46)
(1998, 1, 1444.64)
(1998, 2, 1451.29)
(1998, 3, 1456.22)
(1998, 4, 1459.71)
(1998, 5, 1467.01)
(1998, 6, 1467.3)
(1998, 7, 1465.54)
(1998, 8, 1458.07)
(1998, 9, 1454.86)
(1998, 10, 1455.15)
(1998, 11, 1453.4)
(1998, 12, 1458.2)
(1999, 1, 1468.41)
(1999, 2, 1483.83)
(1999, 3, 1500.15)
(1999, 4, 1508.55)
(1999, 5, 1513.08)
(1999, 6, 1515.95)
(1999, 7, 1532.47)
(1999, 8, 1541.05)
(1999, 9, 1545.83)
(1999, 10, 1564.23)
(1999, 11, 1579.09)
(1999, 12, 1588.56)
(2000, 1, 1598.41)
(2000, 2, 1600.49)
(2000, 3, 1604.01)
(2000, 4, 1610.75)
(2000, 5, 1610.91)
(2000, 6, 1614.62)
(2000, 7, 1640.62)
(2000, 8, 1662.11)
(2000, 9, 1665.93)
(2000, 10, 1668.26)
(2000, 11, 1673.6)
(2000, 12, 1683.47)
(2001, 1, 1693.07)
(2001, 2, 1700.86)
(2001, 3, 1707.32)
(2001, 4, 1717.22)
(2001, 5, 1724.26)
(2001, 6, 1733.23)
(2001, 7, 1756.28)
(2001, 8, 1768.57)
(2001, 9, 1773.52)
(2001, 10, 1788.24)
(2001, 11, 1800.94)
(2001, 12, 1812.65)
(2002, 1, 1822.08)
(2002, 2, 1828.64)
(2002, 3, 1839.61)
(2002, 4, 1854.33)
(2002, 5, 1858.22)
(2002, 6, 1866.02)
(2002, 7, 1888.23)
(2002, 8, 1900.5)
(2002, 9, 1914.18)
(2002, 10, 1939.26)
(2002, 11, 1997.83)
(2002, 12, 2039.78)
(2003, 1, 2085.68)
(2003, 2, 2118.43)
(2003, 3, 2144.49)
(2003, 4, 2165.29)
(2003, 5, 2178.5)
(2003, 6, 2175.23)
(2003, 7, 2179.58)
(2003, 8, 2186.99)
(2003, 9, 2204.05)
(2003, 10, 2210.44)
(2003, 11, 2217.96)
(2003, 12, 2229.49)
(2004, 1, 2246.43)
(2004, 2, 2260.13)
(2004, 3, 2270.75)
(2004, 4, 2279.15)
(2004, 5, 2290.77)
(2004, 6, 2307.03)
(2004, 7, 2328.02)
(2004, 8, 2344.08)
(2004, 9, 2351.82)
(2004, 10, 2362.17)
(2004, 11, 2378.47)
(2004, 12, 2398.92)
(2005, 1, 2412.83)
(2005, 2, 2427.07)
(2005, 3, 2441.87)
(2005, 4, 2463.11)
(2005, 5, 2475.18)
(2005, 6, 2474.68)
(2005, 7, 2480.87)
(2005, 8, 2485.09)
(2005, 9, 2493.79)
(2005, 10, 2512.49)
(2005, 11, 2526.31)
(2005, 12, 2535.4)
(2006, 1, 2550.36)
(2006, 2, 2560.82)
(2006, 3, 2571.83)
(2006, 4, 2577.23)
(2006, 5, 2579.81)
(2006, 6, 2574.39)
(2006, 7, 2579.28)
(2006, 8, 2580.57)
(2006, 9, 2585.99)
(2006, 10, 2594.52)
(2006, 11, 2602.56)
(2006, 12, 2615.05)
(2007, 1, 2626.56)
(2007, 2, 2638.12)
(2007, 3, 2647.88)
(2007, 4, 2654.5)
(2007, 5, 2661.93)
(2007, 6, 2669.38)
(2007, 7, 2675.79)
(2007, 8, 2688.37)
(2007, 9, 2693.21)
(2007, 10, 2701.29)
(2007, 11, 2711.55)
(2007, 12, 2731.62)
(2008, 1, 2746.37)
(2008, 2, 2759.83)
(2008, 3, 2773.08)
(2008, 4, 2788.33)
(2008, 5, 2810.36)
(2008, 6, 2831.16)
(2008, 7, 2846.16)
(2008, 8, 2854.13)
(2008, 9, 2861.55)
(2008, 10, 2874.43)
(2008, 11, 2884.78)
(2008, 12, 2892.86)
(2009, 1, 2906.74)
(2009, 2, 2922.73)
(2009, 3, 2928.57)
(2009, 4, 2942.63)
(2009, 5, 2956.46)
(2009, 6, 2967.1)
(2009, 7, 2974.22)
(2009, 8, 2978.68)
(2009, 9, 2985.83)
(2009, 10, 2994.19)
(2009, 11, 3006.47)
(2009, 12, 3017.59)
(2010, 1, 3040.22)
(2010, 2, 3063.93)
(2010, 3, 3079.86)
(2010, 4, 3097.42)
(2010, 5, 3110.74)
(2010, 6, 3110.74)
(2010, 7, 3111.05)
(2010, 8, 3112.29)
(2010, 9, 3126.29)
(2010, 10, 3149.74)
(2010, 11, 3175.88)
(2010, 12, 3195.89)
(2011, 1, 3222.42)
(2011, 2, 3248.2)
(2011, 3, 3273.86)
(2011, 4, 3299.07)
(2011, 5, 3314.58)
(2011, 6, 3319.55)
(2011, 7, 3324.86)
(2011, 8, 3337.16)
(2011, 9, 3354.85)
(2011, 10, 3369.28)
(2011, 11, 3386.8)
(2011, 12, 3403.73)
(2012, 1, 3422.79)
(2012, 2, 3438.19)
(2012, 3, 3445.41)
(2012, 4, 3467.46)
(2012, 5, 3479.94)
(2012, 6, 3482.72)
(2012, 7, 3497.7)
(2012, 8, 3512.04)
(2012, 9, 3532.06)
(2012, 10, 3552.9)
(2012, 11, 3574.22)
(2012, 12, 3602.46)
(2013, 1, 3633.44)
(2013, 2, 3655.24)
(2013, 3, 3672.42)
(2013, 4, 3692.62)
(2013, 5, 3706.28)
(2013, 6, 3715.92)
(2013, 7, 3717.03)
(2013, 8, 3725.95)
(2013, 9, 3738.99)
(2013, 10, 3760.3)
(2013, 11, 3780.61)
(2013, 12, 3815.39)
(2014, 1, 3836.37)
(2014, 2, 3862.84)
(2014, 3, 3898.38)
(2014, 4, 3924.5)
(2014, 5, 3942.55)
(2014, 6, 3958.32)
(2014, 7, 3958.72)
(2014, 8, 3968.62)
(2014, 9, 3991.24)
(2014, 10, 4008.0)
(2014, 11, 4028.44)
(2014, 12, 4059.86)
(2015, 1, 4110.2)
(2015, 2, 4160.34)
(2015, 3, 4215.26)
(2015, 4, 4245.19)
(2015, 5, 4276.6)
(2015, 6, 4310.39)
(2015, 7, 4337.11)
(2015, 8, 4346.65)
(2015, 9, 4370.12)
(2015, 10, 4405.95)
(2015, 11, 4450.45)
(2015, 12, 4493.17)
(2016, 1, 4550.23)
(2016, 2, 4591.18)
(2016, 3, 4610.92)
(2016, 4, 4639.05)
(2016, 5, 4675.23)
(2016, 6, 4691.59)
(2016, 7, 4715.99)
(2016, 8, 4736.74)
(2016, 9, 4740.53)
(2016, 10, 4752.86)
(2016, 11, 4761.42)
(2016, 12, 4775.7)
(2017, 1, 4793.85)
(2017, 2, 4809.67)
(2017, 3, 4821.69)
(2017, 4, 4828.44)
(2017, 5, 4843.41)
(2017, 6, 4832.27)
(2017, 7, 4843.87)
(2017, 8, 4853.07)
(2017, 9, 4860.83)
(2017, 10, 4881.25)
(2017, 11, 4894.92)
(2017, 12, 4916.46)
(2018, 1, 4930.72)
(2018, 2, 4946.5)
(2018, 3, 4950.95)
(2018, 4, 4961.84)
(2018, 5, 4981.69)
(2018, 6, 5044.46)
(2018, 7, 5061.11)
(2018, 8, 5056.56)
(2018, 9, 5080.83)
(2018, 10, 5103.69)
(2018, 11, 5092.97)
(2018, 12, 5100.61)
(2019, 1, 5116.93)
(2019, 2, 5138.93)
(2019, 3, 5177.47)
(2019, 4, 5206.98)
(2019, 5, 5213.75)
(2019, 6, 5214.27)
(2019, 7, 5224.18)
(2019, 8, 5229.93)
(2019, 9, 5227.84)
(2019, 10, 5233.07)
(2019, 11, 5259.76)
(2019, 12, 5320.25)
Fonte: IPCA > Série histórica
Para facilitar o cache e algumas outras funções (como ter backup dos dados em outros servidores que não o oficial), seria bom os adaptadores terem um to_csv
e from_csv
para exportar os dados em CSV e ler os dados do CSV (ao invés de carregar das fontes oficiais).
Os dados estão em um formato não muito favorável, mas pelo menos lendo só o HTML puro dá para raspar.
Talvez isso já seja implementado no #26, mas em todo caso:
Para melhorar a acurácia dos nossos testes, os testes de valores decimais deveriam adotar o pytest.approx
para evitar falhas desnecessárias em testes locais e na CI.
As datas podem ser objetos date
ou também str
no formato MM/YYYY
, mas não há verificação alguma checando se a data informada, quando str
, está nesse formato. Seria legal verificar isso antes de disparar a requisição HTTP.
Contexto
A fantástica Rows está sem novas versões desde fevereiro de 2019. Um commit de lá, de abril de 2019, resolve um problema de incompatibilidade com os Pythons mais novos:
tests/test_main.py:6: in <module>
from calculadora_do_cidadao import (
calculadora_do_cidadao/__init__.py:1: in <module>
from calculadora_do_cidadao.adapters.cpi import AllUrbanCityAverage # noqa
calculadora_do_cidadao/adapters/__init__.py:8: in <module>
from rows import export_to_csv, import_from_csv, import_from_dicts, import_from_html
../../../.cache/pypoetry/virtualenvs/calculadora-do-cidadao-WBmB8XYj-py3.9/lib/python3.9/site-packages/rows/__init__.py:22: in <module>
import rows.plugins as plugins
../../../.cache/pypoetry/virtualenvs/calculadora-do-cidadao-WBmB8XYj-py3.9/lib/python3.9/site-packages/rows/plugins/__init__.py:24: in <module>
from . import plugin_html as html
../../../.cache/pypoetry/virtualenvs/calculadora-do-cidadao-WBmB8XYj-py3.9/lib/python3.9/site-packages/rows/plugins/plugin_html.py:43: in <module>
unescape = HTMLParser().unescape
E AttributeError: 'HTMLParser' object has no attribute 'unescape'
Problema
Como o commit é posterior ao último release, não temos uma versão publicada com esse ajuste para usar nas dependências da Calculadora do Cidadão. Tentar publicar um pacote que tem uma dependência que não tem uma versão, mas é instalada de um repositório Git com uma branch ou commit dá erro:
Publishing calculadora-do-cidadao (0.5.3) to PyPI
- Uploading calculadora-do-cidadao-0.5.3.tar.gz 100%
UploadError
HTTP Error 400: Invalid value for requires_dist. Error: Can't have direct dependency: 'rows[csv,xls,html] @ git+https://github.com/turicas/rows.git@3a67d9dd65751e89281b28e3f1c30c4d6dc78e11'
Questões
Vejo alguns caminhos, mas não considero nenhum uma solução ainda — então boto para debate e para colher ideais:
frictionless
O que acham?
Atualmente, inicializar um adaptador passando um CSV só funciona se o CSV tiver sido exportado por um adaptador, ou seja, se o CSV tiver duas colunas (date
e value
). Como a versão 0.3.0 tem um formato de exportação geral, de todos os adaptadores, o ideal seria que um adaptador pudesse ser inicializado com esse CSV também, que tem três colunas (date
, value
e serie
):
A API e roadmap seria mais ou menos:
calculadora-do-cidadao.csv
com $ python -m calculadora_do_cidadao
ipca = Ipca('calculadora-do-cidadão.csv')
ipca
ler os valores de calculadora-do-cidadao.csv
apenas nas linhas do CSV onde serie == ipca
O código foi feito pensando apenas na "aba: 1", mas pode ser ampliado para os usos de caso das demais abas da Calculadora do Cidadão.
Olá!
Com Python 3.8.3+, tentando exportar os dados de todos os índices (adaptadores) de uma vez só, tenho o seguinte resultado, aqui:
jhh@jhhc:~$ python -m calculadora_do_cidadao
[1 of 7] Exporting ALLURBANCITYAVERAGE data…
Traceback (most recent call last):
File "/home/jhh/.pyenv/versions/3.8-dev/lib/python3.8/runpy.py", line 194, in _run_module_as_main
return _run_code(code, main_globals, None,
File "/home/jhh/.pyenv/versions/3.8-dev/lib/python3.8/runpy.py", line 87, in _run_code
exec(code, run_globals)
File "/home/jhh/.pyenv/versions/3.8-dev/lib/python3.8/site-packages/calculadora_do_cidadao/main.py", line 44, in
cli()
File "/home/jhh/.pyenv/versions/3.8-dev/lib/python3.8/site-packages/typer/main.py", line 213, in call
return get_command(self)()
File "/home/jhh/.pyenv/versions/3.8-dev/lib/python3.8/site-packages/click/core.py", line 829, in call
return self.main(*args, **kwargs)
File "/home/jhh/.pyenv/versions/3.8-dev/lib/python3.8/site-packages/click/core.py", line 782, in main
rv = self.invoke(ctx)
File "/home/jhh/.pyenv/versions/3.8-dev/lib/python3.8/site-packages/click/core.py", line 1066, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/home/jhh/.pyenv/versions/3.8-dev/lib/python3.8/site-packages/click/core.py", line 610, in invoke
return callback(*args, **kwargs)
File "/home/jhh/.pyenv/versions/3.8-dev/lib/python3.8/site-packages/typer/main.py", line 496, in wrapper
return callback(**use_params) # type: ignore
File "/home/jhh/.pyenv/versions/3.8-dev/lib/python3.8/site-packages/calculadora_do_cidadao/main.py", line 39, in export
table = import_from_dicts(data())
File "/home/jhh/.pyenv/versions/3.8-dev/lib/python3.8/site-packages/rows/plugins/dicts.py", line 35, in import_from_dicts
for index, row in enumerate(data, start=1):
File "/home/jhh/.pyenv/versions/3.8-dev/lib/python3.8/site-packages/calculadora_do_cidadao/main.py", line 33, in data
yield from adapter().export(include_name=True)
File "/home/jhh/.pyenv/versions/3.8-dev/lib/python3.8/site-packages/calculadora_do_cidadao/adapters/init.py", line 64, in init
self.data = {key: value for key, value in self.download()}
File "/home/jhh/.pyenv/versions/3.8-dev/lib/python3.8/site-packages/calculadora_do_cidadao/adapters/init.py", line 64, in
self.data = {key: value for key, value in self.download()}
File "/home/jhh/.pyenv/versions/3.8-dev/lib/python3.8/site-packages/calculadora_do_cidadao/adapters/init.py", line 176, in download
for data in self.read_from(path, **kwargs):
File "/home/jhh/.pyenv/versions/3.8-dev/lib/python3.8/site-packages/rows/plugins/xls.py", line 165, in import_from_xls
book = xlrd.open_workbook(filename, formatting_info=True)
File "/home/jhh/.pyenv/versions/3.8-dev/lib/python3.8/site-packages/xlrd/init.py", line 148, in open_workbook
bk = book.open_workbook_xls(
File "/home/jhh/.pyenv/versions/3.8-dev/lib/python3.8/site-packages/xlrd/book.py", line 82, in open_workbook_xls
bk.biff2_8_load(
File "/home/jhh/.pyenv/versions/3.8-dev/lib/python3.8/site-packages/xlrd/book.py", line 616, in biff2_8_load
raise XLRDError("File size is 0 bytes")
xlrd.biffh.XLRDError: File size is 0 bytes
Implementar adaptador para o IPCA-15.
Implementar adaptador para o IGP-M.
Definir um local padrão (pasta do usuário? raíz de execução do script?) e salvar um pickle dos Adapter.data
s para evitar fazer download a cada vez que for usar o projeto.
Olá!
Tenho aqui instaladas as versões python 2.7 e 3.7, em Linux Mint Debian Edition 4.
Na instalação está dando erro:
jhh@jhhc:~$ sudo pip install calculadora-do-cidadao
[sudo] password for jhh:
Collecting calculadora-do-cidadao
Downloading https://files.pythonhosted.org/packages/0c/71/f57cff817008a0e8f7eadbbf654874de6804c256771933258eaff27ebe8f/calculadora-do-cidadao-0.4.1.tar.gz
Complete output from command python setup.py egg_info:
Traceback (most recent call last):
File "", line 1, in
File "/tmp/pip-install-9DyaE5/calculadora-do-cidadao/setup.py", line 20, in
long_description=Path("README.md").read_text(encoding="utf-8"),
AttributeError: 'PosixPath' object has no attribute 'read_text'
----------------------------------------
Command "python setup.py egg_info" failed with error code 1 in /tmp/pip-install-9DyaE5/calculadora-do-cidadao/
Parabenizo aos desenvolvedores pelo excelente trabalho e agradeço antecipadamente pela atenção.
Encontrei uma divergência no uso da biblioteca para ajustar um valor pela selic,
utilizei o código abaixo :
from calculadora_do_cidadao import Selic
selic = Selic()
v = selic.adjust("2017-10", 800000,"2022-11")
print(v)
v = 1087869.386433910510912246451
Simulando no site da calculadora do cidadão encontra-se :
data inicial : 01/10/2017
data final : 01/11/2022
valor: 800000
valor corrigido = R$ 1.083.812,34 (REAL)
Os valores apresentam um diferença de R$ 4057,04
Rodando os testes locais, tudo passa, mas na CI tem algumas falhas de asserção estranhas que dizem que precisamos entender o porquê disso para tirar esse vermelhinho do pipeline : )datetime.date(2018, 7, 6)
não é igual a date(2018, 7, 6)
—
> assert DateField.deserialize(value) == date(2018, 7, 6)
E AssertionError: assert datetime.date(2018, 7, 7) == datetime.date(2018, 7, 6)
E + where datetime.date(2018, 7, 7) = <bound method DateField.deserialize of <class 'calculadora_do_cidadao.fields.DateField'>>(1530925200)
E + where <bound method DateField.deserialize of <class 'calculadora_do_cidadao.fields.DateField'>> = DateField.deserialize
E + and datetime.date(2018, 7, 6) = date(2018, 7, 6)
Um índice que é importante mas não está no escopo original do projeto é o INPC.
Agora pela manhã fiz um adapter
para ele, pois usa muito do código-base do IPCA e me pareceu simples:
from datetime import date
from decimal import Decimal
from typing import NamedTuple
from calculadora_do_cidadao.base import Adapter
from calculadora_do_cidadao.months import MONTHS
from calculadora_do_cidadao.typing import MaybeIndexesGenerator
class Inpc(Adapter):
file_type = "xls"
url = "ftp://ftp.ibge.gov.br/Precos_Indices_de_Precos_ao_Consumidor/INPC/Serie_Historica/inpc_SerieHist.zip"
IMPORT_KWARGS = {"end_column": 2}
SHOULD_UNZIP = True
def serialize(self, row: NamedTuple) -> MaybeIndexesGenerator:
self.last_year = getattr(self, "last_year", None)
year, month, value = row
if month not in MONTHS.keys():
return
year = int(year or self.last_year)
month = MONTHS[month]
reference_date = self.round_date(date(year, month, 1))
value = Decimal(value) / 100
self.last_year = year
yield reference_date, value
Rodei e funcionou. Ótimo.
Porém, quando rodo tox
, dá erro de tipo:
Input:
from datetime import date
from decimal import Decimal
import pytest
from calculadora_do_cidadao.adapters import Inpc
from calculadora_do_cidadao.base import AdapterDateNotAvailableError
@pytest.mark.parametrize(
"original,value,target,expected",
(
(date(2014, 3, 6), None, None, "1.361007124894175467688242800"),
(date(2011, 5, 8), 9, None, "14.373499236614377437778943450"),
(date(2009, 1, 12), 5, date(2013, 8, 1), "6.410734265150376567640231785"),
),
)
def test_data(original, value, target, expected, inpc_fixture, mocker):
download = mocker.patch("calculadora_do_cidadao.base.Download")
download.return_value.return_value.__enter__.return_value = inpc_fixture
inpc = Inpc()
assert len(inpc.data) == 312
assert inpc.adjust(original, value, target) == Decimal(pytest.approx(expected))
msg = r"This adapter has data from 01/1994 to 12/2019\. 02/2020 is out of range\."
with pytest.raises(AdapterDateNotAvailableError, match=msg):
inpc.adjust(date(2020, 2, 1))
Output:
[...]
__________ test_data[original0-None-None-1.361007124894175467688242800] ___________
original = datetime.date(2014, 3, 6), value = None, target = None
expected = '1.361007124894175467688242800'
inpc_fixture = PosixPath('/home/rodolfo/Documents/Github/calculadora-do-cidadao/tests/fixtures/inpc.xls')
mocker = <pytest_mock.plugin.MockFixture object at 0x7f48bbbd90b8>
@pytest.mark.parametrize(
"original,value,target,expected",
(
(date(2014, 3, 6), None, None, "1.361007124894175467688242800"),
(date(2011, 5, 8), 9, None, "14.373499236614377437778943450"),
(date(2009, 1, 12), 5, date(2013, 8, 1), "6.410734265150376567640231785"),
),
)
def test_data(original, value, target, expected, inpc_fixture, mocker):
download = mocker.patch("calculadora_do_cidadao.base.Download")
download.return_value.return_value.__enter__.return_value = inpc_fixture
inpc = Inpc()
assert len(inpc.data) == 312
> assert inpc.adjust(original, value, target) == Decimal(pytest.approx(expected))
E TypeError: cannot make approximate comparisons to non-numeric values: '1.361007124894175467688242800'
Este erro de tipo se repete nas três respostas esperadas -- não porque estejam erradas, mas porque não são consideradas numéricas (?).
Sabe o motivo?
Prezado(a)(s),
Gostaria de sanar uma dúvida:
from datetime import date
from decimal import Decimal
from calculadora_do_cidadao import Ipca
ipca=Ipca()
str_date_inicial = '01/04/2010'
str_date_final = '05/05/2011'
valorInicial = 1000
multiplicador = ipca.adjust(str_date_inicial, 1, str_date_final)
print(multiplicador*valorInicial)
from datetime import date
from decimal import Decimal
from calculadora_do_cidadao import Selic
selic = Selic()
str_date_inicial = '01/04/2010'
str_date_final = '05/05/2011'
valorInicial = 1000
multiplicador = selic.adjust(str_date_inicial, 1, str_date_final)
print(multiplicador*valorInicial)
Foi então que o sistema retornou o erro abaixo:
`C:\Users\trtrj\AppData\Local\Programs\Python\Python310\python.exe C:\Users\trtrj\Desktop\PycharmProjects\pythonProject2\correçãoMonetária.py
Traceback (most recent call last):
File "C:\Users\trtrj\AppData\Local\Programs\Python\Python310\lib\site-packages\calculadora_do_cidadao\download.py", line 94, in __call__
yield path
File "C:\Users\trtrj\AppData\Local\Programs\Python\Python310\lib\site-packages\calculadora_do_cidadao\adapters\__init__.py", line 206, in download
for data in self.read_from(path, **kwargs): # type: ignore
File "C:\Users\trtrj\AppData\Local\Programs\Python\Python310\lib\site-packages\calculadora_do_cidadao\rows\plugins\plugin_html.py", line 84, in import_from_html
table = tables[index]
IndexError: list index out of range
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "C:\Users\trtrj\AppData\Local\Programs\Python\Python310\lib\shutil.py", line 617, in _rmtree_unsafe
os.unlink(fullname)
PermissionError: [WinError 32] The process cannot access the file because it is being used by another process: 'C:\\Users\\trtrj\\AppData\\Local\\Temp\\tmpt07hhrhc\\taxa-de-juros-selic'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "C:\Users\trtrj\AppData\Local\Programs\Python\Python310\lib\tempfile.py", line 843, in onerror
_os.unlink(path)
PermissionError: [WinError 32] The process cannot access the file because it is being used by another process: 'C:\\Users\\trtrj\\AppData\\Local\\Temp\\tmpt07hhrhc\\taxa-de-juros-selic'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "C:\Users\trtrj\Desktop\PycharmProjects\pythonProject2\correçãoMonetária.py", line 20, in <module>
selic = Selic()
File "C:\Users\trtrj\AppData\Local\Programs\Python\Python310\lib\site-packages\calculadora_do_cidadao\adapters\__init__.py", line 71, in __init__
self.data = {key: value for key, value in self.download()}
File "C:\Users\trtrj\AppData\Local\Programs\Python\Python310\lib\site-packages\calculadora_do_cidadao\adapters\__init__.py", line 71, in <dictcomp>
self.data = {key: value for key, value in self.download()}
File "C:\Users\trtrj\AppData\Local\Programs\Python\Python310\lib\site-packages\calculadora_do_cidadao\adapters\__init__.py", line 204, in download
with download() as path:
File "C:\Users\trtrj\AppData\Local\Programs\Python\Python310\lib\contextlib.py", line 153, in __exit__
self.gen.throw(typ, value, traceback)
File "C:\Users\trtrj\AppData\Local\Programs\Python\Python310\lib\site-packages\calculadora_do_cidadao\download.py", line 86, in __call__
with TemporaryDirectory() as tmp:
File "C:\Users\trtrj\AppData\Local\Programs\Python\Python310\lib\tempfile.py", line 869, in __exit__
self.cleanup()
File "C:\Users\trtrj\AppData\Local\Programs\Python\Python310\lib\tempfile.py", line 873, in cleanup
self._rmtree(self.name, ignore_errors=self._ignore_cleanup_errors)
File "C:\Users\trtrj\AppData\Local\Programs\Python\Python310\lib\tempfile.py", line 855, in _rmtree
_shutil.rmtree(name, onerror=onerror)
File "C:\Users\trtrj\AppData\Local\Programs\Python\Python310\lib\shutil.py", line 749, in rmtree
return _rmtree_unsafe(path, onerror)
File "C:\Users\trtrj\AppData\Local\Programs\Python\Python310\lib\shutil.py", line 619, in _rmtree_unsafe
onerror(os.unlink, fullname, sys.exc_info())
File "C:\Users\trtrj\AppData\Local\Programs\Python\Python310\lib\tempfile.py", line 846, in onerror
cls._rmtree(path, ignore_errors=ignore_errors)
File "C:\Users\trtrj\AppData\Local\Programs\Python\Python310\lib\tempfile.py", line 855, in _rmtree
_shutil.rmtree(name, onerror=onerror)
File "C:\Users\trtrj\AppData\Local\Programs\Python\Python310\lib\shutil.py", line 749, in rmtree
return _rmtree_unsafe(path, onerror)
File "C:\Users\trtrj\AppData\Local\Programs\Python\Python310\lib\shutil.py", line 600, in _rmtree_unsafe
onerror(os.scandir, path, sys.exc_info())
File "C:\Users\trtrj\AppData\Local\Programs\Python\Python310\lib\shutil.py", line 597, in _rmtree_unsafe
with os.scandir(path) as scandir_it:
NotADirectoryError: [WinError 267] The directory name is invalid: 'C:\\Users\\trtrj\\AppData\\Local\\Temp\\tmpt07hhrhc\\taxa-de-juros-selic'
Process finished with exit code 1`
Pode(m) me dar alguma sugestão?
Muito obrigado
O @turicas deu o toque:
coluna
mesano
vai virarmes_ano
com uma correção que fiz na rows (que tá nodevelop
)
Precisamos de alguma alternativa para deixar esse trecho mais flexível (de repente ter um fallback ou algo assim:
calculadora-do-cidadao/calculadora_do_cidadao/adapters/selic.py
Lines 29 to 30 in c377a4e
Conforme identificado na issue #38, o adaptador AllUrbanCityAverage
não funciona mais.
In [6]: fed = AllUrbanCityAverage()
---------------------------------------------------------------------------
XLRDError Traceback (most recent call last)
<ipython-input-6-a02f4981ee86> in <module>
----> 1 fed = AllUrbanCityAverage()
~/Dropbox/Projects/calculadora-do-cidadao/calculadora_do_cidadao/adapters/__init__.py in __init__(self, exported_csv)
62 self.data = {key: value for key, value in self.from_csv(exported_csv)}
63 else:
---> 64 self.data = {key: value for key, value in self.download()}
65 if self.should_aggregate:
66 self.aggregate()
~/Dropbox/Projects/calculadora-do-cidadao/calculadora_do_cidadao/adapters/__init__.py in <dictcomp>(.0)
62 self.data = {key: value for key, value in self.from_csv(exported_csv)}
63 else:
---> 64 self.data = {key: value for key, value in self.download()}
65 if self.should_aggregate:
66 self.aggregate()
~/Dropbox/Projects/calculadora-do-cidadao/calculadora_do_cidadao/adapters/__init__.py in download(self)
174 with download() as path:
175 for kwargs in self.import_kwargs:
--> 176 for data in self.read_from(path, **kwargs):
177 yield from (row for row in self.serialize(data) if row)
178
~/.virtualenvs/tempenv-49a7297032c7a/lib/python3.8/site-packages/rows/plugins/xls.py in import_from_xls(filename_or_fobj, sheet_name, sheet_index, start_row, start_column, end_row, end_column, *args, **kwargs)
163
164 filename, _ = get_filename_and_fobj(filename_or_fobj, mode="rb")
--> 165 book = xlrd.open_workbook(filename, formatting_info=True)
166 if sheet_name is not None:
167 sheet = book.sheet_by_name(sheet_name)
~/.virtualenvs/tempenv-49a7297032c7a/lib/python3.8/site-packages/xlrd/__init__.py in open_workbook(filename, logfile, verbosity, use_mmap, file_contents, encoding_override, formatting_info, on_demand, ragged_rows)
146
147 from . import book
--> 148 bk = book.open_workbook_xls(
149 filename=filename,
150 logfile=logfile,
~/.virtualenvs/tempenv-49a7297032c7a/lib/python3.8/site-packages/xlrd/book.py in open_workbook_xls(filename, logfile, verbosity, use_mmap, file_contents, encoding_override, formatting_info, on_demand, ragged_rows)
80 bk = Book()
81 try:
---> 82 bk.biff2_8_load(
83 filename=filename, file_contents=file_contents,
84 logfile=logfile, verbosity=verbosity, use_mmap=use_mmap,
~/.virtualenvs/tempenv-49a7297032c7a/lib/python3.8/site-packages/xlrd/book.py in biff2_8_load(self, filename, file_contents, logfile, verbosity, use_mmap, encoding_override, formatting_info, on_demand, ragged_rows)
614 f.seek(0, 0) # BOF
615 if size == 0:
--> 616 raise XLRDError("File size is 0 bytes")
617 if self.use_mmap:
618 self.filestr = mmap.mmap(f.fileno(), size, access=mmap.ACCESS_READ)
XLRDError: File size is 0 bytes
Todos esses índices não começam com g
:
calculadora-do-cidadao/calculadora_do_cidadao.py
Lines 16 to 24 in b824b5b
Podemos usar qualquer ferramenta para ajudar nisso, não precisa ser só o setuptools
.
Seguindo o exemplo do @diraol, não precisa, como está agora, explicitamente converter para str
e, depois, usar zfill
-- podemos usar f-strings com :0>2d
.
Algumas ideias:
Criar uma alternativa que use o aiohttp
ao invés do requests
A ideia é que toda a lógica fique no base.Adapter
e as classes de adapters
sejam só configuração.
Vale notar que ainda precisa de suporte para parametrizar fontes de dados que não são compactadas com Zip ou não são em formato Excel.
Ao executar python setup.py test
o Black falha com:
Traceback (most recent call last):
File "/opt/hostedtoolcache/Python/3.8.0/x64/lib/python3.8/runpy.py", line 192, in _run_module_as_main
return _run_code(code, main_globals, None,
File "/opt/hostedtoolcache/Python/3.8.0/x64/lib/python3.8/runpy.py", line 85, in _run_code
exec(code, run_globals)
File "/home/runner/work/calculadora-do-cidadao/calculadora-do-cidadao/.eggs/black-19.3b0-py3.8.egg/black.py", line 79, in <module>
pygram.initialize(CACHE_DIR)
File "/home/runner/work/calculadora-do-cidadao/calculadora-do-cidadao/.eggs/black-19.3b0-py3.8.egg/blib2to3/pygram.py", line 41, in initialize
python_grammar = driver.load_packaged_grammar("blib2to3", _GRAMMAR_FILE,
File "/home/runner/work/calculadora-do-cidadao/calculadora-do-cidadao/.eggs/black-19.3b0-py3.8.egg/blib2to3/pgen2/driver.py", line 198, in load_packaged_grammar
gp = _generate_pickle_name(grammar_source, cache_dir) if cache_dir else None
File "/home/runner/work/calculadora-do-cidadao/calculadora-do-cidadao/.eggs/black-19.3b0-py3.8.egg/blib2to3/pgen2/driver.py", line 151, in _generate_pickle_name
return os.path.join(cache_dir, os.path.basename(name))
File "/opt/hostedtoolcache/Python/3.8.0/x64/lib/python3.8/posixpath.py", line 76, in join
a = os.fspath(a)
TypeError: expected str, bytes or os.PathLike object, not PosixPath
Esse erro não ocorre se instalamos as dependências e rodamos pytest
.
Por questões de implementação o método adjust
espera que as datas chegem com o tipo date
. Para facilitar a vida do usuário, podemos ter uma tentativa de conversão de alguns outros tipos para date
, deixando a API mais flexível:
Entrada | Tipo | Valor convertido |
---|---|---|
"2020-07-31" |
str |
date(2020, 7, 31) |
"31/07/2020" |
str |
date(2020, 7, 31) |
"2020-07" |
str |
date(2020, 7, 1) |
"2020" |
str |
date(2020, 1, 1) |
"07/2020" |
str |
date(2020, 7, 1) |
2020 |
int |
date(2020, 1, 1) |
Também podemos descartar strings maiores que 10 caracteres (oferecer compatibilidade com datetime
no formato ISO), e aceitar datetime
como entrada. Faz sentido?
Tentei rodar o exemplo:
from datetime import date
from decimal import Decimal
from calculadora_do_cidadao import Ipca
ipca = Ipca()
result = ipca.adjust(date(2018, 7, 6))
print(result)
Decimal('1.051202206630561280035407253')
result = ipca.adjust(date(2014, 7, 8), 7)
Decimal('9.407523138792336916983267321')
print(result)
result = ipca.adjust(date(1998, 7, 12), 3, date(2006, 7, 1))
Decimal('5.279855889296777979447848574')
print(result)
Recebi o seguinte erro:
Traceback (most recent call last):
File "analise.py", line 5, in <module>
ipca = Ipca15()
File "/home/perrout/.local/lib/python3.8/site-packages/calculadora_do_cidadao/adapters/__init__.py", line 67, in __init__
self.data = {key: value for key, value in self.download()}
File "/home/perrout/.local/lib/python3.8/site-packages/calculadora_do_cidadao/adapters/__init__.py", line 67, in <dictcomp>
self.data = {key: value for key, value in self.download()}
File "/home/perrout/.local/lib/python3.8/site-packages/calculadora_do_cidadao/adapters/__init__.py", line 200, in download
with download() as path:
File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__
return next(self.gen)
File "/home/perrout/.local/lib/python3.8/site-packages/calculadora_do_cidadao/download.py", line 96, in __call__
path = self.download_to(Path(tmp) / self.file_name)
File "/home/perrout/.local/lib/python3.8/site-packages/calculadora_do_cidadao/download.py", line 87, in ftp
conn.retrbinary(f"RETR {self.parsed_url.path}", fobj.write)
File "/usr/lib/python3.8/ftplib.py", line 425, in retrbinary
with self.transfercmd(cmd, rest) as conn:
File "/usr/lib/python3.8/ftplib.py", line 382, in transfercmd
return self.ntransfercmd(cmd, rest)[0]
File "/usr/lib/python3.8/ftplib.py", line 343, in ntransfercmd
conn = socket.create_connection((host, port), self.timeout,
File "/usr/lib/python3.8/socket.py", line 807, in create_connection
raise err
File "/usr/lib/python3.8/socket.py", line 796, in create_connection
sock.connect(sa)
TimeoutError: [Errno 110] Connection timed out
Testei outros adaptadores e rodaram normalmente (apenas o do IBGE utiliza FTP).
Meu ambiente: WSL2 Ubuntu (Windows 10)
Alterei o protocolo de FTP para HTTP no arquivo ibge.py
e funcionou normalmente:
class Ipca(IbgeAdapter):
"""Adapter for IBGE's IPCA series."""
url = "http://ftp.ibge.gov.br/Precos_Indices_de_Precos_ao_Consumidor/IPCA/Serie_Historica/ipca_SerieHist.zip"
Resultado:
1.161020013396270778544627562
10.39031555654352922156656697
5.279855889296777979447848574
Já que python setup.py test
vai ser descontinuado, implementar tox
para rodar os testes nas versões 3.6, 3.7 e 3.8 do Python (eliminando #17).
Lembrar de testar e documentar que as datas podem ser objetos date
ou também str
no formato MM/YYYY
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.