lvxhnat / pyetfdb-scraper

Scrape ETF data for free - the unofficial python wrapper for ETFDB

Hi there 👋

I am a codie that loves to work on the forefront of intersections between data science and software engineering ~ On my free time, I enjoy creating data workflows to help gain insight into real-world problems. I have worked with MNCs such as Pepsi, BBC in delivering GPU accelerated Graph Analytics & Topic Modelling.

Connect with me at [email protected], or give me a ping on LinkedIn

🕛 Currently: FT/PT Data Software Engineer working on Fuzzy Cognitive Maps @ Synthesis
⚡ Previously: ML Engineering in Trondheim, Norway 🇳🇴

pyetfdb-scraper's Issues

review schema for etfdb results

Add pydantic to dependency list of the package

Describe the bug
after pip install the package, you would have to install pydantic manually.
You forgot to add pydantic as a dependency

Fix
Add one line here: https://github.com/lvxhnat/pyetfdb-scraper/blob/master/setup.py#L40

        "pydantic",

Implement pre-commit hooks

Is your feature request related to a problem? Please describe.
There are no standards for code formatting, and the default formatter on the editor of a contributor formats the code. Adding test cases also implies the need for this.

Describe the solution you'd like
Add pre-commit hooks to format and check for issues, integrate test cases

Describe alternatives you've considered

Additional context

Implement Test Cases

Is your feature request related to a problem? Please describe.
ETF Objects should be tested to ensure that missing fields or ETFs are handled. Currently, some of the fields and the request to ETFDB itself does not handle exceptions such as 404/500 errors gracefully, and has to be caught when calling the object itself.

Describe the solution you'd like
Implemented test cases for the ETF object.

Describe alternatives you've considered

Additional context

Proxy rotations as parameter for scraping pyETFDB

Is your feature request related to a problem? Please describe.

ETFDB blocks a user who has been requesting tickers after 200-300 repetitive requests.

Describe the solution you'd like

There should be a parameter in the ETF object to allow a proxy and user agent to be passed in, so the user can perform proxy rotations as they like.

Describe alternatives you've considered

Additional context

PyPi Package doesn't include user-agents.txt

Describe the bug
On a fresh install of the package, user-agents.txt is missing:

FileNotFoundError: [Errno 2] No such file or directory: '/home/ben/.cache/pypoetry/virtualenvs/finance-ItsvA0nu-py3.11/lib/python3.11/site-packages/pyetfdb_scraper/data/user-agents.txt'

If you download the src directly from pypi: https://files.pythonhosted.org/packages/33/4d/617eb1f807bc5ab4c9c2af13240facb55aed350325ee2b28d1457a3dcdf5/pyetfdb_scraper-0.2.tar.gz

You can see the file is not on the tarball and that the setup.py doesn't have the package_data line found here: https://github.com/lvxhnat/pyetfdb-scraper/blob/master/setup.py#L36

I'm not sure how the build system works but it seems that the build on pypi is out of date and missing commits like 88dcc0c

To Reproduce
Steps to reproduce the behavior:

1. Do fresh install of package from pypi (not github)

etf = ETF("IHAK")
print(etf.to_dict())

Expected behavior
Dictionary output

Screenshots
If applicable, add screenshots to help explain your problem.

Desktop (please complete the following information):

• OS: Ubuntu / Linux 6.6.22-060622-generic
• Python 3.11.6

Smartphone (please complete the following information):
N/A

Additional context
Full traceback:

Traceback (most recent call last):
  File "/home/ben/workplace/finance/./finnhib.py", line 20, in <module>
    etf = ETF("IHAK")
          ^^^^^^^^^^^
  File "/home/ben/.cache/pypoetry/virtualenvs/finance-ItsvA0nu-py3.11/lib/python3.11/site-packages/pyetfdb_scraper/etf.py", line 12, in __init__
    super().__init__(ticker, user_agent)
  File "/home/ben/.cache/pypoetry/virtualenvs/finance-ItsvA0nu-py3.11/lib/python3.11/site-packages/pyetfdb_scraper/etf_scraper.py", line 32, in __init__
    self.user_agents = load_user_agents()
                       ^^^^^^^^^^^^^^^^^^
  File "/home/ben/.cache/pypoetry/virtualenvs/finance-ItsvA0nu-py3.11/lib/python3.11/site-packages/pyetfdb_scraper/etf_scraper.py", line 129, in load_user_agents
    with open(path, "r") as f:
         ^^^^^^^^^^^^^^^
FileNotFoundError: [Errno 2] No such file or directory: '/home/ben/.cache/pypoetry/virtualenvs/finance-ItsvA0nu-py3.11/lib/python3.11/site-packages/pyetfdb_scraper/data/user-agents.txt'
(8:42:48) ben@fralix:~/workplace/finance (master* ^ detached) (No VPN)    
(finance-py3.11) % poetry run python ./finnhib.py
Traceback (most recent call last):
  File "/home/ben/workplace/finance/./finnhib.py", line 20, in <module>
    etf = ETF("IHAK")
          ^^^^^^^^^^^
  File "/home/ben/.cache/pypoetry/virtualenvs/finance-ItsvA0nu-py3.11/lib/python3.11/site-packages/pyetfdb_scraper/etf.py", line 12, in __init__
    super().__init__(ticker, user_agent)
  File "/home/ben/.cache/pypoetry/virtualenvs/finance-ItsvA0nu-py3.11/lib/python3.11/site-packages/pyetfdb_scraper/etf_scraper.py", line 32, in __init__
    self.user_agents = load_user_agents()
                       ^^^^^^^^^^^^^^^^^^
  File "/home/ben/.cache/pypoetry/virtualenvs/finance-ItsvA0nu-py3.11/lib/python3.11/site-packages/pyetfdb_scraper/etf_scraper.py", line 129, in load_user_agents
    with open(path, "r") as f:
         ^^^^^^^^^^^^^^^
FileNotFoundError: [Errno 2] No such file or directory: '/home/ben/.cache/pypoetry/virtualenvs/finance-ItsvA0nu-py3.11/lib/python3.11/site-packages/pyetfdb_scraper/data/user-agents.txt'

Support ETFDB Pro Users

Is your feature request related to a problem? Please describe.
Currently ETFDB Pro users are unable to access pro data on this package.

Describe the solution you'd like
Explore what are some methods to access pro data, whether an API key is distributed, or selenium have to be used etc.

Describe alternatives you've considered

Additional context