iamrony777 / javinfo-api Goto Github PK

Hello, can you add new provider javdb ?
We have 4 version in the old apis, and many series only found on javdb.

Btw, I also curious on what happen if we set providers to all, and the series only found in r18, but not on jvdtbs and jvlib, will it show as 404 error or return successful response ?

/public returns 429 instead of 404 even if it isn't rate limited

It seems like javdb and javlibrary are now working anymore.

Exception in request module, requests.raise_for_status()
404 Client Error: Not Found for url: https://javinfo-api.herokuapp.com/api/public?name=ssis-386&provider=javdb

Recently, any request from jvlib began to return a 404 error, despite the fact that the site itself is working and the page exists

r18.py line 37
extra_details["release_data"] = response.get("release_date").split(" ")[0]

misspelling, should be extra_details["release_date"]

This is causing json data return to be inconsistent from other provider sources.

Can we add priority list to the provider ?
I mean the order of scraping, instead of get data from the fastest provider, it will search the priority provider
for example
It will go first from r18-javdb-javdatabase-javlibrary in the way we arrange / setting the priority, because each site has different category / tags, make the data seems complicated

it will search first from r18, if no result it goes to javdb, then the next, not just random scraping the fastest.

I think there's some structure changed on the website that cause some functions of the scraper fails, for now I only use /search to do the api request, the other not tested yet.

javlibrary working fine but no screenshots scraped.

javdb all function is working

javdatabase and r18 not working at all, even though the jav code is valid on the site, it said not found on the logs contain a lot of none type error. I also provided login on the environment , javdatabase still failed.

• provider: javdb

error

INFO:     10.220.130.11:0 - "GET /jav/search?code=KSXA-001&provider=all&includeActressUrl=true HTTP/1.1" 500 Internal Server Error
Nov 23 09:22:32 AMERRORERROR:    Exception in ASGI application
Nov 23 09:22:32 AMINFOTraceback (most recent call last):
Nov 23 09:22:32 AMINFO  File "/usr/local/lib/python3.12/site-packages/uvicorn/protocols/http/h11_impl.py", line 408, in run_asgi
Nov 23 09:22:32 AMINFO    result = await app(  # type: ignore[func-returns-value]
Nov 23 09:22:32 AMINFO             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Nov 23 09:22:32 AMINFO  File "/usr/local/lib/python3.12/site-packages/uvicorn/middleware/proxy_headers.py", line 84, in __call__
Nov 23 09:22:32 AMINFO    return await self.app(scope, receive, send)
Nov 23 09:22:32 AMINFO           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Nov 23 09:22:32 AMINFO  File "/usr/local/lib/python3.12/site-packages/fastapi/applications.py", line 1106, in __call__
Nov 23 09:22:32 AMINFO    await super().__call__(scope, receive, send)
Nov 23 09:22:32 AMINFO  File "/usr/local/lib/python3.12/site-packages/starlette/applications.py", line 122, in __call__
Nov 23 09:22:32 AMINFO    await self.middleware_stack(scope, receive, send)
Nov 23 09:22:32 AMINFO  File "/usr/local/lib/python3.12/site-packages/starlette/middleware/errors.py", line 184, in __call__
Nov 23 09:22:32 AMINFO    raise exc
Nov 23 09:22:32 AMINFO  File "/usr/local/lib/python3.12/site-packages/starlette/middleware/errors.py", line 162, in __call__
Nov 23 09:22:32 AMINFO    await self.app(scope, receive, _send)
Nov 23 09:22:32 AMINFO  File "/usr/local/lib/python3.12/site-packages/starlette/middleware/exceptions.py", line 79, in __call__
Nov 23 09:22:32 AMINFO    raise exc
Nov 23 09:22:32 AMINFO  File "/usr/local/lib/python3.12/site-packages/starlette/middleware/exceptions.py", line 68, in __call__
Nov 23 09:22:32 AMINFO    await self.app(scope, receive, sender)
Nov 23 09:22:32 AMINFO  File "/usr/local/lib/python3.12/site-packages/fastapi/middleware/asyncexitstack.py", line 20, in __call__
Nov 23 09:22:32 AMINFO    raise e
Nov 23 09:22:32 AMINFO  File "/usr/local/lib/python3.12/site-packages/fastapi/middleware/asyncexitstack.py", line 17, in __call__
Nov 23 09:22:32 AMINFO    await self.app(scope, receive, send)
Nov 23 09:22:32 AMINFO  File "/usr/local/lib/python3.12/site-packages/starlette/routing.py", line 718, in __call__
Nov 23 09:22:32 AMINFO    await route.handle(scope, receive, send)
Nov 23 09:22:32 AMINFO  File "/usr/local/lib/python3.12/site-packages/starlette/routing.py", line 276, in handle
Nov 23 09:22:32 AMINFO    await self.app(scope, receive, send)
Nov 23 09:22:32 AMINFO  File "/usr/local/lib/python3.12/site-packages/starlette/routing.py", line 66, in app
Nov 23 09:22:32 AMINFO    response = await func(request)
Nov 23 09:22:32 AMINFO               ^^^^^^^^^^^^^^^^^^^
Nov 23 09:22:32 AMINFO  File "/usr/local/lib/python3.12/site-packages/fastapi/routing.py", line 274, in app
Nov 23 09:22:32 AMINFO    raw_response = await run_endpoint_function(
Nov 23 09:22:32 AMINFO                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Nov 23 09:22:32 AMINFO  File "/usr/local/lib/python3.12/site-packages/fastapi/routing.py", line 191, in run_endpoint_function
Nov 23 09:22:32 AMINFO    return await dependant.call(**values)
Nov 23 09:22:32 AMINFO           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Nov 23 09:22:32 AMINFO  File "/app/src/routes/jav.py", line 68, in search
Nov 23 09:22:32 AMINFO    response = search_all_providers(code, provider, includeActressUrl)
Nov 23 09:22:32 AMINFO               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Nov 23 09:22:32 AMINFO  File "/app/src/__init__.py", line 43, in search_all_providers
Nov 23 09:22:32 AMINFO    result = task.result()
Nov 23 09:22:32 AMINFO             ^^^^^^^^^^^^^
Nov 23 09:22:32 AMINFO  File "/usr/local/lib/python3.12/concurrent/futures/_base.py", line 449, in result
Nov 23 09:22:32 AMINFO    return self.__get_result()
Nov 23 09:22:32 AMINFO           ^^^^^^^^^^^^^^^^^^^
Nov 23 09:22:32 AMINFO  File "/usr/local/lib/python3.12/concurrent/futures/_base.py", line 401, in __get_result
Nov 23 09:22:32 AMINFO    raise self._exception
Nov 23 09:22:32 AMINFO  File "/usr/local/lib/python3.12/concurrent/futures/thread.py", line 58, in run
Nov 23 09:22:32 AMINFO    result = self.fn(*self.args, **self.kwargs)
Nov 23 09:22:32 AMINFO             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Nov 23 09:22:32 AMINFO  File "/app/src/providers/javdb.py", line 140, in search
Nov 23 09:22:32 AMINFO    return self.__getJsonResult(
Nov 23 09:22:32 AMINFO           ^^^^^^^^^^^^^^^^^^^^^
Nov 23 09:22:32 AMINFO  File "/app/src/providers/javdb.py", line 57, in __getJsonResult
Nov 23 09:22:32 AMINFO    _id = page.xpath('//a[@title="Copy ID"]')[0].get('data-clipboard-text')
Nov 23 09:22:32 AMINFO          ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^
Nov 23 09:22:32 AMINFO  IndexError: list index out of range

I wonder if will be better having a local database which created and cached by the websracpers. The local database can be built in this way, then it would provide two options for users. let me know if there r any concerns. Thanks

Hi iamrony777, apologies for a noob question in advance.
While trying to build a docker image I am getting the following error.

#0 160.8 check.warn(importable)
#0 160.8 /usr/local/lib/python3.11/site-packages/setuptools/command/build_py.py:202: SetuptoolsDeprecationWarning: Installing 'uvloop.includes' as data is deprecated, please list it in packages.
#0 160.8 !!
#0 160.8
#0 160.8
#0 160.8 ############################
#0 160.8 # Package would be ignored #
#0 160.8 ############################
#0 160.8 Python recognizes 'uvloop.includes' as an importable package,
#0 160.8 but it is not listed in the packages configuration of setuptools.
#0 160.8
#0 160.8 'uvloop.includes' has been automatically added to the distribution only
#0 160.8 because it may contain data files, but this behavior is likely to change
#0 160.8 in future versions of setuptools (and therefore is considered deprecated).
#0 160.8
#0 160.8 Please make sure that 'uvloop.includes' is included as a package by using
#0 160.8 the packages configuration field or the proper discovery methods
#0 160.8 (for example by using find_namespace_packages(...)/find_namespace:
#0 160.8 instead of find_packages(...)/find:).
#0 160.8
#0 160.8 You can read more about "package discovery" and "data files" on setuptools
#0 160.8 documentation page.
#0 160.8
#0 160.8
#0 160.8 !!

#0 160.8 building 'uvloop.loop' extension
#0 160.8 creating build/temp.linux-aarch64-cpython-311
#0 160.8 creating build/temp.linux-aarch64-cpython-311/uvloop
#0 160.8 gcc -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -DTHREAD_STACK_SIZE=0x100000 -fPIC -I/usr/local/include/python3.11 -I/tmp/pip-wheel-1j2g1hhw/uvloop_f485260deea54c04b493ff91e427feea/vendor/libuv/include -c uvloop/loop.c -o build/temp.linux-aarch64-cpython-311/uvloop/loop.o -O2
#0 160.8 uvloop/loop.c:214:12: fatal error: longintrepr.h: No such file or directory
#0 160.8 214 | #include "longintrepr.h"
#0 160.8 | ^~~~~~~~~~~~~~~
#0 160.8 compilation terminated.
#0 160.8 error: command '/usr/bin/gcc' failed with exit code 1
#0 160.8 [end of output]
#0 160.8
#0 160.8 note: This error originates from a subprocess, and is likely not a problem with pip.
#0 160.8 ERROR: Failed building wheel for uvloop
#0 160.8 Running setup.py clean for uvloop
#0 161.0 Building wheel for watchdog (setup.py): started
#0 161.2 Building wheel for watchdog (setup.py): finished with status 'done'
#0 161.2 Created wheel for watchdog: filename=watchdog-2.1.9-py3-none-any.whl size=78393 sha256=f48fad5499747856b33332f8c65395a464d5acdcc884bc908e5a04954dcb44be
#0 161.2 Stored in directory: /root/.cache/pip/wheels/6c/83/70/756b9cdd0256de299434cdfc81375baffca3e3a6a305c83afc
#0 161.2 Building wheel for wrapt (setup.py): started
#0 162.0 Building wheel for wrapt (setup.py): finished with status 'done'
#0 162.0 Created wheel for wrapt: filename=wrapt-1.14.1-cp311-cp311-linux_aarch64.whl size=35027 sha256=7c90efbae1dbd9a58ea033e2b3b0d03f04278a4577fd61cc71f9c2a6dbc32a65
#0 162.0 Stored in directory: /root/.cache/pip/wheels/6c/5f/ff/a07f5f99c11c1480c6d2d1d6e0adadd8c1d05bdd668173bc42
#0 162.0 Successfully built lxml markupsafe pillow pyyaml watchdog wrapt
#0 162.0 Failed to build uvloop
#0 162.0 ERROR: Failed to build one or more wheels

failed to solve: executor failed running [/bin/sh -c MAKEFLAGS="-j$(nproc)" SETUPFLAGS="-j$(nproc)" ; pip wheel --wheel-dir=/app/wheels -r requirements.txt]: exit code: 1

fixed

Hey,

I've configured the .env file API username and password, and build the docker using docker-compose, but when using with CURL, passing the API and username and password, I receive the following error:

{"detail":"Access Denied"}

Any suggestion on what can be wrong? I tried to rebuild and reset the .env file, but the same error occurs.

Thanks in advance