ivory-tower-private-power / sirup Goto Github PK
View Code? Open in Web Editor NEWSimple IP rotation using python.
License: Apache License 2.0
Simple IP rotation using python.
License: Apache License 2.0
Fetching the IP address sometimes fails.
One example traceback:
2023-08-11 18:26:08 urllib3.connectionpool WARNING Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7ff8c8bb0e50>: Failed to establish a new connection: [Errno -3] Temporary failure in name resolution')': /
2023-08-11 18:26:19 urllib3.connectionpool WARNING Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7ff8c8bb1650>: Failed to establish a new connection: [Errno -3] Temporary failure in name resolution')': /
2023-08-11 18:26:27 urllib3.connectionpool WARNING Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7ff8c8bb20d0>: Failed to establish a new connection: [Errno -3] Temporary failure in name resolution')': /
2023-08-11 18:26:32 root INFO Got an exception: HTTPSConnectionPool(host='ifconfig.me', port=443): Max retries exceeded with url: / (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7ff8c8bb2b50>: Failed to establish a new connection: [Errno -3] Temporary failure in name resolution'))
2023-08-11 18:26:32 root INFO Got exception ConnectionError('Failed to get the IP address')
2023-08-11 18:26:32 root INFO rotator.connector.base_ip='185.143.104.11'
2023-08-11 18:26:32 root INFO rotator.connector.current_ip='154.47.25.139'
2023-08-11 18:26:32 root INFO No openvpn processes found to be killed.
2023-08-11 18:26:47 urllib3.connectionpool WARNING Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7ff8c8b64910>: Failed to establish a new connection: [Errno -3] Temporary failure in name resolution')': /
2023-08-11 18:26:54 urllib3.connectionpool WARNING Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7ff8c8b654d0>: Failed to establish a new connection: [Errno -3] Temporary failure in name resolution')': /
2023-08-11 18:27:01 urllib3.connectionpool WARNING Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7ff8c8b66110>: Failed to establish a new connection: [Errno -3] Temporary failure in name resolution')': /
2023-08-11 18:27:05 root INFO Got an exception: HTTPSConnectionPool(host='ifconfig.me', port=443): Max retries exceeded with url: / (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7ff8c8b66c90>: Failed to establish a new connection: [Errno -3] Temporary failure in name resolution'))
Traceback (most recent call last):
File "/home/flavio/repositories/websearch-public/env/lib/python3.11/site-packages/urllib3/connection.py", line 174, in _new_conn
conn = connection.create_connection(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/flavio/repositories/websearch-public/env/lib/python3.11/site-packages/urllib3/util/connection.py", line 72, in create_connection
for res in socket.getaddrinfo(host, port, family, socket.SOCK_STREAM):
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/flavio/repositories/websearch-public/env/lib/python3.11/socket.py", line 962, in getaddrinfo
for res in _socket.getaddrinfo(host, port, family, type, proto, flags):
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
socket.gaierror: [Errno -3] Temporary failure in name resolution
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/flavio/repositories/websearch-public/env/lib/python3.11/site-packages/urllib3/connectionpool.py", line 714, in urlopen
httplib_response = self._make_request(
^^^^^^^^^^^^^^^^^^^
File "/home/flavio/repositories/websearch-public/env/lib/python3.11/site-packages/urllib3/connectionpool.py", line 403, in _make_request
self._validate_conn(conn)
File "/home/flavio/repositories/websearch-public/env/lib/python3.11/site-packages/urllib3/connectionpool.py", line 1053, in _validate_conn
conn.connect()
File "/home/flavio/repositories/websearch-public/env/lib/python3.11/site-packages/urllib3/connection.py", line 363, in connect
self.sock = conn = self._new_conn()
^^^^^^^^^^^^^^^^
File "/home/flavio/repositories/websearch-public/env/lib/python3.11/site-packages/urllib3/connection.py", line 186, in _new_conn
raise NewConnectionError(
urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPSConnection object at 0x7ff8c8bb2b50>: Failed to establish a new connection: [Errno -3] Temporary failure in name resolution
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/flavio/repositories/websearch-public/env/lib/python3.11/site-packages/requests/adapters.py", line 486, in send
resp = conn.urlopen(
^^^^^^^^^^^^^
File "/home/flavio/repositories/websearch-public/env/lib/python3.11/site-packages/urllib3/connectionpool.py", line 826, in urlopen
return self.urlopen(
^^^^^^^^^^^^^
File "/home/flavio/repositories/websearch-public/env/lib/python3.11/site-packages/urllib3/connectionpool.py", line 826, in urlopen
return self.urlopen(
^^^^^^^^^^^^^
File "/home/flavio/repositories/websearch-public/env/lib/python3.11/site-packages/urllib3/connectionpool.py", line 826, in urlopen
return self.urlopen(
^^^^^^^^^^^^^
File "/home/flavio/repositories/websearch-public/env/lib/python3.11/site-packages/urllib3/connectionpool.py", line 798, in urlopen
retries = retries.increment(
^^^^^^^^^^^^^^^^^^
File "/home/flavio/repositories/websearch-public/env/lib/python3.11/site-packages/urllib3/util/retry.py", line 592, in increment
raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='ifconfig.me', port=443): Max retries exceeded with url: / (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7ff8c8bb2b50>: Failed to establish a new connection: [Errno -3] Temporary failure in name resolution'))
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/flavio/repositories/websearch-public/env/lib/python3.11/site-packages/sirup/utils.py", line 24, in get_ip
response = session.get("https://ifconfig.me", timeout=3)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/flavio/repositories/websearch-public/env/lib/python3.11/site-packages/requests/sessions.py", line 602, in get
return self.request("GET", url, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/flavio/repositories/websearch-public/env/lib/python3.11/site-packages/requests/sessions.py", line 589, in request
resp = self.send(prep, **send_kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/flavio/repositories/websearch-public/env/lib/python3.11/site-packages/requests/sessions.py", line 703, in send
r = adapter.send(request, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/flavio/repositories/websearch-public/env/lib/python3.11/site-packages/requests/adapters.py", line 519, in send
raise ConnectionError(e, request=request)
requests.exceptions.ConnectionError: HTTPSConnectionPool(host='ifconfig.me', port=443): Max retries exceeded with url: / (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7ff8c8bb2b50>: Failed to establish a new connection: [Errno -3] Temporary failure in name resolution'))
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/flavio/repositories/websearch-public/src/get_linkedin_data.py", line 138, in <module>
rotator.rotate()
File "/home/flavio/repositories/websearch-public/env/lib/python3.11/site-packages/sirup/IPRotator.py", line 83, in rotate
self.disconnect()
File "/home/flavio/repositories/websearch-public/env/lib/python3.11/site-packages/sirup/IPRotator.py", line 77, in disconnect
self.connector.disconnect(self.pwd)
File "/home/flavio/repositories/websearch-public/env/lib/python3.11/site-packages/sirup/VPNConnector.py", line 83, in disconnect
self.current_ip = get_ip()
^^^^^^^^
File "/home/flavio/repositories/websearch-public/env/lib/python3.11/site-packages/sirup/utils.py", line 37, in get_ip
raise requests.ConnectionError("Failed to get the IP address")
requests.exceptions.ConnectionError: Failed to get the IP address
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/flavio/repositories/websearch-public/env/lib/python3.11/site-packages/urllib3/connection.py", line 174, in _new_conn
conn = connection.create_connection(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/flavio/repositories/websearch-public/env/lib/python3.11/site-packages/urllib3/util/connection.py", line 72, in create_connection
for res in socket.getaddrinfo(host, port, family, socket.SOCK_STREAM):
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/flavio/repositories/websearch-public/env/lib/python3.11/socket.py", line 962, in getaddrinfo
for res in _socket.getaddrinfo(host, port, family, type, proto, flags):
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
socket.gaierror: [Errno -3] Temporary failure in name resolution
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/flavio/repositories/websearch-public/env/lib/python3.11/site-packages/urllib3/connectionpool.py", line 714, in urlopen
httplib_response = self._make_request(
^^^^^^^^^^^^^^^^^^^
File "/home/flavio/repositories/websearch-public/env/lib/python3.11/site-packages/urllib3/connectionpool.py", line 403, in _make_request
self._validate_conn(conn)
File "/home/flavio/repositories/websearch-public/env/lib/python3.11/site-packages/urllib3/connectionpool.py", line 1053, in _validate_conn
conn.connect()
File "/home/flavio/repositories/websearch-public/env/lib/python3.11/site-packages/urllib3/connection.py", line 363, in connect
self.sock = conn = self._new_conn()
^^^^^^^^^^^^^^^^
File "/home/flavio/repositories/websearch-public/env/lib/python3.11/site-packages/urllib3/connection.py", line 186, in _new_conn
raise NewConnectionError(
urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPSConnection object at 0x7ff8c8b66c90>: Failed to establish a new connection: [Errno -3] Temporary failure in name resolution
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/flavio/repositories/websearch-public/env/lib/python3.11/site-packages/requests/adapters.py", line 486, in send
resp = conn.urlopen(
^^^^^^^^^^^^^
File "/home/flavio/repositories/websearch-public/env/lib/python3.11/site-packages/urllib3/connectionpool.py", line 826, in urlopen
return self.urlopen(
^^^^^^^^^^^^^
File "/home/flavio/repositories/websearch-public/env/lib/python3.11/site-packages/urllib3/connectionpool.py", line 826, in urlopen
return self.urlopen(
^^^^^^^^^^^^^
File "/home/flavio/repositories/websearch-public/env/lib/python3.11/site-packages/urllib3/connectionpool.py", line 826, in urlopen
return self.urlopen(
^^^^^^^^^^^^^
File "/home/flavio/repositories/websearch-public/env/lib/python3.11/site-packages/urllib3/connectionpool.py", line 798, in urlopen
retries = retries.increment(
^^^^^^^^^^^^^^^^^^
File "/home/flavio/repositories/websearch-public/env/lib/python3.11/site-packages/urllib3/util/retry.py", line 592, in increment
raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='ifconfig.me', port=443): Max retries exceeded with url: / (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7ff8c8b66c90>: Failed to establish a new connection: [Errno -3] Temporary failure in name resolution'))
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/flavio/repositories/websearch-public/env/lib/python3.11/site-packages/sirup/utils.py", line 24, in get_ip
response = session.get("https://ifconfig.me", timeout=3)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/flavio/repositories/websearch-public/env/lib/python3.11/site-packages/requests/sessions.py", line 602, in get
return self.request("GET", url, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/flavio/repositories/websearch-public/env/lib/python3.11/site-packages/requests/sessions.py", line 589, in request
resp = self.send(prep, **send_kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/flavio/repositories/websearch-public/env/lib/python3.11/site-packages/requests/sessions.py", line 703, in send
r = adapter.send(request, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/flavio/repositories/websearch-public/env/lib/python3.11/site-packages/requests/adapters.py", line 519, in send
raise ConnectionError(e, request=request)
requests.exceptions.ConnectionError: HTTPSConnectionPool(host='ifconfig.me', port=443): Max retries exceeded with url: / (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7ff8c8b66c90>: Failed to establish a new connection: [Errno -3] Temporary failure in name resolution '))
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "<frozen runpy>", line 198, in _run_module_as_main
File "<frozen runpy>", line 88, in _run_code
File "/home/flavio/repositories/websearch-public/src/get_linkedin_data.py", line 145, in <module>
rotator.connect()
File "/home/flavio/repositories/websearch-public/env/lib/python3.11/site-packages/sirup/IPRotator.py", line 52, in connect
connector = VPNConnector(self.config_queue.pop_append(), self.auth_file)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/flavio/repositories/websearch-public/env/lib/python3.11/site-packages/sirup/VPNConnector.py", line 23, in __init__
ip = get_ip()
^^^^^^^^
File "/home/flavio/repositories/websearch-public/env/lib/python3.11/site-packages/sirup/utils.py", line 37, in get_ip
raise requests.ConnectionError("Failed to get the IP address")
requests.exceptions.ConnectionError: Failed to get the IP address
3ff2072 adds the option to not fetch the IP after each connection. This is a workaround for now, particularly because one may hit the rate limit of ifconfig.me when doing a lot of re-connections. Still, the above kind of error should be dealt with appropriately.
Your repository has a workflow which lints your code after every push and when creating a pull request.
Linter workflow may fail if description
or keywords
field in setup.cfg is empty. Please update these fields. To validate your changes run:
prospector
Enabling githook will automatically lint your code in every commit. You can enable it by running the command below.
git config --local core.hooksPath .githooks
the logo and the local links to files in the repository don't work on pypi.
-> Compare to DIANNA: https://pypi.org/project/dianna/
By enabling Zenodo integration, your package will automatically get a DOI which can be used to cite your package. After enabling Zenodo integration for your GitHub repository, Zenodo will create a snapshot and archive each release you make on GitHub. Moreover, Zenodo will create a new DOI for each GitHub release of your code.
To enable Zenodo integration:
On
toggle button.10.5281/zenodo.1310751
Edit README.md and replace the badge placeholder with the badge link you copied in previous step.
The badge placeholder is shown below.
[![DOI](https://zenodo.org/badge/DOI/<replace-with-created-DOI>.svg)](https://doi.org/<replace-with-created-DOI>)
For FAQ about Zenodo please visit https://help.zenodo.org/.
It is likely that your CITATION.cff
currently doesn't pass validation. The error messages you get from the cffconvert
GitHub Action are unfortunately a bit cryptic, but doing the following helps:
given-name
and family-name
keys need updating. If your family name has a name particle like von
or van
or de
, use the name-particle
key; if your name has a suffix like Sr
or IV
, use name-suffix
. For details, refer to the schema description: https://github.com/citation-file-format/citation-file-formatorcid
key. If you do not have an orcid yet, you can get one here https://orcid.org/.date-released
using the YYYY-MM-DD format.doi
key with the conceptDOI for your repository (see https://help.zenodo.org for more information on what a conceptDOI is). If your project doesn't have a DOI yet, you can use the string 10.0000/FIXME
to pass validation.keywords
array accurately describes your project.Once you do all the steps above, the cffconvert
workflow will tell you what content it expected to see in .zenodo.json
. Copy-paste from the GitHub Action log into a new file .zenodo.json
. Afterwards, the cffconvert
GitHub Action should be green.
To help you keep the citation metadata up to date and synchronized, the cffconvert
GitHub Action checks the following 6 aspects:
Whether your repository includes a CITATION.cff
file.
By including this file, authors of the software can receive credit for the work they put in.
Whether your CITATION.cff
is valid YAML.
Visit http://www.yamllint.com/ to see if the contents of your CITATION.cff are valid YAML.
Whether your CITATION.cff
adheres to the schema (as listed in the CITATION.cff
file itself under key cff-version
).
The Citation File Format schema can be found here, along with an explanation of all the keys. You're advised to use the latest available schema version.
Whether your repository includes a .zenodo.json
file.
With this file, you can control what metadata should be associated with any future releases of your software on Zenodo: things like the author names, along with their affiliations and their ORCIDs, the license under which the software has been released, as well as the name of your software and a short description. If your repository doesn't have a .zenodo.json file, Zenodo will take a somewhat crude guess to assign these metadata.
The cffconvert
GitHub action will tell you what it expects to find in .zenodo.json
, just copy and paste it to a new file named .zenodo.json
. The suggested text ignores CITATION.cff's version
, commit
, and date-released
. cffconvert
considers these keys suspect
in the sense that they are often out of date, and there is little purpose to telling Zenodo about these properties: Zenodo already knows.
Whether .zenodo.json
is valid JSON.
Currently unimplemented, but you can check for yourself on https://jsonlint.com/.
Whether CITATION.cff
and .zenodo.json
contain equivalent data.
This final check verifies that the two files are in sync. The check ignores CITATION.cff's version
, commit
, and date-released
.
explain how to change resolv.conf and other files to make sure the IP address changes as it should
Continuous code quality can be handled by Sonarcloud. This repository is configured to use Sonarcloud to perform quality analysis and code coverage report on each push.
In order to configure Sonarcloud analysis GitHub Action workflow you must follow the steps below:
Number of days
optionSONAR_TOKEN
to secrets on GitHubYour Python package should have publicly available documentation, including API documentation for your users.
Read the Docs can host your user documentation for you.
To host the documentation of this repository please perform the following instructions:
ivory-tower-private-power/sirup
in list and press +
button.
See README.dev.md# how to build documentation site locally.
0475ad4 turns off some linting because of how the subprocess module is used. try to find a workable solution that complies with the recommendations.
Error:
Traceback (most recent call last):
File "<frozen runpy>", line 198, in _run_module_as_main
File "<frozen runpy>", line 88, in _run_code
File ".../sirup/utils.py", line 128, in kill_all_connections
subprocess.run(kill_command, input=pwd.encode(), capture_output=True, check=True)
File ".../env/lib/python3.11/subprocess.py", line 571, in run
raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['sudo', '-S', 'kill', '-15', '2469678']' returned non-zero exit status 1.
Not sure how to reproduce and fix
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.