GithubHelp home page GithubHelp logo

censys / censys-python Goto Github PK

View Code? Open in Web Editor NEW
383.0 17.0 89.0 2.37 MB

An easy-to-use and lightweight API wrapper for Censys APIs.

Home Page: https://censys-python.rtfd.io/

License: Apache License 2.0

Python 99.87% Shell 0.13%
censys python3 search pypi api python attack-surface-management hacktoberfest

censys-python's People

Contributors

bobdickinson avatar brahmlower avatar chazlever avatar d33tah avatar dadrian avatar dependabot[bot] avatar elliotcubit avatar elnappo avatar ezishiri avatar fantasquex avatar grace-murphy avatar harningt avatar jnazariocensys avatar jxdv avatar mattclement avatar mbremerkamp avatar obilodeau avatar rickydia avatar scolby33 avatar snowgears avatar thehappydinoa avatar utkonos avatar zakird avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

censys-python's Issues

censys.common.exceptions.CensysRateLimitExceededException: 429 (Too Many Requests): You have used your full quota for this billing period.

Describe the bug
I'm assuming the free account can perform "Limited queries* per month" and "Up to 1,000 results* per query" but for new fresh free account and censys CLI installation I'm getting the error below.

ricardo@zion:~$ censys view 8.8.8.8
Traceback (most recent call last):
  File "/home/ricardo/.local/bin/censys", line 8, in <module>
    sys.exit(main())
  File "/home/ricardo/.local/lib/python3.9/site-packages/censys/cli/__init__.py", line 21, in main
    args.func(args)
  File "/home/ricardo/.local/lib/python3.9/site-packages/censys/cli/commands/view.py", line 44, in cli_view
    document = index.view(args.document_id, **view_args)
  File "/home/ricardo/.local/lib/python3.9/site-packages/censys/search/v2/api.py", line 262, in view
    return self._get(self.view_path + document_id, args)["result"]
  File "/home/ricardo/.local/lib/python3.9/site-packages/censys/common/base.py", line 213, in _get
    return self._make_call(self._session.get, endpoint, args)
  File "/home/ricardo/.local/lib/python3.9/site-packages/censys/common/base.py", line 41, in _wrapper
    return _impl()
  File "/home/ricardo/.local/lib/python3.9/site-packages/backoff/_sync.py", line 94, in retry
    ret = target(*args, **kwargs)
  File "/home/ricardo/.local/lib/python3.9/site-packages/censys/common/base.py", line 39, in _impl
    return method(self, *args, *kwargs)
  File "/home/ricardo/.local/lib/python3.9/site-packages/censys/common/base.py", line 203, in _make_call
    raise censys_exception(
censys.common.exceptions.CensysRateLimitExceededException: 429 (Too Many Requests): You have used your full quota for this billing period. Please see https://censys.io/account/billing or contact [email protected].
ricardo@zion:~$ censys --version
Censys Python Version: 2.0.7
ricardo@zion:~$ 

To Reproduce
Steps to reproduce the behavior:

  1. Create a new free account
  2. Install censys CLI
  3. Run "censys view 8.8.8.8"

Expected behavior
Get the results.

Device (please complete the following information):

  • OS [e.g. macOS]: Kali
  • Censys Python Version [e.g. 1.1.0]: 2.0.7
  • Shell [e.g. bash, zsh]: Bash

Additional context
Add any other context about the problem here.

how to solve this error No module named 'censys.certificates'

No module named 'censys.certificates'
sys:1: RuntimeWarning: coroutine 'start..store' was never awaited
RuntimeWarning: Enable tracemalloc to get the object allocation traceback
during work with theHarvester
censys installed and certificates has been set

ImportError: No module named certificates

Hi, we are getting No module named certificates when using this lib.
Our script is currently running on an ubuntu box with python Python 3.8.0.

Distributor ID: Ubuntu
Description:    Ubuntu 20.04.1 LTS
Release:        20.04
Codename:       focal

To Reproduce

  1. python3 -m venv .venv
  2. source .venv/bin/activate
  3. python
  4. import censys.certificates <-- Error ModuleNotFoundError: No module named 'censys.certificates'

pip show censys

Name: censys
Version: 2.0.0
Summary: An easy-to-use and lightweight API wrapper for Censys APIs (censys.io).
Home-page: UNKNOWN
Author: Censys Team
Author-email: [email protected]
License: Apache License, Version 2.0
Location: /home/tony/expanse/.venv/lib/python3.8/site-packages
Requires: requests, backoff
Required-by:

We also check previous issues related to import errors.
Our python script is called censys_subdomain.py and we also removed pyc files as suggested here
We are following the guide provided in your docs at https://censys.io/api.
image

Support for `Fields` in bulk method

Is your feature request related to a problem? Please describe.
Let's say we have some certficates/IPs specific to an threat actor, now we want to know to what is/are common(s) to those certificates/IPs, so we can pivot based on that specific fields' values, and build a query to search for other similar certificates/IPs. It will be very helpful for quick pivoting and infrastructure hunting.

Describe the solution you'd like
I wan to get the specific fields' value while querying the certificates/IPs(if it ever gets support for that), in bulk to pivot further, based on returned field results.

Additional context
Currently, the bulk method is only available for CensysCertificates index as stated here, do you have any plan to include to it for CensysIPv4/CensysHosts?

Rate Limit

I notice we have a rate limit on our account: "api 1.0 actions/second (300.0 per 5 minute interval)"

Is this taken into account as part of this python library (is there a built in back off) and what happens if we go over this limit ? e.g. are blank records returned or is an error message is returned somewhere etc

My usage approx usage is

import censys.ipv4
ip  = "1.1.1.1"
c = censys.ipv4.CensysIPv4(api_id="blanked", api_secret="blanked")
c.view(ip)

Thanks

Diff Host Endpoint

Describe the solution you'd like
I would like to be able to access and use the diff host endpoint.

{Bug}

Hi.
Why?
Hosts
Results: 872 Time: 0.00s

image

Upgrade the Python version of dependency importlib-metadata to latest version (3.10)

Hi there! I would like make a ArchLinux package of python-censys. However I find importlib-metadata = { version = "*", python = "<3.8" } which means python-censys should be install with Python version < 3.8. ArchLinux only keeps the latest version software. So I would like to know what blocks you to upgrade the version Python installed with importlib-metadata. If there is anything I can do, I will offer help happily.

v2.0.7 breaks with Python <= 3.7

Describe the bug
The following commit introduced a dependency on Python 3.8: b1cfe43.

typing.Protocol is available only since Python 3.8 (see https://docs.python.org/3/library/typing.html#typing.Protocol)

However, censys-python package is marked as compatible with Python 3.6+.

To Reproduce
Steps to reproduce the behavior:

  1. Install censys-python
  2. import censys.common.base
File "...\site-packages\censys\common\base.py", line 6, in <module>
    from typing import Any, Callable, List, Optional, Protocol, Type
ImportError: cannot import name 'Protocol' from 'typing' (...\typing.pyc)

Expected behavior
No exceptions

Device (please complete the following information):

  • OS [e.g. macOS]: Windows 10 (2004)
  • Censys Python Version [e.g. 1.1.0]: 2.0.7
  • Shell [e.g. bash, zsh]: cmd
  • Python: 3.7.9

Add a view_all function to the Query object

This would allow users to automatically view all the hosts that show up in their search and returns a dict like:

>>> hosts = h.search("service.service_name: HTTP", per_page=5).view_all()
>>> print(hosts)
{
  "1.1.1.1": {"services": []}
}

Acceptance Criteria

  • Multithreaded

ImportError: No module named censys.certificates

$ pip show censys

Name: censys
Version: 0.0.8
Summary: Python library for interacting with Censys Search Engine (censys.io)
Home-page: https://github.com/censys/censys-python
Author: Censys Team
Author-email: [email protected]
License: Apache License, Version 2.0
Location: /usr/local/lib/python3.8/dist-packages
Requires: netaddr, requests
Required-by: 

$ python --version

Python 2.7.18rc1

pip install censys not working getting import error

ImportError: cannot import name IPv4

I'm trying to write a Python script but when I try to import the censys package and run the script I got the following error:
ImportError: cannot import name IPv4
The script is really simple:
import sys
import json
import requests
import socket
import censys
c = censys.ipv4.CensysIPv4(api_id="XXX", api_secret="XXX")
print c.search("ETH")

`setup.py` depends in censys package

Hi,

I was trying to install censys on Windows and the following error:

> python -mpip install censys
Collecting censys
  Downloading censys-0.0.8.tar.gz (10 kB)
    ERROR: Command errored out with exit status 1:
      command: 'C:\%SOME_PATH%\Python\Python.exe' -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'C:\\Users\\%USER%\\AppData\\Local\\Temp\\pip-install-k9vn638_\\censys\\setup.py'"'"'; __file__='"'"'C:\\Users\\%USER%\\AppData\\Local\\Temp\\pip-install-k9vn638_\\censys\\setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' egg_info --egg-base 'C:\Users\%USER%\AppData\Local\Temp\pip-install-k9vn638_\censys\pip-egg-info'
        cwd: C:\Users\%USER%\AppData\Local\Temp\pip-install-k9vn638_\censys\
    Complete output (5 lines):
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "C:\Users\%USER%\AppData\Local\Temp\pip-install-k9vn638_\censys\setup.py", line 4, in <module>
        import censys
    ModuleNotFoundError: No module named 'censys'
----------------------------------------
ERROR: Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.

This is using Windows embedded Python 3.7.7 with pip 20.0.2. Since the cwd is the censys package it should be in sys.path but it doesn't seem the case. Maybe because setup is in in an inline command. I'm not sure if importing the still uninstalled package is recommended.

Install worked fine on macOS.

Query multiple pages from censys results

I would like to get all IP's for my query.

Right now i use censys like this:

h = CensysHosts(api_secret=secret, api_id=uid)
            for page in h.search(query):
                print(page)
                
                    for results in page:
                        print(results["ip"])
                        ips.append(results["ip"])

But it gives me only first 100 results instead of all the censys results (like 5500).
In documentation its an example of the " # Multiple pages of search results "

Am i doing something wrong or theres a bug in the function?

Do I need to parse nextCursorToken by myself?

Gracefully handle JSON decoding errors

A certificate search call gave me a (non-reproducible, probably related to server error) JSON decoding error:

  File "/home/eric/.pyenv/versions/3.5.1/lib/python3.5/site-packages/censys/base.py", line 151, in search
    payload = self._post(self.search_path, data=data)
  File "/home/eric/.pyenv/versions/3.5.1/lib/python3.5/site-packages/censys/base.py", line 98, in _post
    return self._make_call(self._session.post, endpoint, args, data)
  File "/home/eric/.pyenv/versions/3.5.1/lib/python3.5/site-packages/censys/base.py", line 81, in _make_call
    message = res.json()["error"]
  File "/home/eric/.pyenv/versions/3.5.1/lib/python3.5/site-packages/requests/models.py", line 812, in json
    return complexjson.loads(self.text, **kwargs)
  File "/home/eric/.pyenv/versions/3.5.1/lib/python3.5/json/__init__.py", line 319, in loads
    return _default_decoder.decode(s)
  File "/home/eric/.pyenv/versions/3.5.1/lib/python3.5/json/decoder.py", line 339, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/home/eric/.pyenv/versions/3.5.1/lib/python3.5/json/decoder.py", line 357, in raw_decode
    raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

This is something that the censys-python package should catch internally. Ideally, potentially error-throwing calls should absorb any lower level exceptions internally, and then throw a well-defined exception class, or one of a set of them, defined by censys-python. That way I can use (hypothetically) except censys.CensysError: on calls to .search(), without having to choose between catching a host of lower-level errors, or just catching all errors and potentially swallowing errors in my own code or from other libraries.

Account Usage Percentage showing decimal and percent sign.

Describe the bug

CLI account command is show quota as decimal but including a percent sign too.. so it's shows 0.70% instead of 0.70 or 70%.

To Reproduce
Execute: censys account

Expected behavior
A clear and concise description of what you expected to happen.

Device (please complete the following information):

  • OS [e.g. macOS]: Linux
  • Censys Python Version: 2.1.0
  • Shell [e.g. bash, zsh]: bash

Parsing error after X responsive pages

domain-scan's gather feature queries the Censys API to collect .gov subdomains. The query parsed.subject.common_name:".gov" or parsed.extensions.subject_alt_name.dns_names:".gov" works on API pages 1-100, but errors out on any page ≥ 101:

$ ./gather censys --suffix=.gov --start=101 --end=101 --force --delay=5 --debug --parents=current-federal.csv --censys_id=<id> --censys_key=<key>

Starting new HTTPS connection (1): www.censys.io
"GET /api/v1/account HTTP/1.1" 200 245
Censys query:
parsed.subject.common_name:".gov" or parsed.extensions.subject_alt_name.dns_names:".gov"

Fetching up to 100 records, starting at page 101.
Fetching page 101.
"POST /api/v1/search/certificates HTTP/1.1" 400 116
Traceback (most recent call last):

  File "/home/hmft/Desktop/domain-scan-master/gatherers/censys.py", line 100, in gather
    certs = list(certificate_api.search(query, fields=fields, page=current_page, max_records=page_size))

  File "/usr/local/lib/python3.5/dist-packages/censys/base.py", line 151, in search
    payload = self._post(self.search_path, data=data)

  File "/usr/local/lib/python3.5/dist-packages/censys/base.py", line 98, in _post
    return self._make_call(self._session.post, endpoint, args, data)

  File "/usr/local/lib/python3.5/dist-packages/censys/base.py", line 92, in _make_call
    const=const)

censys.base.CensysException: 400 (es_transport_error): Your search was invalid and could not be parsed.

This same query on the web indicates 7,661 responsive pages:

image

...but any page ≥ 401 produces an error:

image

cc: @konklone

Exception : Only the first 1,000 search results are available through the Censys search interface

I got the exception

Enter the domain:apache.org
[+] Finding the subdomains of apache.org
Traceback (most recent call last):
File "censysearch.py", line 48, in
main(domain,censys_id,censys_secret)
File "censysearch.py", line 40, in main
subdomains = subdomain_find(domain,censys_id,censys_secret)
File "censysearch.py", line 16, in subdomain_find
for s in cert_search_results:
File "/usr/local/lib/python2.7/site-packages/censys/base.py", line 172, in search
payload = self._post(self.search_path, data=data)
File "/usr/local/lib/python2.7/site-packages/censys/base.py", line 117, in _post
return self._make_call(self._session.post, endpoint, args, data)
File "/usr/local/lib/python2.7/site-packages/censys/base.py", line 111, in _make_call
const=const)
censys.CensysException: 400 (max_results): Only the first 1,000 search results are available
through the Censys search interface. Please
upgrade your Censys account or contact
[email protected].

whois

Does whois information have API?

CLI Quota Command

Describe the solution you'd like
I would like there to be a command like $ censys quota that returns your current quota printed nicely.

Describe alternatives you've considered

$ censys quota
Email: [email protected]
Used: 2
Allowance: 250
Resets At: 12:00 am on Nov 1st, 2021

$ censys quota --json
{"used": 2, "allowance": 250, "resets_at": "2021-10-01 12:00:00"}

Unable to use Censys CLI

I copied & pasted cli command from documentation at https://censys-python.readthedocs.io/en/stable/usage-cli.html.

I can not use censys cli because I got errors if I tried to search hosts. When I tried to search ipv4, I will get empty response... How can I get data from censys API?


CLI command

censys search 'services.http.response.html_title: "Tesla Energy Powerpack"' --index-type hosts


Error message

censys search: error: argument --index-type: invalid choice: 'hosts' (choose from 'ipv4', 'certs', 'websites')


Upload latest 0.0.4 version to PyPI

Hi,
Looks like the package on PyPI for 0.0.4 (https://pypi.python.org/pypi/censys) was uploaded on 2015-11-26, but there are a number of new commits for 0.0.4 in this repo look to resolve some install issues. Would you update the package on PyPI to the current HEAD?

Here's the stack I received when installing via pip:

root@vagrant-ubuntu-trusty-64:~# pip3 install censys==0.0.4
Downloading/unpacking censys==0.0.4
  Downloading censys-0.0.4.tar.gz
  Running setup.py (path:/tmp/pip_build_root/censys/setup.py) egg_info for package censys

Requirement already satisfied (use --upgrade to upgrade): requests in /usr/local/lib/python3.4/dist-packages (from censys==0.0.4)
Downloading/unpacking netaddr (from censys==0.0.4)
  Downloading netaddr-0.7.18-py2.py3-none-any.whl (1.5MB): 1.5MB downloaded
Installing collected packages: censys, netaddr
  Running setup.py install for censys

    Installing censys script to /usr/local/bin
      File "/usr/local/lib/python3.4/dist-packages/censys/ipv4.py", line 41
        print json.dumps(self._api.view("84.206.102.184"))
                 ^
    SyntaxError: invalid syntax

      File "/usr/local/lib/python3.4/dist-packages/censys/websites.py", line 32
        print self._api.view("google.com")
                 ^
    SyntaxError: invalid syntax

      File "/usr/local/lib/python3.4/dist-packages/censys/admin/maxmind.py", line 73
        print rec
                ^
    SyntaxError: Missing parentheses in call to 'print'

      File "/usr/local/lib/python3.4/dist-packages/censys/query.py", line 53
        print self._api.get_series()
                 ^
    SyntaxError: invalid syntax

      File "/usr/local/lib/python3.4/dist-packages/censys/export.py", line 47
        print j
              ^
    SyntaxError: Missing parentheses in call to 'print'

Successfully installed censys netaddr
Cleaning up...

Quota Stats via API Query

Manually keeping track of how many queries we have used is error prone and challenging with system restarts. It would be really helpful if we could hit the API and ask for the query quota and how much we have used so that we could programmatically set our burn rate. This would still be helpful even if it counted against our queries for the month (but it would be nice if it didn't 😄 ). Thanks in advance!

I have found a way to make separate connection with the v1 api (but not v2) and get the quota info, but it would be nice not to have to have two separate connections for this functionality.

SSL error coming from censys internals

With censys 0.0.6 and requests 2.11.1, I get a super strange SSL error:

  File "[localpath]/censys.py", line 35, in gather
    certificate_api = certificates.CensysCertificates(uid, api_key)
  File "/home/eric/.pyenv/versions/3.5.2/lib/python3.5/site-packages/censys/base.py", line 112, in __init__
    CensysAPIBase.__init__(self, *args, **kwargs)
  File "/home/eric/.pyenv/versions/3.5.2/lib/python3.5/site-packages/censys/base.py", line 59, in __init__
    self.account()
  File "/home/eric/.pyenv/versions/3.5.2/lib/python3.5/site-packages/censys/base.py", line 104, in account
    return self._get("account")
  File "/home/eric/.pyenv/versions/3.5.2/lib/python3.5/site-packages/censys/base.py", line 95, in _get
    return self._make_call(self._session.get, endpoint, args)
  File "/home/eric/.pyenv/versions/3.5.2/lib/python3.5/site-packages/censys/base.py", line 76, in _make_call
    res = method(url, params=args)
  File "/home/eric/.pyenv/versions/3.5.2/lib/python3.5/site-packages/requests/sessions.py", line 487, in get
    return self.request('GET', url, **kwargs)
  File "/home/eric/.pyenv/versions/3.5.2/lib/python3.5/site-packages/requests/sessions.py", line 475, in request
    resp = self.send(prep, **send_kwargs)
  File "/home/eric/.pyenv/versions/3.5.2/lib/python3.5/site-packages/requests/sessions.py", line 585, in send
    r = adapter.send(request, **kwargs)
  File "/home/eric/.pyenv/versions/3.5.2/lib/python3.5/site-packages/requests/adapters.py", line 477, in send
    raise SSLError(e, request=request)
requests.exceptions.SSLError: EOF occurred in violation of protocol (_ssl.c:645)

I am not yet sure of the root cause, but wanted to open the issue while I had the information.

Return some payload metadata fields from search method

Right now there's no way for a client that's using this library to tell how many pages are involved in a query, and to manage pagination for itself. This would be especially helpful for large queries.

The search() method uses an iterator to return results in a streaming fashion for as long as the caller would like. I propose either:

  • a higher level method, such as query(), that does what search() does but doesn't iterate through the results, instead just returning the decoded payload (which includes metadata about pagination)
  • or, another method intended to just return the metadata for a given query. The caller would understand they are incurring an HTTP request and discarding a page of results just to get the metadata.

Quota call doesn't use proxy settings

Describe the bug
When we call CensysHost.quota() in censys.search, ignores proxy settings which was specified in CensysHosts constructor.

To Reproduce
Steps to reproduce the behavior:

  1. Install 2.0.6 and run following code.
    (Which have valid api_id, api_secret and proxy settings)
from censys.search import CensysHosts
client = CensysHosts(api_id="XXXXXXX", api_secret="XXXXXXXX", proxies={"https":"http://<proxy_ip>:<proxy_port>"})
print(client.quota())
  1. Then confirm actual TCP access destination.
$ netstat -p | grep 65247
tcp        0      1 <running server>:48232 104.26.10.245:https     SYN_SENT    65247/python 

Expected behavior
https access should head to our proxy, not 104.26.10.245.

This ip is search.censys.io.

$ nslookup search.censys.io
Name:   search.censys.io
Address: 104.26.11.245

Device (please complete the following information):

  • OS [e.g. Cent OS]: Cent OS
  • Censys Python Version [e.g. 1.1.0]: 2.0.6
  • Shell [e.g. bash, zsh]: bash

Additional context
Add any other context about the problem here.

To compare, following code which calls "view" function works as expected.
$ python test2.py

from censys.search import CensysHosts
client = CensysHosts(api_id="XXXXXXX", api_secret="XXXXXXXX", proxies={"https":"http://<proxy_ip>:<proxy_port>"})
print(client.view("104.26.11.245"))
{'ip': '104.26.11.245', 'services': [{'_decoded': 'http', '_encoding': {'banner': 'DISPLAY_UTF8', 'banner_hex': 'DISPLAY_HEX'}, 'banner': 'HTTP/1.1 403 Forbidden\nVary: Accept-Encoding\nCf-Ray: 6868f5302a3e2986-ORD\nServer: cloudflare\nX-Frame-Options: SAMEORIGIN\nExpires: Thu, 01 Jan 1970 00:00:01 GMT\nConnection:....

Bug

Hi
page 1
image

Click Next Page:

image

Censys Search CLI with jq incorrectly accessing port

hi Admin
me try code but not show port and not save txt or other file.
code:
censys search 'services.http.request.uri: /test' --index-type hosts | jq -c '.[] | {ip: .ip, port: .port}'
report site:
Hosts
Results: 340 Time: 0.54s

python try :
99
can help me fixed.

Virtual Host Search CLI

Is your feature request related to a problem? Please describe.
No

Describe the solution you'd like
I would want a optional cli argument to be able to interact with virtual hosts.

non paginated generator

Write an iterator over search results that paginates in the backend, but just presents a single iterator to users.

Getting unexpected CensysUnauthorizedException post-initialization

My credentials are accepted fine when initializing a new export API instance:

export_api = export.CensysExport(api_id=uid, api_secret=api_key)

But then actually initiating a new job causes a CensysUnauthorizedException. This line does it:

job = export_api.new_job(query, format='csv', flatten=True)

This occurs in both 0.0.6 and 0.0.7, and is relatively recent behavior without any code changes on my client's part, so I'm going to guess it's from a server-side change of some kind.

sort order

how to set sort order while "censys search 'query'" ?
by default it sorted by digits (1.235.95.2, 2.93.82.44....) asceding
in the web version : sort oder by relevance, asc, desc and random)

Specify Config File Location

Is there a way to configure the api to write the config file to a specific location. This causes issues in environments where the root is not writable.

Publish an update as 0.0.6

To use the censys library with its current (helpful) API, where a page parameter can be sent in to the main search method, I need to clone the repo and run python setup.py install rather than pip install. A version hasn't been published since April, so a 0.0.6 (which still doesn't imply any stability or backwards compatibility) would be helpful.

403 (unauthorized) with authorized account

I'm trying to use the example code for the sdk with the api-key and secret I got on my profile on censys.io. however I get the following error:

Traceback (most recent call last):
  File "main.py", line 10, in <module>
    res = c.new_job("select count(*) from certificates.certificates")
  File "/venv/lib/python2.7/site-packages/censys/query.py", line 13, in new_job
    return self._post("query", data=data)
  File "/venv/lib/python2.7/site-packages/censys/base.py", line 111, in _post
    return self._make_call(self._session.post, endpoint, args, data)
  File "/venv/lib/python2.7/site-packages/censys/base.py", line 105, in _make_call
    const=const)
censys.base.CensysUnauthorizedException: 403 (unauthorized): Unauthorized. You do not have access to this service.

Results from historical view return `CensysNotFoundException`

I'm attempting to pull historical results from the domains dataset.

from censys.data import CensysData

series = 'domain_2018'
censys = CensysData()
series_view = censys.view_series(series)
series_results = censys.view_result(series, '20190812')
...
CensysNotFoundException: 404 (unknown): The requested page does not exist.

# however when I look for '20190812' within the historical results I find a record that makes me think it should exist
[record for record in series_view['results']['historical'] if record['id'] == '20190812']

[{'timestamp': '20190813T034051',
  'id': '20190812',
  'details_url': 'https://www.censys.io/api/v1/data/domain_2018/20190812'}]

Is there a reason the historical results would include details URLs that don't exist?

Allow backoff 2.x

Is your feature request related to a problem? Please describe.

There is a dependency lock on backoff via a Poetry carot requirement. So far there has been no problems that I have observed using the new 2.x branch of backoff. The changes to backoff that warranted a major version change appear to be adding and dropping Python version support rather than API changes according to their release notes here:

https://github.com/litl/backoff/releases

Describe the solution you'd like

Change the backoff carot requirement from the following:

backoff = "^1.11.1"

To:

backoff = "^2.0.1"

Additional context

Doing a bit of blame research, the following change is the one which locked backoff to the major version via carot requirement:

https://github.com/censys/censys-python/blame/7e2faa4424d9345ae597711615152eeff2770ff6/pyproject.toml#L56

This is the commit which made this change along with a number of others:

b9dea61

Here is the only location in the code base that uses backoff:

# Wrapper to make max_retries configurable at runtime
def _backoff_wrapper(method: Callable):
@wraps(method)
def _wrapper(self, *args, **kwargs):
@backoff.on_exception(
backoff.expo,
(
CensysTooManyRequestsException,
requests.exceptions.Timeout,
),
max_tries=self.max_retries,
max_time=self.timeout,
)
def _impl():
return method(self, *args, **kwargs)
return _impl()
return _wrapper

Dependency resolver error in pip:

ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
censys 2.1.3 requires backoff<2.0.0,>=1.11.1, but you have backoff 2.0.1 which is incompatible.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.