pxmyh / helios Goto Github PK

Dependabot couldn't authenticate with https://pypi.org/simple/.

You can provide authentication details in your Dependabot dashboard by clicking into the account menu (in the top right) and selecting 'Config variables'.

View the update logs.

@PXMYH commented on Fri Oct 19 2018

requirements:

• scrape basic information

For the rental market in Vancouver (make the location variable), information to be collected:

• posted date time
• updated date time
• price
• housing type
• sqft
• structure (# of beds # of baths)
• availability
• ad body (print information) => for further text analysis

• accept URL for scraping

Installation

Using pip

pip install -U airbrake

Setup

The easiest way to get set up is with a few environment variables:

(You can find your project ID and API KEY with your project's settings):

export AIRBRAKE_API_KEY=<Your project API KEY>
export AIRBRAKE_PROJECT_ID=<Your project ID>
export AIRBRAKE_ENVIRONMENT=production

and you're done!

Otherwise, you can instantiate your AirbrakeHandler by passing these values as arguments to the getLogger() helper:

import airbrake


logger = airbrake.getLogger(api_key="<Your project API KEY>", project_id=<Your project ID>)

try:
    1/0
except Exception:
    logger.exception("Bad math.")

For more information please visit our official GitHub repo.

Dependabot can't resolve your Python dependency files.

As a result, Dependabot couldn't update your dependencies.

The error Dependabot encountered was:

pipenv.patched.notpip._internal.exceptions.UnsupportedPythonVersion: pyrsistent requires Python '>=3.5' but the running Python is 2.7.18

If you think the above is an error on Dependabot's side please don't hesitate to get in touch - we'll do whatever we can to fix it.

View the update logs.

Dependabot can't resolve your Python dependency files.

As a result, Dependabot couldn't update your dependencies.

The error Dependabot encountered was:

pipenv.patched.notpip._internal.exceptions.UnsupportedPythonVersion: pyrsistent requires Python '>=3.5' but the running Python is 2.7.18

If you think the above is an error on Dependabot's side please don't hesitate to get in touch - we'll do whatever we can to fix it.

View the update logs.

@PXMYH commented on Fri Oct 19 2018

Currently to add a new data source we need to create a new spider and properly configure it, this leads to snowflake configuration and lots of duplicate code base. Have a common service responsible for creating/adding/configuring bots.

Dependabot can't resolve your Python dependency files.

As a result, Dependabot couldn't update your dependencies.

The error Dependabot encountered was:

pipenv.patched.notpip._internal.exceptions.UnsupportedPythonVersion: pyrsistent requires Python '>=3.5' but the running Python is 2.7.18

If you think the above is an error on Dependabot's side please don't hesitate to get in touch - we'll do whatever we can to fix it.

View the update logs.

Dependabot can't resolve your Python dependency files.

As a result, Dependabot couldn't update your dependencies.

The error Dependabot encountered was:

pipenv.patched.notpip._internal.exceptions.UnsupportedPythonVersion: pyrsistent requires Python '>=3.5' but the running Python is 2.7.18

If you think the above is an error on Dependabot's side please don't hesitate to get in touch - we'll do whatever we can to fix it.

View the update logs.

Dependabot can't resolve your Python dependency files.

As a result, Dependabot couldn't update your dependencies.

The error Dependabot encountered was:

pipenv.patched.notpip._internal.exceptions.UnsupportedPythonVersion: pyrsistent requires Python '>=3.5' but the running Python is 2.7.18

If you think the above is an error on Dependabot's side please don't hesitate to get in touch - we'll do whatever we can to fix it.

View the update logs.

Need to have the ability to fetch data from multiple sources and avoid convoluted into one repo. \

Data source plugin:

• craigslist
• mls
• zumper
• zoocasa
  etc.

Reference: concourse resource framework

With Swagger to build up API documentation:
https://swagger.io/

Dependabot can't resolve your Python dependency files.

As a result, Dependabot couldn't update your dependencies.

The error Dependabot encountered was:

pipenv.patched.notpip._internal.exceptions.UnsupportedPythonVersion: pyrsistent requires Python '>=3.5' but the running Python is 2.7.18

If you think the above is an error on Dependabot's side please don't hesitate to get in touch - we'll do whatever we can to fix it.

View the update logs.

basic operations to read/write/search Cassandra database

text count
wording
speech analysis
etc.

Dependabot can't resolve your Python dependency files.

As a result, Dependabot couldn't update your dependencies.

The error Dependabot encountered was:

pipenv.patched.notpip._internal.exceptions.UnsupportedPythonVersion: pyrsistent requires Python '>=3.5' but the running Python is 2.7.18

If you think the above is an error on Dependabot's side please don't hesitate to get in touch - we'll do whatever we can to fix it.

View the update logs.

Save fetched rental market data into postgres for record keeping and later displaying purpose

Dependabot can't resolve your Python dependency files.

As a result, Dependabot couldn't update your dependencies.

The error Dependabot encountered was:

pipenv.patched.notpip._internal.exceptions.UnsupportedPythonVersion: fastavro requires Python '>=3.6' but the running Python is 2.7.18

If you think the above is an error on Dependabot's side please don't hesitate to get in touch - we'll do whatever we can to fix it.

View the update logs.

Bootstrap flask framework:

• run flask app
• can invoke scrapy spider

Intent is to use docker config create to create a docker configuration and provide it to Cassandra cluster running in Docker Swarm. Steps as follows:

docker config create cassandra.yaml ./res/cassandra.yaml
docker service create --replicas 1 --name cassandra_cluster --config source=cassandra.yaml,target=/etc/cassandra/cassandra.yaml,mode=644 -d cassandra:3.11.3

However, after feeding config to cluster, getting error:

[atlantis@kali cassandra]$ docker service logs cassandra_cluster
cassandra_cluster.1.scgjmt4g17kk@kali    | chown: changing ownership of '/etc/cassandra/cassandra.yaml': Read-only file system
cassandra_cluster.1.nr7entee1ahm@kali    | chown: changing ownership of '/etc/cassandra/cassandra.yaml': Read-only file system
cassandra_cluster.1.w1h0b6lwfd0s@kali    | chown: changing ownership of '/etc/cassandra/cassandra.yaml': Read-only file system
cassandra_cluster.1.qgjzujcwsokz@kali    | chown: changing ownership of '/etc/cassandra/cassandra.yaml': Read-only file system
[atlantis@kali cassandra]$ 

Reason being Cassandra locked permission to read only and docker config can't overwrite the file.

Possible solution:
instruct Cassandra instead of looking for config in default location /etc/cassandra, it can find config under /cassandra.yaml which is default docker config loading behavior

• scrape the website periodically
• run as background job

@PXMYH commented on Fri Oct 19 2018

@PXMYH commented on Fri Oct 19 2018