docker-compose / cache / queues about hoaxy-backend

[chathuriw]

Thank you for your contribution @lucmski. I will check the docker files and let you know.

[filmenczer]

Working on Docker (#26). In the meanwhile, an AMI is available for EC2 (see https://github.com/IUNetSci/hoaxy-backend#hoaxy-backend-as-aws-image-ami).

Not sure about other questions...

[lucmski]

I meant is it possible to distribute a list of twitter usernames to fetch and get their botometer score async like with payloads managed by rabbitmq or celery. Globally, it is more about handling more data and being able to resume tasks. (get the last tweet id scraped so it prevents using resources).

[lucmski]

Example for a front/server dockerfile (alpine/nodejs v11.x):
https://gist.github.com/lucmski/b4a143b0bf026448f159e6bf463ed0c2

Example for hoaxy-backend (alpine 3.9/mininconda2):
https://gist.github.com/lucmski/87f13b669422c2f057b500404a01d359

If needed, I can do a miniconda3...

[chathuriw]

@lucmski, I'm trying the docker file for hoaxy-backend. I was able to follow first three steps. I'm stucked with step 4 (https://gist.github.com/lucmski/87f13b669422c2f057b500404a01d359#run-docker-container).
I think there is a typo in the first command in that step. (docker -ti --rm lucmichalskip/hoaxy-backend:alpine-miniconda2-jdk8 hoaxy -h)

I'm getting this error. "unknown shorthand flag: 't' in -ti"

[lucmski]

forgot to add 'run' command sorry, the gist was updated

[chathuriw]

@lucmski Can you share the Dockerfile lucmichalskip/hoaxy-backend:alpine-miniconda2-jdk8 with us. I'm trying to follow your instructions again and got an error in docker run command. I think it will be easier to troubleshoot if I can see the Dockerfile.

chathuri@chathuri-Latitude-E7450:~/IUNI/hoaxy-backend$ docker run -ti --rm lucmichalskip/hoaxy-backend:alpine-miniconda2-jdk8 hoaxy -h
/opt/conda/lib/python2.7/site-packages/hoaxy-0.1.0-py2.7.egg/hoaxy/init.py:30: YAMLLoadWarning: calling yaml.load() without Loader=... is deprecated, as the default Loader is unsafe. Please read https://msg.pyyaml.org/load for full details.
Traceback (most recent call last):
File "/opt/conda/bin/hoaxy", line 11, in
load_entry_point('hoaxy==0.1.0', 'console_scripts', 'hoaxy')()
File "build/bdist.linux-x86_64/egg/hoaxy/commands/cmdline.py", line 80, in main
File "build/bdist.linux-x86_64/egg/hoaxy/utils/init.py", line 20, in list_cls_under_mod
File "/opt/conda/lib/python2.7/importlib/init.py", line 37, in import_module
import(name)
File "build/bdist.linux-x86_64/egg/hoaxy/commands/crawl.py", line 12, in
File "build/bdist.linux-x86_64/egg/hoaxy/crawl/spiders/article.py", line 11, in
File "build/bdist.linux-x86_64/egg/hoaxy/database/init.py", line 27, in
File "/opt/conda/lib/python2.7/site-packages/sqlalchemy/engine/init.py", line 423, in create_engine
return strategy.create(*args, **kwargs)
File "/opt/conda/lib/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 87, in create
dbapi = dialect_cls.dbapi(**dbapi_args)
File "/opt/conda/lib/python2.7/site-packages/sqlalchemy/dialects/postgresql/psycopg2.py", line 605, in dbapi
import psycopg2
File "/opt/conda/lib/python2.7/site-packages/psycopg2/init.py", line 50, in
from psycopg2._psycopg import BINARY, NUMBER, STRING, DATETIME, ROWID
ImportError: libssl.so.1.0.0: cannot open shared object file: No such file or directory

[lucmski]

Hi,

Sounds that even my dockerfile is outdated; i need to update it.
Why do not you split into separate service with a docker-compose ?

My telegram is @deepocrates for faster communication.

Cheers,
L

[roscopecoltran]

From fork, https://github.com/lucmski/hoaxy-backend/tree/dockerize-p3/docker

please try:

git clone https://github.com/lucmski/hoaxy-backend
cd hoaxy-backend/docker
docker build -t hoaxy .
docker run -ti --rm hoaxy -h

[lucmski]

should be interesting you https://github.com/twintproject/twint-search

[chathuriw]

@roscopecoltran, @deepocrates I was able to build docker for hoaxy backend successfully with your instructions. Thank you for the contribution.

But when I run hoaxy -h, it gives me this error.

(base) chathuri@chathuri-Latitude-E7450:~/IUNI/hoaxy/lucmski/hoaxy-backend/docker$ docker run -ti --rm hoaxy -h
/opt/conda/lib/python3.7/site-packages/psycopg2-2.7.6.1-py3.7-linux-x86_64.egg/psycopg2/init.py:144: UserWarning: The psycopg2 wheel package will be renamed from release 2.8; in order to keep installing from binary please use "pip install psycopg2-binary" instead. For details see: http://initd.org/psycopg/docs/install.html#binary-install-from-pypi.
""")

**************************** ERROR ******************************************
NO HOAXY CONFIGURATION FILE FOUND: (DEFAULT '/home/.hoaxy/conf.yaml')!

(1) If you have put the conf.yaml to somewhere other than default location
'/root/.hoaxy/',
please set your enviroment variable HOAXY_HOME properly.
(2) Use hoaxy config [--home=] to get a sample of configuration file
named conf.sample.yaml
(3) Edit and rename conf.sample.yaml to conf.yaml. And you should edit and
rename other sample files.

Currently, hoaxy uses default settings and may not work properly.

---

Usage:
hoaxy [options] [...]
hoaxy -h | --help
hoaxy -v | --version

Subcommands are:
config Generate sample of configuration files
crawl Crawl news sites
init Initialize DB tables and load site data if exists
site News site management: add, alter and dump
lucene Lucene Indexing and Searching
report Report and generate statistics of hoaxy collection
sns Online social network services management

For subcommands, use hoaxy <command> --help to get the usage of the
subcommand !

Global options are:
--help Show help for this subcommand.
--console-log-level= Set logging level for console output.
[default: debug]

[chathuriw]

@lucmski merged the docker changes to master. Thank you for your contribution.