matteoredaelli / ebot Goto Github PK

For better scalability, ebot should be a real scalable NOSQL database
Couchdb is more a replicated than a distribuited NOSQL (document) database...

Switching to riak? apache cassandra? Disco?

now ebot works only with a couchdb server running in localhost
hostname, user and password should be put in priv/ebot_db.conf

cd ebot
while [ 1 ] ; do
curl http://localhost:8000/worker/html/check_recover
curl http://localhost:8000/worker/web/check_recover
./utils/update-rrd.sh; sleep 300
done

                <<"http://www.mingw.org/docs.shtml">>}}

=ERROR REPORT==== 8-May-2010::20:12:48 ===
Error in process <0.119.0> with exit value: {{case_clause,<<10468 bytes>>},[{mochiweb_html,find_qgt,2},{mochiweb_html,tokenize,2},{mochiweb_html,tokens,3},{ebot_html_util,get_links,2},{ebot_html_util,get_links,2},{ebot_web,fetch_url_links,2},{ebot_web,analyze_url_body,2},{ebot_web...

ebot_web:fetch_url_links( <<"http://www.uscaratese.it/">>).

{ok,[<<"http://www.uscaratese.it/">>,
<<"http://www.uscaratese.it/Javascript: openSearchHelp253();">>,

By default urls are oushed to a FIFO amqp queue..

But sometimes it would be useful to submit and immediately analyze some urls...

Now urls are old if older than one day

I suppose there is a need to change mochiweb html parsing to something else as at present state there are memory leaks in mochiweb code somewhere and during ebot run process first hangs and then goes down with "Can't allocate memory" error.

now ebot works only with a amqp server running in localhost
hostname, user and password should be put in priv/ebot_amqp.conf

it would be usefull adding at configuration level any custom function to be executed when a body page is visited. the output could be a key/value list that wll be added in the database using ebot_db:update_url(Url, Options)

<<"NOT_ALLOWED - cannot redeclare exchange 'EBOT' in vhost '/' with different type, durable or autodelete value">>}

=ERROR REPORT==== 18-Sep-2010::14:42:28 ===
** Generic server <0.86.0> terminating
** Last message in was {'EXIT',<0.91.0>,socket_closed}
** When Server state == {nc_state,
{amqp_params,<<"guest">>,<<"guest">>,<<"/">>,
"localhost",5672,0,0,0,none,[]},
#Port<0.2815>,<0.91.0>,<0.90.0>,<0.89.0>,0,0,
{nc_closing,server_initiated_close,
{'connection.close',530,
<<"NOT_ALLOWED - cannot redeclare exchange 'EBOT' in vhost '/' with different type, durable or autodelete value">>,
40,10},
none,wait_socket_close},

4> ebot_web:fetch_url_links(<<"http://www.adobe.com/go/getflashplayer">>).
Error: no_scheme{error,no_scheme}

removing options embedded in urls. maybe adding url rewrites regexps?

http://www.gettyre.it/motoweb/login_input.action;jsessionid=0D6C52EE922

now also internal referrals are tracked.

Why not integrating ebot logs with mochiweb/webmachine logs files?

it could be useful to permit the user to run a custom function for normalizing urls, couldn't it?

maybe with client merle (http://github.com/joewilliams/merle)

urls like http://binarioperlafrica.wordpress.com/2010/01/31/416/#comment-349
should be truncated to http://binarioperlafrica.wordpress.com/2010/01/31/416/

ebot_web:fetch_url_links( <<"http://www.redaelli.org/matteo/">>).
{ok,[]}

Hi there,

I successfully compile ebot and run it. but nothing happen except seeing endless INFO REPORT as following. So, could you tell how to use it? thank you!

=INFO REPORT==== 14-Jul-2014::19:46:34 ===
{ebot_web,194,{run,no_queued_urls,waiting}}

=INFO REPORT==== 14-Jul-2014::19:46:34 ===
{ebot_web,194,{run,no_queued_urls,waiting}}

=INFO REPORT==== 14-Jul-2014::19:46:34 ===
{ebot_web,194,{run,no_queued_urls,waiting}}

=INFO REPORT==== 14-Jul-2014::19:46:34 ===
{ebot_web,194,{run,no_queued_urls,waiting}}

=INFO REPORT==== 14-Jul-2014::19:46:34 ===
{ebot_web,194,{run,no_queued_urls,waiting}}

=INFO REPORT==== 14-Jul-2014::19:46:34 ===
{ebot_web,194,{run,no_queued_urls,waiting}}

=INFO REPORT==== 14-Jul-2014::19:46:34 ===
{ebot_web,194,{run,no_queued_urls,waiting}}

=INFO REPORT==== 14-Jul-2014::19:46:34 ===
{ebot_web,194,{run,no_queued_urls,waiting}}

User-Mac-Pro:ebot Saber$ ./start.sh
Erlang R15B02 (erts-5.9.2) [source] [64-bit] [smp:2:2] [async-threads:0] [hipe] [kernel-poll:false]

Eshell V5.9.2 (abort with ^G)
1> ** Found 0 name clashes in code paths
{"init terminating in do_boot",{{case_clause,{error,{not_started,mochiweb}}},[{ebot,ensure_started,1,[{file,"src/ebot.erl"},{line,28}]},{ebot,start,0,[{file,"src/ebot.erl"},{line,40}]},{init,start_it,1,[]},{init,start_em,1,[]}]}}

Crash dump was written to: erl_crash.dump
init terminating in do_boot ()

it would be useful to reload crawlers options without restarting ebot...