fabric8-analytics / fabric8-analytics-firehose-fetcher Goto Github PK
View Code? Open in Web Editor NEW[DEPRECATED] Simple service, which reads libraries.io firehose-stream and schedules analysis for released packages.
fabric8-analytics-firehose-fetcher's People
Contributors
Stargazers
Watchers
fabric8-analytics-firehose-fetcher's Issues
Report only ecosystems that we currently support
I can see log messages that report rubygems updates:
2017-09-11 21:17:25,241 - f8a_firehose_fetcher.firehose_fetcher - INFO - Received event for EPV: 'rubygems':'websocket-driver':'0.7.0'
I haven't checked the source code, but hopefully we are not scheduling analyses for rubygems (we do not support them yet). We should suppress these messages since they are confusing.
ChunkedEncodingError: ('Connection broken: IncompleteRead(0 bytes read)', IncompleteRead(0 bytes read))
2017-08-25 07:59:58,053 - f8a_firehose_fetcher.firehose_fetcher - INFO - Received event for EPV: 'npm':'quesbook-component':'0.0.83'
Traceback (most recent call last):
File "/usr/lib/python3.4/site-packages/urllib3/response.py", line 539, in _update_chunk_length
self.chunk_left = int(line, 16)
ValueError: invalid literal for int() with base 16: b''
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/lib/python3.4/site-packages/urllib3/response.py", line 302, in _error_catcher
yield
File "/usr/lib/python3.4/site-packages/urllib3/response.py", line 594, in read_chunked
self._update_chunk_length()
File "/usr/lib/python3.4/site-packages/urllib3/response.py", line 543, in _update_chunk_length
raise httplib.IncompleteRead(line)
http.client.IncompleteRead: IncompleteRead(0 bytes read)
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/lib/python3.4/site-packages/requests/models.py", line 747, in generate
for chunk in self.raw.stream(chunk_size, decode_content=True):
File "/usr/lib/python3.4/site-packages/urllib3/response.py", line 432, in stream
for line in self.read_chunked(amt, decode_content=decode_content):
File "/usr/lib/python3.4/site-packages/urllib3/response.py", line 622, in read_chunked
self._original_response.close()
File "/usr/lib64/python3.4/contextlib.py", line 77, in __exit__
self.gen.throw(type, value, traceback)
File "/usr/lib/python3.4/site-packages/urllib3/response.py", line 320, in _error_catcher
raise ProtocolError('Connection broken: %r' % e, e)
urllib3.exceptions.ProtocolError: ('Connection broken: IncompleteRead(0 bytes read)', IncompleteRead(0 bytes read))
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/bin/cli", line 9, in <module>
load_entry_point('fabric8-analytics-firehose-fetcher==0.1', 'console_scripts', 'cli')()
File "/usr/lib/python3.4/site-packages/click/core.py", line 722, in __call__
return self.main(*args, **kwargs)
File "/usr/lib/python3.4/site-packages/click/core.py", line 697, in main
rv = self.invoke(ctx)
File "/usr/lib/python3.4/site-packages/click/core.py", line 895, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/usr/lib/python3.4/site-packages/click/core.py", line 535, in invoke
return callback(*args, **kwargs)
File "/usr/bin/fetcher_cli.py", line 16, in cli
fetcher.run()
File "/usr/lib/python3.4/site-packages/f8a_firehose_fetcher/firehose_fetcher.py", line 96, in run
for event in client.events():
File "/usr/lib/python3.4/site-packages/sseclient/__init__.py", line 58, in events
for chunk in self._read():
File "/usr/lib/python3.4/site-packages/sseclient/__init__.py", line 48, in _read
for chunk in self._event_source:
File "/usr/lib/python3.4/site-packages/requests/models.py", line 750, in generate
raise ChunkedEncodingError(e)
requests.exceptions.ChunkedEncodingError: ('Connection broken: IncompleteRead(0 bytes read)', IncompleteRead(0 bytes read))
NewConnectionError
Maven' failed: HTTPSConnectionPool(host='mvnrepository.com', port=443): Max retries exceeded with url: /artifact/org.glassfish.tyrus/websocket-osgi (Caused by NewConnectionEr
ror('<urllib3.connection.VerifiedHTTPSConnection object at 0x7ff8e9aad668>: Failed to establish a new connection: [Errno 113] No route to host',))
Traceback (most recent call last):
File "/root/fabric8-analytics-tagger/venv/lib/python3.6/site-packages/urllib3/connection.py", line 141, in _new_conn
(self.host, self.port), self.timeout, **extra_kw)
File "/root/fabric8-analytics-tagger/venv/lib/python3.6/site-packages/urllib3/util/connection.py", line 83, in create_connection
raise err
File "/root/fabric8-analytics-tagger/venv/lib/python3.6/site-packages/urllib3/util/connection.py", line 73, in create_connection
sock.connect(sa)
OSError: [Errno 113] No route to host
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/root/fabric8-analytics-tagger/venv/lib/python3.6/site-packages/urllib3/connectionpool.py", line 601, in urlopen
chunked=chunked)
File "/root/fabric8-analytics-tagger/venv/lib/python3.6/site-packages/urllib3/connectionpool.py", line 346, in _make_request
self._validate_conn(conn)
File "/root/fabric8-analytics-tagger/venv/lib/python3.6/site-packages/urllib3/connectionpool.py", line 850, in _validate_conn
conn.connect()
File "/root/fabric8-analytics-tagger/venv/lib/python3.6/site-packages/urllib3/connection.py", line 284, in connect
conn = self._new_conn()
File "/root/fabric8-analytics-tagger/venv/lib/python3.6/site-packages/urllib3/connection.py", line 150, in _new_conn
self, "Failed to establish a new connection: %s" % e)
urllib3.exceptions.NewConnectionError: <urllib3.connection.VerifiedHTTPSConnection object at 0x7ff8e9aad668>: Failed to establish a new connection: [Errno 113] No route to h$
st
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/root/fabric8-analytics-tagger/venv/lib/python3.6/site-packages/requests/adapters.py", line 440, in send
timeout=timeout
File "/root/fabric8-analytics-tagger/venv/lib/python3.6/site-packages/urllib3/connectionpool.py", line 639, in urlopen
_stacktrace=sys.exc_info()[2])
File "/root/fabric8-analytics-tagger/venv/lib/python3.6/site-packages/urllib3/util/retry.py", line 388, in increment
raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='mvnrepository.com', port=443): Max retries exceeded with url: /artifact/org.glassfish.tyrus/websocket-osgi (Cause$ by NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0x7ff8e9aad668>: Failed to establish a new connection: [Errno 113] No route to host',))
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/root/fabric8-analytics-tagger/venv/lib/python3.6/site-packages/f8a_tagger-0.1-py3.6.egg/f8a_tagger/recipes.py", line 195, in collect
keywords_set.union(collector_instance.execute(ignore_errors, use_progressbar))
File "/root/fabric8-analytics-tagger/venv/lib/python3.6/site-packages/f8a_tagger-0.1-py3.6.egg/f8a_tagger/collectors/maven.py", line 48, in execute
response = get(self._MVNREPOSITORY_URL + package_name)
File "/root/fabric8-analytics-tagger/venv/lib/python3.6/site-packages/requests/api.py", line 72, in get
return request('get', url, params=params, **kwargs)
File "/root/fabric8-analytics-tagger/venv/lib/python3.6/site-packages/requests/api.py", line 58, in request
return session.request(method=method, url=url, **kwargs)
File "/root/fabric8-analytics-tagger/venv/lib/python3.6/site-packages/requests/sessions.py", line 508, in request
resp = self.send(prep, **send_kwargs)
File "/root/fabric8-analytics-tagger/venv/lib/python3.6/site-packages/requests/sessions.py", line 618, in send
r = adapter.send(request, **kwargs)
File "/root/fabric8-analytics-tagger/venv/lib/python3.6/site-packages/requests/adapters.py", line 508, in send
raise ConnectionError(e, request=request)
requests.exceptions.ConnectionError: HTTPSConnectionPool(host='mvnrepository.com', port=443): Max retries exceeded with url: /artifact/org.glassfish.tyrus/websocket-osgi (Ca```
ENABLE_SCHEDULING parameter missing in OS template
ENABLE_SCHEDULING parameter needs to be listed here: https://github.com/fabric8-analytics/fabric8-analytics-firehose-fetcher/blob/master/openshift/template.yaml#L120-L143
Otherwise the value cannot be configured during deployment.
Get rid of unnecessary dependencies from worker
At this moment we download google-cloud-visio or numpy as worker dependencies which are not required. Because of this docker image is too big.
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.