stangirard / seo-audits-toolkit Goto Github PK
View Code? Open in Web Editor NEWSEO & Security Audit for Websites. Lighthouse & Security Headers crawler, Sitemap/Keywords/Images Extractor, Summarizer, etc ...
SEO & Security Audit for Websites. Lighthouse & Security Headers crawler, Sitemap/Keywords/Images Extractor, Summarizer, etc ...
Hi guys,
I'm unable to install the tool due to following errors:
warning lighthouse > raven > [email protected]: Please upgrade to version 7 or higher. Older versions may use Math.random() in certain circumstances, which is known to be problematic. See https://v8.dev/blog/math-random for details.
warning lighthouse > intl-messageformat > [email protected]: We've written a new parser that's 6x faster and is backwards compatible. Please use @formatjs/icu-messageformat-parser
error [email protected]: The engine "node" is incompatible with this module. Expected version ">=14.15". Got "12.22.10"
error Found incompatible module.
info Visit https://yarnpkg.com/en/docs/cli/global for documentation about this command.
The command '/bin/sh -c yarn global add lighthouse' returned a non-zero code: 1
ERROR: Service 'beats' failed to build : Build failed
I've followed all the steps except:
docker-compose pull ## docker-compose build if you want to build it on your own
Starting the project "as is", without modifications to the admin module/part, I get a blank screen on http://localhost:3000/
react-dom.production.min.js:216 TypeError: Cannot read properties of null (reading 'breakpoints')
at Button.js:48:22
at c (useMediaQuery.js:19:50)
at b (Button.js:47:20)
at oi (react-dom.production.min.js:157:137)
at Wc (react-dom.production.min.js:267:460)
at kl (react-dom.production.min.js:250:347)
at Sl (react-dom.production.min.js:250:278)
at xl (react-dom.production.min.js:250:138)
at bl (react-dom.production.min.js:243:163)
at fl (react-dom.production.min.js:237:175)
cc @ react-dom.production.min.js:216
react-dom.production.min.js:216 TypeError: Cannot read properties of null (reading 'breakpoints')
at AppBar.js:78:22
at c (useMediaQuery.js:19:50)
at Z (AppBar.js:77:20)
at oi (react-dom.production.min.js:157:137)
at Wc (react-dom.production.min.js:267:460)
at kl (react-dom.production.min.js:250:347)
at Sl (react-dom.production.min.js:250:278)
at xl (react-dom.production.min.js:250:138)
at bl (react-dom.production.min.js:243:163)
at fl (react-dom.production.min.js:237:175)
cc @ react-dom.production.min.js:216
react-dom.production.min.js:216 TypeError: Cannot read properties of null (reading 'breakpoints')
at Sidebar.js:36:22
at c (useMediaQuery.js:19:50)
at _ (Sidebar.js:35:20)
at oi (react-dom.production.min.js:157:137)
at Wc (react-dom.production.min.js:267:460)
at kl (react-dom.production.min.js:250:347)
at Sl (react-dom.production.min.js:250:278)
at xl (react-dom.production.min.js:250:138)
at bl (react-dom.production.min.js:243:163)
at fl (react-dom.production.min.js:237:175)
cc @ react-dom.production.min.js:216
react-dom.production.min.js:216 TypeError: Cannot read properties of null (reading 'breakpoints')
at Dashboard.js:31:15
at c (useMediaQuery.js:19:50)
at Mt (Dashboard.js:30:22)
at oi (react-dom.production.min.js:157:137)
at Wc (react-dom.production.min.js:267:460)
at kl (react-dom.production.min.js:250:347)
at Sl (react-dom.production.min.js:250:278)
at xl (react-dom.production.min.js:250:138)
at bl (react-dom.production.min.js:243:163)
at fl (react-dom.production.min.js:237:175)
cc @ react-dom.production.min.js:216
react-dom.production.min.js:216 TypeError: Cannot read properties of null (reading 'breakpoints')
at Button.js:48:22
at c (useMediaQuery.js:19:50)
at b (Button.js:47:20)
at oi (react-dom.production.min.js:157:137)
at Wc (react-dom.production.min.js:267:460)
at kl (react-dom.production.min.js:250:347)
at Sl (react-dom.production.min.js:250:278)
at xl (react-dom.production.min.js:250:138)
at bl (react-dom.production.min.js:243:163)
at react-dom.production.min.js:123:115
cc @ react-dom.production.min.js:216
react-dom.production.min.js:216 TypeError: Cannot read properties of null (reading 'breakpoints')
at AppBar.js:78:22
at c (useMediaQuery.js:19:50)
at Z (AppBar.js:77:20)
at oi (react-dom.production.min.js:157:137)
at Wc (react-dom.production.min.js:267:460)
at kl (react-dom.production.min.js:250:347)
at Sl (react-dom.production.min.js:250:278)
at xl (react-dom.production.min.js:250:138)
at bl (react-dom.production.min.js:243:163)
at react-dom.production.min.js:123:115
cc @ react-dom.production.min.js:216
react-dom.production.min.js:216 TypeError: Cannot read properties of null (reading 'breakpoints')
at Sidebar.js:36:22
at c (useMediaQuery.js:19:50)
at _ (Sidebar.js:35:20)
at oi (react-dom.production.min.js:157:137)
at Wc (react-dom.production.min.js:267:460)
at kl (react-dom.production.min.js:250:347)
at Sl (react-dom.production.min.js:250:278)
at xl (react-dom.production.min.js:250:138)
at bl (react-dom.production.min.js:243:163)
at react-dom.production.min.js:123:115
cc @ react-dom.production.min.js:216
react-dom.production.min.js:123 Uncaught TypeError: Cannot read properties of null (reading 'breakpoints')
at Button.js:48:22
at c (useMediaQuery.js:19:50)
at b (Button.js:47:20)
at oi (react-dom.production.min.js:157:137)
at Wc (react-dom.production.min.js:267:460)
at kl (react-dom.production.min.js:250:347)
at Sl (react-dom.production.min.js:250:278)
at xl (react-dom.production.min.js:250:138)
at bl (react-dom.production.min.js:243:163)
at react-dom.production.min.js:123:115
I tried using the Dockerfile.dev
under admin
instead of Dockerfile
(also editing the docker-compose.yml
to fit the changes), and now I see a page with Network error
when I try to login, but it's much better !
when I visit web app page at http://localhost:3000 it redirects to login address http://localhost:3000/#/login
and I expected to be able to login with userpass admin/admin but I get "Network Error" toast message
What should I do to get to admin panel?
Following step by step tutorial I will always get the network error. Logs are empty (there is no error logged on dashboard/server).
I have tried adding new user but it is the same issue.
I have tried to add:
`# General
NAMESPACE=sas
SERVER_PORT=5301
FRONT_PORT=5302
JSON_URL=""
SERVER_URL=http://IP:5301
FRONTEND_URL=http://IP:5302
SCAN_DEFAULT_MAX_REQUESTS=0
YAKE_SERVER_URL=http://yake:5000/yake/
With the IP of the server, but it doesn't work either.
How can I setup it to access it remotely ???
Hi
Steps to reproduce :
mv .env-example .env
Traceback (most recent call last):
File "manage.py", line 22, in <module>
main()
File "manage.py", line 18, in main
execute_from_command_line(sys.argv)
File "/opt/venv/lib/python3.8/site-packages/django/core/management/__init__.py", line 401, in execute_from_command_line
utility.execute()
File "/opt/venv/lib/python3.8/site-packages/django/core/management/__init__.py", line 395, in execute
self.fetch_command(subcommand).run_from_argv(self.argv)
File "/opt/venv/lib/python3.8/site-packages/django/core/management/base.py", line 330, in run_from_argv
self.execute(*args, **cmd_options)
File "/opt/venv/lib/python3.8/site-packages/django/core/management/base.py", line 371, in execute
output = self.handle(*args, **options)
File "/opt/venv/lib/python3.8/site-packages/django/core/management/commands/loaddata.py", line 72, in handle
self.loaddata(fixture_labels)
File "/opt/venv/lib/python3.8/site-packages/django/core/management/commands/loaddata.py", line 114, in loaddata
self.load_label(fixture_label)
File "/opt/venv/lib/python3.8/site-packages/django/core/management/commands/loaddata.py", line 181, in load_label
obj.save(using=self.using)
File "/opt/venv/lib/python3.8/site-packages/django/core/serializers/base.py", line 223, in save
models.Model.save_base(self.object, using=using, raw=True, **kwargs)
File "/opt/venv/lib/python3.8/site-packages/django/db/models/base.py", line 790, in save_base
updated = self._save_table(
File "/opt/venv/lib/python3.8/site-packages/django/db/models/base.py", line 872, in _save_table
updated = self._do_update(base_qs, using, pk_val, values, update_fields,
File "/opt/venv/lib/python3.8/site-packages/django/db/models/base.py", line 926, in _do_update
return filtered._update(values) > 0
File "/opt/venv/lib/python3.8/site-packages/django/db/models/query.py", line 803, in _update
return query.get_compiler(self.db).execute_sql(CURSOR)
File "/opt/venv/lib/python3.8/site-packages/django/db/models/sql/compiler.py", line 1522, in execute_sql
cursor = super().execute_sql(result_type)
File "/opt/venv/lib/python3.8/site-packages/django/db/models/sql/compiler.py", line 1156, in execute_sql
cursor.execute(sql, params)
File "/opt/venv/lib/python3.8/site-packages/django/db/backends/utils.py", line 98, in execute
return super().execute(sql, params)
File "/opt/venv/lib/python3.8/site-packages/django/db/backends/utils.py", line 66, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/opt/venv/lib/python3.8/site-packages/django/db/backends/utils.py", line 75, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/opt/venv/lib/python3.8/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/opt/venv/lib/python3.8/site-packages/django/db/utils.py", line 90, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/opt/venv/lib/python3.8/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.IntegrityError: Problem installing fixture '/opt/app/init_data.json': Could not load contenttypes.ContentType(pk=19): duplicate key value violates unique constraint "django_content_type_app_label_model_76bd3d3b_uniq"
DETAIL: Key (app_label, model)=(django_celery_beat, solarschedule) already exists.
Can you add to this project one new plugin - any Feedreader?
Hi @StanGirard ,
Hope you are all well !
I managed to deploy seo-audits-toolkit via docker and I have some errors.
For example, /audit/lighthouse/score gives an 500 error:
Or, the serp rank or graphs are running endlessly.
Any idea how to fix it ?
Thanks for any insights or inputs on that.
Ps. From my point of view, there is a lot of things we can add to your project and is there a way to DM you on twitter about my ideas (I speak french too). My twitter handle is https://twitter.com/lucmichalski
Cheers,
Luc Michalski
Hello,
thanks for the nice job.
Hope I can help, although with more scientific background it's a it not very easy for me.
what other audit tools except for light house can we add?
Hi Stan,
I came across your software while researching automated lighthouse-ing of URLs taken from sitemap.xml files.
During setting up the project with Docker, I came across a few dep warnings, mostly from NPM related packages:
I guess these need to be addresses sooner or later?
[1/4] Resolving packages...
warning react-scripts > [email protected]: babel-eslint is now @babel/eslint-parser. This package will no longer receive updates.
warning react-scripts > webpack-dev-server > [email protected]: Chokidar 2 will break on node v14+. Upgrade to chokidar 3 with 15x less dependencies.
warning react-scripts > webpack-dev-server > chokidar > [email protected]: fsevents 1 will break on node v14+ and could be using insecure binaries. Upgrade to fsevents 2.
warning react-scripts > resolve-url-loader > rework > css > [email protected]: Please see https://github.com/lydell/urix#deprecated
warning react-scripts > resolve-url-loader > rework > css > source-map-resolve > [email protected]: Please see https://github.com/lydell/urix#deprecated
warning react-scripts > resolve-url-loader > rework > css > source-map-resolve > [email protected]: https://github.com/lydell/resolve-url#deprecated
warning react-scripts > workbox-webpack-plugin > workbox-build > @hapi/[email protected]: Switch to 'npm install joi'
warning react-scripts > workbox-webpack-plugin > workbox-build > [email protected]: This package has been deprecated and is no longer maintained. Please use @rollup/plugin-babel.
warning react-scripts > webpack > watchpack > watchpack-chokidar2 > [email protected]: Chokidar 2 will break on node v14+. Upgrade to chokidar 3 with 15x less dependencies.
warning react-scripts > workbox-webpack-plugin > workbox-build > @hapi/joi > @hapi/[email protected]: This version has been deprecated and is no longer supported or maintained
warning react-scripts > workbox-webpack-plugin > workbox-build > @hapi/joi > @hapi/[email protected]: Moved to 'npm install @sideway/address'
warning react-scripts > workbox-webpack-plugin > workbox-build > @hapi/joi > @hapi/[email protected]: This version has been deprecated and is no longer supported or maintained
warning react-scripts > workbox-webpack-plugin > workbox-build > @hapi/joi > @hapi/[email protected]: This version has been deprecated and is no longer supported or maintained
warning react-scripts > workbox-webpack-plugin > workbox-build > @hapi/joi > @hapi/topo > @hapi/[email protected]: This version has been deprecated and is no longer supported or maintained
warning react-scripts > workbox-webpack-plugin > workbox-build > strip-comments > babel-plugin-transform-object-rest-spread > babel-runtime > [email protected]: core-js@<3 is no longer maintained and not recommended for usage due to the number of issues. Please, upgrade your dependencies to the actual version of core-js@3.
warning react-scripts > jest > @jest/core > jest-config > jest-environment-jsdom > jsdom > [email protected]: request-promise-native has been deprecated because it extends the now deprecated request package, see https://github.com/request/request/issues/3142
warning react-scripts > jest > @jest/core > jest-config > jest-environment-jsdom > jsdom > [email protected]: request has been deprecated, see https://github.com/request/request/issues/3142
warning react-scripts > jest > @jest/core > jest-config > jest-environment-jsdom > jsdom > request > [email protected]: this library is no longer supported
[2/4] Fetching packages...
info [email protected]: The platform "linux" is incompatible with this module.
info "[email protected]" is an optional dependency and failed compatibility check. Excluding it from installation.
info [email protected]: The platform "linux" is incompatible with this module.
info "[email protected]" is an optional dependency and failed compatibility check. Excluding it from installation.
[3/4] Linking dependencies...
warning " > @material-ui/[email protected]" has unmet peer dependency "@material-ui/core@^4.9.12 || ^5.0.0-alpha.22".
warning " > @testing-library/[email protected]" has unmet peer dependency "@testing-library/dom@>=7.21.4".
warning " > [email protected]" has unmet peer dependency "ra-core@^3.5.3".
warning "react-admin > [email protected]" has unmet peer dependency "immutable@^3.8.1 || ^4.0.0-rc.1".
warning "react-admin > [email protected]" has unmet peer dependency "history@^4.7.2".
warning "react-admin > [email protected]" has incorrect peer dependency "react@^16.4.0".
warning "react-admin > [email protected]" has unmet peer dependency "seamless-immutable@^7.1.3".
warning "react-scripts > @typescript-eslint/eslint-plugin > [email protected]" has unmet peer dependency "typescript@>=2.8.0 || >= 3.2.0-dev || >= 3.3.0-dev || >= 3.4.0-dev || >= 3.5.0-dev || >= 3.6.0-dev || >= 3.6.0-beta || >= 3.7.0-dev || >= 3.7.0-beta".
[4/4] Building fresh packages...
success Saved lockfile.
As mentioned in project features list it must be possible to find the rank of a website on google result page
- SERP Rank - Get the rank of your website on google for specific queries
But I could not find any option in UI to do Serp rank.
Where should I go?
can i get an API access for this in python or javascript
Despite not a lot of features, the project is getting bigger. In order to avoid huge refactors down the line, a few things need to be addressed.
I'm eager to add new features. But each new feature complexifies the project even more. Taking things slow to avoid huge refactors and bottlenecks.
TODO:
Does not work on Windows.
When I run docker-compose --env-file .env-example up -d I get error I cannot resolve.
Hi, I can't use the demo url, can you fix it? Thank you
I like the direction in which the project is going. Thanks for adding new endpoints. For some urls the call works fine but for the following it crashed:
Log
06/20/2020 01:47:31 PM - INFO - 127.0.0.1 - - [20/Jun/2020 13:47:31] "GET /api/extract/images?url=https://linuxcommandlibrary.com/basics.html HTTP/1.1" 500 -
06/20/2020 01:47:31 PM - INFO - 127.0.0.1 - - [20/Jun/2020 13:47:31] "GET /favicon.ico HTTP/1.1" 404 -
06/20/2020 01:47:37 PM - DEBUG - Starting new HTTPS connection (1): linuxcommandlibrary.com:443
06/20/2020 01:47:37 PM - DEBUG - https://linuxcommandlibrary.com:443 "GET /basics.html HTTP/1.1" 200 None
06/20/2020 01:47:37 PM - ERROR - Exception on /api/extract/images [GET]
Traceback (most recent call last):
File "/usr/local/lib/python3.7/site-packages/flask/app.py", line 2446, in wsgi_app
response = self.full_dispatch_request()
File "/usr/local/lib/python3.7/site-packages/flask/app.py", line 1951, in full_dispatch_request
rv = self.handle_user_exception(e)
File "/usr/local/lib/python3.7/site-packages/flask/app.py", line 1820, in handle_user_exception
reraise(exc_type, exc_value, tb)
File "/usr/local/lib/python3.7/site-packages/flask/_compat.py", line 39, in reraise
raise value
File "/usr/local/lib/python3.7/site-packages/flask/app.py", line 1949, in full_dispatch_request
rv = self.dispatch_request()
File "/usr/local/lib/python3.7/site-packages/flask/app.py", line 1935, in dispatch_request
return self.view_functionsrule.endpoint
File "flask_api.py", line 123, in find_all_images_page
return find_all_images(value)
File "/Users/simon/projects/SEOToolkit/toolkit/seo/images.py", line 13, in find_all_images
alt = image['alt']
File "/usr/local/lib/python3.7/site-packages/bs4/element.py", line 1401, in getitem
return self.attrs[key]
KeyError: 'alt'
Hi there ! I was wondering whether anyone of you encountered issues with redis?
It is the only thing that doesn't seem to work. Using the path as defined in the example.
volumes: - redis-data:/data - redis-logs:${REDIS_LOG_PREFIX_PATH}
Terminal output
`Recreating osat-redis ... error
ERROR: for osat-redis Cannot create container for service redis: failed to mount local volume: mount /Users/phil/dev/seo-audits-toolkit/shared/logs/redis:/var/lib/docker/volumes/seo-audits-toolkit_redis-logs/_data, flags: 0x1000: no such file or directory
ERROR: for redis Cannot create container for service redis: failed to mount local volume: mount /Users/phil/dev/seo-audits-toolkit/shared/logs/redis:/var/lib/docker/volumes/seo-audits-toolkit_redis-logs/_data, flags: 0x1000: no such file or directory
ERROR: Encountered errors while bringing up the project.`
Thank you very much in advance - If you have any ideas that would be great - otherwise I'll dedicate my research to Docker stuff on the weekend :D
Hi Stan,
thanks for cleaning up the dependencies.
I only now realized you are using PyTorch with CUDA built in.
Any chance to run this on MacOS/without CUDA?
Thanks
Sascha
Build stops at Lighthouse Step 15 with Node v12. I updated the Dockerfile to pull Node v14 and was able to proceed, but may need to update your dependencies here.
Step 15/34 : RUN yarn global add lighthouse
---> Running in 8b4b96307626
yarn global v1.22.17
[1/4] Resolving packages...
warning lighthouse > intl-messageformat > [email protected]: We've written a new parser that's 6x faster and is backwards compatible. Please use @formatjs/icu-messageformat-parser
warning lighthouse > raven > [email protected]: Please upgrade to version 7 or higher. Older versions may use Math.random() in certain circumstances, which is known to be problematic. See https://v8.dev/blog/math-random for details.
[2/4] Fetching packages...
[3/4] Linking dependencies...
[4/4] Building fresh packages...
success Installed "[email protected]" with binaries:
- lighthouse
- chrome-debug
- smokehouse
Done in 7.71s.
Removing intermediate container 8b4b96307626
I'm running the latest master without any modification.
Log:
06/19/2020 01:15:58 PM - INFO - 127.0.0.1 - - [19/Jun/2020 13:15:58] "GET /api/graph?url=https://linuxcommandlibrary.com/tips.html HTTP/1.1" 500 -
Traceback (most recent call last):
File "/usr/local/lib/python3.7/site-packages/flask/app.py", line 2463, in call
return self.wsgi_app(environ, start_response)
File "/usr/local/lib/python3.7/site-packages/flask/app.py", line 2449, in wsgi_app
response = self.handle_exception(e)
File "/usr/local/lib/python3.7/site-packages/flask/app.py", line 1866, in handle_exception
reraise(exc_type, exc_value, tb)
File "/usr/local/lib/python3.7/site-packages/flask/_compat.py", line 39, in reraise
raise value
File "/usr/local/lib/python3.7/site-packages/flask/app.py", line 2446, in wsgi_app
response = self.full_dispatch_request()
File "/usr/local/lib/python3.7/site-packages/flask/app.py", line 1951, in full_dispatch_request
rv = self.handle_user_exception(e)
File "/usr/local/lib/python3.7/site-packages/flask/app.py", line 1820, in handle_user_exception
reraise(exc_type, exc_value, tb)
File "/usr/local/lib/python3.7/site-packages/flask/_compat.py", line 39, in reraise
raise value
File "/usr/local/lib/python3.7/site-packages/flask/app.py", line 1949, in full_dispatch_request
rv = self.dispatch_request()
File "/usr/local/lib/python3.7/site-packages/flask/app.py", line 1935, in dispatch_request
return self.view_functionsrule.endpoint
File "/Users/simon/projects/SEOToolkit/flask_api.py", line 115, in interactive_graph
return generate_interactive_graph(conn, urls, relaunch, maxi_urls)
File "/Users/simon/projects/SEOToolkit/flask_api.py", line 103, in generate_interactive_graph
return update_or_insert_graph_in_db(conn, urls, maximum_urls)
File "/Users/simon/projects/SEOToolkit/flask_api.py", line 52, in update_or_insert_graph_in_db
return render_template("bokeh.html", script=script, div=div, domain=domain, template="Flask", time=datetime.now().strftime("%m/%d/%Y, %H:%M:%S"))
File "/usr/local/lib/python3.7/site-packages/flask/templating.py", line 138, in render_template
ctx.app.jinja_env.get_or_select_template(template_name_or_list),
File "/usr/local/lib/python3.7/site-packages/jinja2/environment.py", line 930, in get_or_select_template
return self.get_template(template_name_or_list, parent, globals)
File "/usr/local/lib/python3.7/site-packages/jinja2/environment.py", line 883, in get_template
return self._load_template(name, self.make_globals(globals))
File "/usr/local/lib/python3.7/site-packages/jinja2/environment.py", line 857, in _load_template
template = self.loader.load(self, name, globals)
File "/usr/local/lib/python3.7/site-packages/jinja2/loaders.py", line 115, in load
source, filename, uptodate = self.get_source(environment, name)
File "/usr/local/lib/python3.7/site-packages/flask/templating.py", line 60, in get_source
return self._get_source_fast(environment, template)
File "/usr/local/lib/python3.7/site-packages/flask/templating.py", line 89, in _get_source_fast
raise TemplateNotFound(template)
jinja2.exceptions.TemplateNotFound: bokeh.html
Please use this issue to request new features
I have since gotten then dashboard to show by using either edge or chrome, after find the reason in the dev console for Firefox:
Cross-Origin Request Blocked: The Same Origin Policy disallows reading the remote resource at http://localhost:8000/dj-rest-auth/login/. (Reason: CORS request did not succeed). Status code: (null).
Error for windows users:
Change path SQLite in SQLALCHEMY Config from 'sqlite:////tmp/test114.db' to r'sqlite:///C:\tmp\test114.db'
Implement OS Detection to avoid this issue
Hi,
I get Internal Server Errors whenever I try to generate a report (Lighthouse or other) from a site, which is hosted on localhost.
What can be the reason?
Traceback (most recent call last):
File "/usr/local/lib/python3.7/site-packages/nltk/corpus/util.py", line 86, in __load
root = nltk.data.find('{}/{}'.format(self.subdir, zip_name))
File "/usr/local/lib/python3.7/site-packages/nltk/data.py", line 701, in find
raise LookupError(resource_not_found)
LookupError:
Resource stopwords not found.
Please use the NLTK Downloader to obtain the resource:
import nltk
nltk.download('stopwords')
For more information see: https://www.nltk.org/data.html
Attempted to load corpora/stopwords.zip/stopwords/
Searched in:
- '/Users/simon/nltk_data'
- '/usr/local/Cellar/python/3.7.7/Frameworks/Python.framework/Versions/3.7/nltk_data'
- '/usr/local/Cellar/python/3.7.7/Frameworks/Python.framework/Versions/3.7/share/nltk_data'
- '/usr/local/Cellar/python/3.7.7/Frameworks/Python.framework/Versions/3.7/lib/nltk_data'
- '/usr/share/nltk_data'
- '/usr/local/share/nltk_data'
- '/usr/lib/nltk_data'
- '/usr/local/lib/nltk_data'
I'm on: 4082a37
I executed the following end point: http://localhost:5000/api/extract/links?url=https://schubert-simon.de
I expected that child 9 and 10 of the 200
array should be:
"https://schubert-simon.de#places"
"https://schubert-simon.de#photos"
But had been:
"http://whatch.online#places"
"http://whatch.online#photos"
And #projects
in 500
should be "https://schubert-simon.de#projects".
HTML example:
<a href="#photos" id="photos">
<h2>PHOTOS</h2>
</a>
Maybe it has something todo with the caching but I did a "fresh" install of the repo. Can you reproduce it? Thanks a lot.
Hello, could you make the installation steps for docker more detailed? I tried the steps from the main page, and read other sources but I couldn't make this tool to work/open, thanks in advance.
Lighthouse audits not working, i tested both in docker and normal installation it just wont save the url when we submit the url. Google api is added to env
Hi guys,
Hope you are all well !
Is it possible to add a prefix path variable to flask in order to ease the integration in traefik ?
In fact, I d like to access seo-audits-toolkit from /devops/seo-toolkit path
here is the traefik label config:
- "traefik.http.routers.seo-toolkit.rule=Host(`xxxx-xxxx.xxx-xxxxx.com`) && PathPrefix(`/devops/seo-toolkit`)"
here is an excerpt of my docker-compose:
seo-toolkit:
image: lucmichalski/seo-audits-toolkit:latest
container_name: seo-audits-toolkit
build:
context: .docker/seo-audits-toolkit
dockerfile: Dockerfile
expose:
- 5000
depends_on:
- redis
- postgres
- seo-celery
networks:
- internal
- web
environment:
- SECRET_KEY=changeme
- URL_APP=http://localhost:5000/
- FLASK_APP=SEOToolkit
- FLASK_ENV=development
- GOOGLE_API_KEY=None
- CELERY_BROKER_URL=redis://redis:6379/0
- CELERY_RESULT_BACKEND=redis://redis:6379/0
- SQLALCHEMY_DATABASE_URI=postgresql+psycopg2://unicorn_user:magical_password@postgres:5432/rainbow_database
labels:
- "org.label-schema.group=seo-toolkits"
- "traefik.enable=true"
- "traefik.docker.network=web"
- "traefik.http.routers.seo-toolkit.rule=Host(`para-test.gsi-network.com`) && PathPrefix(`/devops/seo-toolkit`)"
- 'traefik.http.routers.seo-toolkit.tls=true'
- 'traefik.http.routers.seo-toolkit.tls.certresolver=letsencrypt'
- "traefik.http.routers.seo-toolkit.entrypoints=https"
- "traefik.http.services.seo-toolkit.loadbalancer.server.port=5000"
- "traefik.http.middlewares.seo-toolkit.compress=true"
- "traefik.http.routers.seo-toolkit.middlewares=seo-toolkit-replacepathregex,seo-toolkit-basicauth"
- "traefik.http.middlewares.seo-toolkit-replacepathregex.replacepathregex.regex=^/devops/seo-toolkit(.*)"
- "traefik.http.middlewares.seo-toolkit-replacepathregex.replacepathregex.replacement=/devops/seo-toolkit$$1"
- 'traefik.http.middlewares.seo-toolkit-basicauth.basicauth.removeheader=true'
- "traefik.http.middlewares.seo-toolkit-basicauth.basicauth.realm=EG-Seo Audit Toolkits"
- "traefik.http.middlewares.seo-toolkit-basicauth.basicauth.users=${USER_PASSWORD}"
restart: unless-stopped
seo-celery:
build:
context: .docker/seo-audits-toolkit
dockerfile: Dockerfile
networks:
- internal
- web
depends_on:
- redis
environment:
- SECRET_KEY=changeme
- URL_APP=http://localhost:5000/
- FLASK_APP=SEOToolkit
- FLASK_ENV=development
- GOOGLE_API_KEY=None
- CELERY_BROKER_URL=redis://redis:6379/0
- CELERY_RESULT_BACKEND=redis://redis:6379/0
- SQLALCHEMY_DATABASE_URI=postgresql+psycopg2://unicorn_user:magical_password@postgres:5432/rainbow_database
labels:
- "traefik.enable=false"
- "org.label-schema.group=seo-audits"
command: "celery worker -A celery_worker.celery --loglevel=info"
restart: unless-stopped
postgres:
image: "postgres" # use latest official postgres version
container_name: postgres
networks:
- internal
environment:
- POSTGRES_USER=unicorn_user
- POSTGRES_PASSWORD=magical_password
- POSTGRES_DB=rainbow_database
volumes:
- postgres-data:/var/lib/postgresql/data/
redis:
image: redis:6-alpine
container_name: redis
networks:
- internal
volumes:
- redis-data:/data
labels:
- "org.label-schema.group=storage"
- "traefik.enable=false"
command:
- "redis-server"
- "--maxmemory 3G"
- "--maxmemory-policy allkeys-lru"
restart: unless-stopped
Would be awesome to add something like app.config["APPLICATION_ROOT"] = "/devops/seo-toolkit/" but I am not sure for the rest of the integration ^^
Thanks for any insights or inputs on that.
Cheers,
Luc Michalski
docker: Error response from daemon: pull access denied for seo-toolkit, repository does not exist or may require 'docker login': denied: requested access to the resource is denied.
Getting that error while doing docker run -d -p 5000:5000 seo-toolkit:latest
Hello,
I tried to install the app as described in the README, but got the following error:
=> ERROR [15/24] RUN yarn global add lighthouse 18.3s
------
> [15/24] RUN yarn global add lighthouse:
1.472 yarn global v1.22.19
1.583 [1/4] Resolving packages...
4.343 [2/4] Fetching packages...
17.90 error [email protected]: The engine "node" is incompatible with this module. Expected version ">=18.16". Got "18.13.0"
17.94 error Found incompatible module.
17.94 info Visit https://yarnpkg.com/en/docs/cli/global for documentation about this command.
------
Dockerfile:22
--------------------
20 | RUN apt install -y chromium
21 | # Install Lighthouse
22 | >>> RUN yarn global add lighthouse
23 |
24 | # Install Pipenv
--------------------
ERROR: failed to solve: process "/bin/sh -c yarn global add lighthouse" did not complete successfully: exit code: 1
ERROR: Service 'beats' failed to build : Build failed
My software versions are following:
root@ubuntu:/opt/seo-audits-toolkit# yarn --version
1.22.21
root@ubuntu:/opt/seo-audits-toolkit# npm --version
10.1.0
root@ubuntu:/opt/seo-audits-toolkit# nodejs --version
v20.9.0
When I run yarn install
on it's own, there are no errors shown:
root@ubuntu:/opt/seo-audits-toolkit# yarn install
yarn install v1.22.21
[1/4] Resolving packages...
success Already up-to-date.
Done in 0.13s.
Do you have any idea what would may be causing the trouble here?
For this to be even more helpful, seeing trends over time helps prioritize decisions on what to fix/when, etc.
This really needs better documentation, but in any event I got it working and it's great to have everything ready so quickly and without BS paywalls, precisely as the dev intended to achieve. Great work!
Hi ! .... I have been trying to install OSAT for a while without finding a solution to the following problem... I verified that /root/seo-audits-toolkit/shared/logs/redis exists, tested redis with PONG output, created the path /var/lib/ docker/volumes/seo-audits-toolkit_redis-logs/_data without success in any case. Attached photo of the kernel. Also bind conf redis to 0.0.0.0
I am working with:
--> Ubuntu 22.04.1 with WSL2
--> Python 3.10.6
--> docker-compose version 1.29.2
--> Docker version 20.10.17
--> Redis server v=7.0.5
Thank you!
issue 1= /docker-compose.yml" is unsupported. You might be seeing this error because you're using the wrong Compose file version. Either specify a supported version (e.g "2.2" or "3.3") and place your service definitions under the services
key, or omit the version
key and place your service definitions at the root of the file to use version 1.
issue 2= WARN[0000] The "TINI_VERSION" variable is not set. Defaulting to a blank string.
WARN[0000] The "POSTGRES_PASSWORD" variable is not set. Defaulting to a blank string.
WARN[0000] The "DEBUG" variable is not set. Defaulting to a blank string.
WARN[0000] The "ALLOWED_HOSTS" variable is not set. Defaulting to a blank string.
WARN[0000] The "REST_FRAMEWORK_PAGE_SIZE" variable is not set. Defaulting to a blank string.
WARN[0000] The "CELERY_RESULT_BACKEND" variable is not set. Defaulting to a blank string.
WARN[0000] The "DATABASE_ENGINE" variable is not set. Defaulting to a blank string.
WARN[0000] The "POSTGRES_USER" variable is not set. Defaulting to a blank string.
WARN[0000] The "POSTGRES_PORT" variable is not set. Defaulting to a blank string.
WARN[0000] The "POSTGRES_DB" variable is not set. Defaulting to a blank string.
WARN[0000] The "SECRET_KEY" variable is not set. Defaulting to a blank string.
WARN[0000] The "CELERY_BROKER_URL" variable is not set. Defaulting to a blank string.
WARN[0000] The "CORS_ALLOWED_ORIGINS" variable is not set. Defaulting to a blank string.
WARN[0000] The "NAMESPACE" variable is not set. Defaulting to a blank string.
WARN[0000] The "TINI_VERSION" variable is not set. Defaulting to a blank string.
WARN[0000] The "NAMESPACE" variable is not set. Defaulting to a blank string.
WARN[0000] The "CELERY_BROKER_URL" variable is not set. Defaulting to a blank string.
WARN[0000] The "DATABASE_ENGINE" variable is not set. Defaulting to a blank string.
WARN[0000] The "DEBUG" variable is not set. Defaulting to a blank string.
WARN[0000] The "CELERY_RESULT_BACKEND" variable is not set. Defaulting to a blank string.
WARN[0000] The "ALLOWED_HOSTS" variable is not set. Defaulting to a blank string.
WARN[0000] The "REST_FRAMEWORK_PAGE_SIZE" variable is not set. Defaulting to a blank string.
WARN[0000] The "POSTGRES_PORT" variable is not set. Defaulting to a blank string.
WARN[0000] The "POSTGRES_DB" variable is not set. Defaulting to a blank string.
WARN[0000] The "POSTGRES_PASSWORD" variable is not set. Defaulting to a blank string.
WARN[0000] The "POSTGRES_USER" variable is not set. Defaulting to a blank string.
WARN[0000] The "SECRET_KEY" variable is not set. Defaulting to a blank string.
WARN[0000] The "CORS_ALLOWED_ORIGINS" variable is not set. Defaulting to a blank string.
WARN[0000] The "NAMESPACE" variable is not set. Defaulting to a blank string.
WARN[0000] The "TINI_VERSION" variable is not set. Defaulting to a blank string.
WARN[0000] The "CELERY_RESULT_BACKEND" variable is not set. Defaulting to a blank string.
WARN[0000] The "POSTGRES_DB" variable is not set. Defaulting to a blank string.
WARN[0000] The "POSTGRES_PASSWORD" variable is not set. Defaulting to a blank string.
WARN[0000] The "POSTGRES_PORT" variable is not set. Defaulting to a blank string.
WARN[0000] The "POSTGRES_USER" variable is not set. Defaulting to a blank string.
WARN[0000] The "SECRET_KEY" variable is not set. Defaulting to a blank string.
WARN[0000] The "CORS_ALLOWED_ORIGINS" variable is not set. Defaulting to a blank string.
WARN[0000] The "ALLOWED_HOSTS" variable is not set. Defaulting to a blank string.
WARN[0000] The "DEBUG" variable is not set. Defaulting to a blank string.
WARN[0000] The "DATABASE_ENGINE" variable is not set. Defaulting to a blank string.
WARN[0000] The "REST_FRAMEWORK_PAGE_SIZE" variable is not set. Defaulting to a blank string.
WARN[0000] The "CELERY_BROKER_URL" variable is not set. Defaulting to a blank string.
WARN[0000] The "NAMESPACE" variable is not set. Defaulting to a blank string.
WARN[0000] The "NODE_VERSION" variable is not set. Defaulting to a blank string.
WARN[0000] The "TINI_VERSION" variable is not set. Defaulting to a blank string.
WARN[0000] The "POSTGRES_USER" variable is not set. Defaulting to a blank string.
WARN[0000] The "POSTGRES_PASSWORD" variable is not set. Defaulting to a blank string.
WARN[0000] The "POSTGRES_DB" variable is not set. Defaulting to a blank string.
WARN[0000] The "NAMESPACE" variable is not set. Defaulting to a blank string.
WARN[0000] The "REDIS_VERSION" variable is not set. Defaulting to a blank string.
WARN[0000] The "REDIS_LOG_PREFIX_PATH" variable is not set. Defaulting to a blank string.
WARN[0000] The "NAMESPACE" variable is not set. Defaulting to a blank string.
WARN[0000] The "REDIS_LOG_PREFIX_PATH" variable is not set. Defaulting to a blank string.
WARN[0000] The "REDIS_MEMORY_MAX" variable is not set. Defaulting to a blank string.
WARN[0000] The "REDIS_LOG_FILE" variable is not set. Defaulting to a blank string.
WARN[0000] The "PWD" variable is not set. Defaulting to a blank string.
WARN[0000] The "PWD" variable is not set. Defaulting to a blank string.
WARN[0000] The "PWD" variable is not set. Defaulting to a blank string.
Hi guys,
Hope you are all well !
Can you add to the README how to get the GOOGLE_API_KEY for what services ?
Thanks for any insights or inputs on that.
Cheers,
Luc Michalski
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.