leopardslab / crawlerx Goto Github PK

Please note: View this issue after enabling ZenHub

• Project - [GSoC 2021] CrawlerX - On-demand auto-scaling platform on Kubernetes
• Description - This epic relates to the deployment pattern of the CrawlerX on Kubernetes.
• Student: sangagomipieris
• Mentor: sajithaliyanage/ prabushitha

Is your feature request related to a problem? Please describe.
This issue introduces Kubernetes artifacts for CrawlerX project. Currently, we can only deploy the platform in Docker orchestration frameworks.

Is your feature request related to a problem? Please describe.
$subject

Type/Improvement

We need to create following k8s artifacts for the $subject

• Deployment
• Service
• ConfigMap
• Secret
• HPA

Add crawl spiders for the following or popular websites.

• Youtube
• Quora
• Facebook
• Reddit
• GitHub

Currently implemented spiders can be found in - https://github.com/leopardslab/CrawlerX/tree/master/scrapy_app/scrapy_app/spiders

We can improve the JSON viewer by adding the followings,

• Add a syntax highlighter
• Add Copy to Clipboard button
• Add some readability improvements

Create Dockerfiles for following modules.

• VueJs web application
• Django server
• Scrapy server

Is your feature request related to a problem? Please describe.

Airflow is a platform created by the community to programmatically author, schedule and monitor workflows. It provides following advantages.

• Easy to manage workflows
• Guranteed delivery
• Easy to monitor workflows
• Easy to schedule Crawler jobs
• Supports many many Message broker implementations
• Container native support
• Dashboard support

Is your feature request related to a problem? Please describe.
Current crawlerX_app doesn't have forgot password functionality.

Describe the bug
Even though firebase has only used for authentication purpose in CrawlerX it is not nice to put them in public repo. instead of that we can add a env file to the project and put all the environmental variables there

Is your feature request related to a problem? Please describe.
Currently it only possible to view data via the embedded JSON viewer. It would be great if we can export these data as a JSON, CSV file in each project.

We need to create following k8s artifacts for the $subject

• Deployment
• Service
• Ingress
• ConfigMap
• Secret

Describe the bug
.gitignore doesn't contain necessary line to ignore the node modules.

Is your feature request related to a problem? Please describe.
As per the current implementation, CrawlerX supports only for HTTP and HTTPS urls. This need to extend for Tor browser Urls.

We need to create following k8s artifacts for the $subject

• Deployment
• Service
• Ingress
• ConfigMap
• Secret
• NFS provisioner

this issue tracks the documentation progress of the project.

We need to create following k8s artifacts for the $subject

• Deployment
• Service
• Ingress
• ConfigMap
• Secret

Hi.
I run docker-compose up --build and get an error:

I tried installing Twisted's dependencies in Dockerfile or change version Twisted in requirements.txt but it didn't solve the problem.
RUN apt-get update && apt-get install -y gcc libc6-dev

Can you help me?
Thank you!

At the end of the project, Readme.md file needs to be updated with the relevant step by step user guide.

Is your feature request related to a problem? Please describe.
$subject

Create a new logo for the CrawlerX platform and insert it inside the folder called logo in the root directory. Also, update the favicon of the web-app

Describe the bug
There are few console warnings related to firebase and vue router

To Reproduce
Steps to reproduce the behavior:

1. Go to 'crawlerx_app'
2. Run the project by typing 'npm run serve'
3. Open Console
4. See warnings

Screenshots

Desktop (please complete the following information):

• OS: Windows
• Browser :chrome

We need to create following k8s artifacts for the $subject

• Deployment
• Service
• HPA
• ConfigMap
• Secret

Is your feature request related to a problem? Please describe.

Currently it supports basic data saving mechanism for each user. Since this is a data scraping server, this should have a capability to manage data in more user friendly manner .

Hello Team, Thank you very much for the project.

Is it possible to replace Firebase with an open-source tools like appwrite, feathersjs, etc?

We can update the Schedule new Job section with the following improvements.

• New section for scheduled jobs
• New section for file upload URLs
• New design for corn schedule jobs

Please note: View this issue after enabling ZenHub

• Project - [GSoC 2021] Improve CrawlerX web application
• Description - This epic relates to the project improve CrawlerX web applicaation.
• Student: beshiniii
• Mentor: sajithaliyanage/ prabushitha

We need to create following k8s artifacts for the $subject

• Deployment
• Service
• ConfigMap
• Secret

Is your feature request related to a problem? Please describe.
$subject. We need to create relevant composer.yml for the platform.

Describe the solution you'd like
Type/Improvement