dbjpanda / feeds_advance_crawler

Provides advance crawler for Drupal 8 Feeds module using Node.js as backend server | Developed in GSOC'18

feeds_advance_crawler

Provides advance crawler for Drupal 8 Feeds module using Node.js as backend server | Developed in GSOC'18

Installation

To make the module work you need to configure its backend server. More info here https://github.com/shibasisp/feeds_advance_crawler_backend

feeds_advance_crawler's Issues

Add validation in feeds_advance_crawler for checking the existance of the server.

The configuration form in feeds_advance_crawler asks for the Nodejs server host and port, So before saving, It would be great to validate the server running on that host and port. It can be done by sending a ping request to the server before saving.

Curl unable to resolve permanently moved domain name

Container_name:9000 works but a domain name like node.localhost:80 (Set up by proxy server ) doesn't work. Also it should display a status message for successful connection

This module should be able to handle multiple requests/ response in parallel

For example we have one container for "advance crawler" and it has the url test-1.node.localhost . We wanted to scale up our container to 10 containers with "docker-compose scale" command. So we will have 10 node crawlers which will have 10 urls like test-2.node.localhost, test-3.node.ocalhost......and so on. How feeds_advance_crawler is going to send request to these node in parallel.