The crawler leverages the Google's Custom Search Json API.
By calling the API the crawler issues requests against an existing Custom Search Engine (the API key and the
Search Engine Id can be found in the GoogleSearcher class). The API provides the response in JSON format. It will
contain also the relevant links.
The crawler (WebCrawler) distributes the processing of the response links to a number of workers (ICrawlerWorker)
(each worker will be responsible for processing one page).
The crawler worker (ICrawlerWorker) will read the html of the URL, given as input, and count the occurrences
of each library. Using the TechnologyAnalizer class it is able to determine the name of the library when given the
value of the src attribute of a script tag as input (for more details see the next paragraph).
The WebCrawler will wait for the completion of all of the workers (ICrawlerWorker). Upon completion, it will sum up the
count of libraries computed by each worker and then output the top 5 most frequently used Javascript libraries.
The deduplication algorithm is based on regex patterns. The application loads a list of script regex patterns from the
apps.json file (located in project root). One library name may be associated with one or more regex patterns.
Each script/src attribute is then matched with all of the known script regex patterns and the actual name of
the library is determined.
The apps.json file was inspired from the open source project AliasIO/wappalyzer.
The original apps.json could not be used in the current application. In each of the apps.script regex patterns
the ending \\;version:\1 had to be removed.
"A-Frame": {"cats": [25], "script": "/?([\\d.]+)?/aframe(?:\\.min)?\\.js\\;version:\\1"}
was changed to:
"A-Frame": {"cats": [25],"script": "/?([\\d.]+)?/aframe(?:\\.min)?\\.js"}
- Class diagram - the diagram was built using Online Visual Paradigm.
