davidglezz / tfg

Prototype of a web application that indexes microdata (schema.org) exposed by electronic stores and allows the search of products.

Agregador de Productos y comparador de precios

Proyecto Fin de Grado de David González García que consiste en una aplicacion web que extrae e indexa productos a la venta de tiendas electrónicas mediante los datos estructurados que estas ponen a disposición.

Demostración

http://tfg.davidgagnon.org/

tfg's Issues

[MdListOption] selectChange/deselected events don't fire

On products filter list, currently using Click event

Blocked by
angular/components#6864
PR:
https://github.com/angular/material2/pull/6901/files

TaskUpdateUrls stops after long time.

After several hours of operation, the task stops working.

Support robots.txt

Support robots.txt to get the sitemap url and restrictions.

Allow to edit shop info

QueryFailedError: Incorrect string value: '\xF4\x80\x80

Problem:
QueryFailedError: ER_TRUNCATED_WRONG_VALUE_FOR_FIELD: Incorrect string value: '\xF4\x80\x80\x80<2...' for column 'description' at row 1

Posible solution:
Change encoding from utf8 to utf8mb4

https://stackoverflow.com/questions/10957238/incorrect-string-value-when-trying-to-insert-utf-8-into-mysql-via-jdbc

Frontend filters does not work in production: ng build --prod

Md-expansion panel opened and close events not firing on prod build (beta.11)

angular/components#7172

angular/components#7309

Búsqueda sencilla de productos

htmlmetaparser TypeError: Cannot read property 'text' of undefined

TypeError: Cannot read property 'text' of undefined
at Handler.onclosetag (D:\Clase\TFG\src\crawler\node_modules\htmlmetaparser\dist\index.js:445:9)

URL when characters with accents appear

Problem with accents:

%25C3 stretches to:

%2525252525252525252525252525252525252525252525252525252525252525252525252525252525252525252525252525252525252525252525252525252525252525252525252525252525252525252525252525252525252525252525252525252525252525252525C3

Allow to view shop info

TypeError: Cannot read property 'text' of undefined

TypeError: Cannot read property 'text' of undefined
     at Handler.onclosetag (/app/crawler/node_modules/htmlmetaparser/src/index.ts:753:5)
     at Parser.onend (/app/crawler/node_modules/htmlparser2/lib/Parser.js:307:14)
     at Tokenizer._finish (/app/crawler/node_modules/htmlparser2/lib/Tokenizer.js:838:12)
     at Tokenizer.end (/app/crawler/node_modules/htmlparser2/lib/Tokenizer.js:829:25)
     at Parser.end (/app/crawler/node_modules/htmlparser2/lib/Parser.js:338:18)
     at Stream.<anonymous> (/app/crawler/node_modules/htmlparser2/lib/WritableStream.js:15:10)
     at Object.onceWrapper (events.js:314:30)
     at emitNone (events.js:105:13)
     at Stream.emit (events.js:207:7)
     at finishMaybe (_stream_writable.js:604:14)

Support html link rel=alternate

Link rel=alternate tells the user (search engines) that there is an other URL with the same content but in an other form (eg. language, mobile, ...).

Ignore SSL errors

SSL validation errors:
[ERROR] Error certificate has expired

Potential security vulnerabilities in dependencies.

Fetch node as Chrome extensions for distributed crawl

https://developer.chrome.com/extensions/samples

SitemapParser Error read ECONNRESET

Al parsear algunos sitemaps:

[ERROR] Error read ECONNRESET
Error: write after end
    at writeAfterEnd (_stream_writable.js:236:12)
    at PassThrough.Writable.write (_stream_writable.js:287:5)
    at Unzip.ondata (_stream_readable.js:640:20)
    at emitOne (events.js:115:13)
    at Unzip.emit (events.js:210:7)
    at Unzip.Readable.read (_stream_readable.js:476:10)
    at flow (_stream_readable.js:847:34)
    at resume_ (_stream_readable.js:829:3)
    at _combinedTickCallback (internal/process/next_tick.js:138:11)
    at process._tickCallback (internal/process/next_tick.js:180:9)
Error: write after end
    at writeAfterEnd (_stream_writable.js:236:12)
    at PassThrough.Writable.write (_stream_writable.js:287:5)
    at Unzip.ondata (_stream_readable.js:640:20)
    at emitOne (events.js:115:13)
    at Unzip.emit (events.js:210:7)
    at Unzip.Readable.read (_stream_readable.js:476:10)
    at flow (_stream_readable.js:847:34)
    at resume_ (_stream_readable.js:829:3)
    at _combinedTickCallback (internal/process/next_tick.js:138:11)
    at process._tickCallback (internal/process/next_tick.js:180:9)

False positive in price change when there is no price

Prices can be null values.

startup.sh: mv: cannot move '...' to 'api-server/public/assets': Directory not empty

mv: cannot move '/app/frontend/dist/assets' to '/app/api-server/public/assets': Directory not empty

Support for canonical urls

HTML tag: <link rel="canonical" href="http://www.example.com/" />
HTTP header: Link: <http://www.example.com/>; rel="canonical"

Allow to add shops

Handle as / all non /api routes

Update frontend to Angular 5

https://angular-update-guide.firebaseapp.com/