eklem / vinmonopolet-crawler

Crawling Vinmonopolet-data and indexing it to a norch search index

vinmonopolet-crawler

Crawling Vinmonopolet-data and indexing it to a norch search index, using vinmonopolet and search-index-indexer library.

Has these features:

• Removing unwanted key/value pars by it's keys
• Changing values to arrays for defined keys. Preparing for being used as filters.
• Copying value of defined key to id-value. Makes it easy to update index.
• Creating image URL based on id/code.
• Pushing filter.category to product.productType when different for better filter functionality.

vinmonopolet-crawler's Issues

log which file is indexed

Nice to have when the bash-script is running to see where you are...

Filter on productType

If filter is set to a productType, only process this productType.

Make script take input on supercategory

Should be able to index from crontab with supercategories

url-encode file-names

Get an error message on Norwegian characters. Should url-encode the file-name before writing the file.

separator no longer needed on categorizing

Can remove stuff like this

  "fieldOptions": {
    "productType": {
      "separator": "§",
      "nGramLength": [1]
    }

filters for all supercategories

add filters for all supercategories, i.e. "brennevin"

"Bitter" is both a liquor and a beer type

Need to see how to distinguish bitter in liquor and beer category.