New search concept about cammintegrationportal

[JochenHWezel]

requires plugin concept

[JochenHWezel]

Pipeline

1. craler,sitemap scan list
2. crawler record data
3. split up into words
4. usage by search form with Logical parser

[JochenHWezel]

apps might provide crawler Setup for standard search index of Server Group, they might additionally or alternatively specify additional search index names for purpose of in-app-search-index

[JochenHWezel]

pages being crawled by crawler provide the data from the crawler Setup as follows:

• sitemap as link list in non-visible,div
• record data as ...???
  • maybe additional sitemap URL to Special URL for providing record data?

[JochenHWezel]

crawler tracks page Status - on repetitive page error of an URL, it stops crawling for a configured time

[JochenHWezel]

search page can be set up with a Parser pre-set - requiring that user search text is logically valid/closed nexted logic Levels automatically on demand

[JochenHWezel]

crawler should be able to crawl

• pdf
• docx
• xlsx,cells as formattedtext
• plain text
• html
• jpg meta data
• etc.

[JochenHWezel]

CrawlerRequestSetup must configure

• follow links for X Levels (e.g. as xenu)
• follow links
  • within nav URL Folder (default)
  • within custum list of URLs
  • everywhere (up,to Limit of X Levels)

[JochenHWezel]

crawler,should consider meta robots/index no-follow/follow, /robots.txt commands, /robots.txt-sitemap

[JochenHWezel]

might make sense to make indexing use a 2nd database for big Environments/heavy load and use the cwm db only for small Environments/light load

might make sense to re-use other technics/modules like luscene or other engines