Issue with Greek characters & searchNormalize about virtual-select

[gnbm]

Hello @nikolasr200
I've run a couple of tests in order to change the function normalizeString to:

static normalizeString(text) {
const NON_WORD_REGEX = /[\u0300-\u036f]/g;

// Normalize the text to lowercase and remove accents
let normalizedText = text
    .toLowerCase()
    .normalize('NFD')
    .replace(NON_WORD_REGEX, '');

// Define a mapping of characters to remove accents for
const accentMappings = {
    'a': ['à', 'á', 'â', 'ã', 'ä', 'å'],
    'e': ['è', 'é', 'ê', 'ë','έ','ε'],
    'i': ['ì', 'í', 'î', 'ï'],
    'o': ['ò', 'ó', 'ô', 'õ', 'ö'],
    'u': ['ù', 'ú', 'û', 'ü'],
    'c': ['ç'],
    'g': ['ğ'],
    'n': ['ñ'],
};

// Replace accented characters with their non-accented equivalents
for (const baseChar in accentMappings) {
    const accentedChars = accentMappings[baseChar].join('');
    const regex = new RegExp(`[${accentedChars}]`, 'g');
    normalizedText = normalizedText.replace(regex, baseChar);
}
return normalizedText;   }

Another alternative would be:

`static normalizeString(text) {
//const NON_WORD_REGEX = /[^\w]/g;
//return text.normalize("NFD").replace(NON_WORD_REGEX, "");

const NON_WORD_REGEX = /[^a-z0-9\s]/g;

// Define a mapping of characters to remove accents for
const accentMappings = {
  'à': 'a', 'á': 'a', 'â': 'a', 'ã': 'a', 'ä': 'a', 'å': 'a',
  'è': 'e', 'é': 'e', 'ê': 'e', 'ë': 'e', 'έ': 'e', 'ε': 'e',
  'ì': 'i', 'í': 'i', 'î': 'i', 'ï': 'i',
  'ò': 'o', 'ó': 'o', 'ô': 'o', 'õ': 'o', 'ö': 'o',
  'ù': 'u', 'ú': 'u', 'û': 'u', 'ü': 'u',
  'ç': 'c',
  'ğ': 'g',
  'ñ': 'n'

};

// Create a regex pattern to match accented characters
const accentPattern = new RegExp([${Object.keys(accentMappings).join('')}], 'g');

// Normalize the text to lowercase, remove accents, and non-word characters
const normalizedText = text
.toLowerCase()
.normalize('NFD')
.replace(accentPattern, char => accentMappings[char] || char)
.replace(NON_WORD_REGEX, '');

return normalizedText;
}`

Could you run some tests on top of this in order to make sure it won't break the current use cases and fulfil the requirements you mentioned? I'm also worried about performance in these scenarios.

cc: @sa-si-dev

[gnbm]

IMHO, it would be probably a good idea to allow to pass a searchCompareValues function, to be able to fully customize the search. Thus one could implements fuzzy search, custom normalization, etc...

I personally will see that as a new feature (even to be able to be backwards compatible) and here just try to include new use cases without further impacts.
But feel free to propose that via PR.

[abenhamdine]

I think it's more complex to handle all diacritics, and it would be better to not reinvent the wheel : for example, this package is well maintained and tested https://github.com/motss/normalize-diacritics
It's why I would prefer to be able to apply a custom normalization function

[abenhamdine]

IMHO, it would be probably a good idea to allow to pass a searchCompareValues function, to be able to fully customize the search.
Thus one could implements fuzzy search, custom normalization, etc...

[gnbm]

Hi @nikolasr200
Did you get the chance to test the alternatives I suggested?
I would like more input before moving forward with a PR.
Cheers

[nikolasr200]

Hi @gnbm, sorry i didn't have the time to do any tests so far. I believe your approach is to the right direction, and i ll try to test it asap.

[gnbm]

I think it's more complex to handle all diacritics, and it would be better to not reinvent the wheel : for example, this package is well maintained and tested https://github.com/motss/normalize-diacritics It's why I would prefer to be able to apply a custom normalization function

Feel free to add that to the library via PR so that we can avoid breaking changes and @sa-si-dev might be able to review it