Analyse text-similarity to filter out duplicates on other platforms or by other recruiters about re-employment-kraken

[uschtwill]

@isensee-bastian I've been thinking about this, and it could really be the killer feature of the kraken. I think many people must be just as annoyed with all the duplicates on these platforms, as we are. And a way to cut through that noise would probably add significant value and drive some adoption.

I've also been thinking about how to do it...

• Given that we do have a real DB (#8)
• We could actually visit the URL to the listing, which we are so far only copying (the 'href')
• We then copy the listing's text (introducing a new per-strategy selector)
• Load into memory all listings of the last two days (from the database)
• Calculate text similarity values for the listing at hand versus all the listings in memory
• If none of the indices is higher than the threshold, write it to the DB as a new entry
• If it passes the threshold and is thus identified as the same listing, amend it to an "also-seen-on" list/array-field on the already existing entry

This would be nice for the board notifiers/UIs, because you get a list of sites, where this listing has been posted. This allows you to then go to and apply on the site you like most (e.g. freelance.de instead of freelancermap.de). If we just drop duplicates silently, then we lose the value of this information.

On the other hand, this is something that only works for the board UIs. We'd have to see how we can wring the most value out of this for notifiers like Telegram or Slack.

[isensee-bastian]

@uschtwill I totally agree. Manually identifying duplicates requires time and mental energy that could be spent in a better way.

The steps you listed make total sense. I think it is a good blueprint to follow for implementation.

About the tracking of duplicates: You are right about the value of storing multiple URLs per project. It helps to have a choice for applying. Moreover, by seeing the number of duplicates, one can roughly estimate how high or low the chances are for winning a project.
True, we need a different concept for notifying about duplicates in messengers vs board apps.