New TextSearch in 3.38 yields bad results about sanity

[rexxars]

We've released v3.39.1 which rolls back the new search, for now. We will have a deeper look at these issues and reintroduce it when we feel it is sufficiently tested.

[juice49]

Hey @jorngeorg, thank you for reporting this. Would you be able to share the API request that's produced for this search query? You can identify it by filtering your network logs on textsearch. Sharing as a curl request would be perfect, but you should redact your API token.

[Ojay]

We've been experiencing a few issues too, the main one is sometimes an obvious result just won't appear.

For example, we have an entry titled: JAX BEACH '64: YOU WIN THIS ONE, JOHNNY LAW!
Yesterday when searching "Jax Beach" it wasn't returning. This morning it's the second result. This has been the case with a few other items too. I haven't managed to find a pattern to it yet, though it felt like some behind-the-scenes indexing hasn't finished? Who knows.

[jorngeorg]

@juice49 Thanks for getting back to me, Ash. Is this the right one?

curl 'https://cvvjxa4t.api.sanity.io/v2024-03-12/data/textsearch/production_new?tag=sanity.studio.search.global' \
  -H 'accept: application/json' \
  -H 'accept-language: en-US,en;q=0.9' \
  -H 'content-type: application/json' \
  -H 'cookie: sanitySession=API-TOKEN' \
  -H 'dnt: 1' \
  -H 'origin: http://localhost:3333' \
  -H 'priority: u=1, i' \
  -H 'sec-ch-ua: "Not-A.Brand";v="99", "Chromium";v="124"' \
  -H 'sec-ch-ua-mobile: ?0' \
  -H 'sec-ch-ua-platform: "macOS"' \
  -H 'sec-fetch-dest: empty' \
  -H 'sec-fetch-mode: cors' \
  -H 'sec-fetch-site: cross-site' \
  -H 'user-agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/124.0.0.0 Safari/537.36' \
  --data-raw '{"query":{"string":"jeg savner gro*"},"filter":"_type in $__types","params":{"__types":["media.tag","article","blogArticle","bydel","documentFile","frontPage","fylke","issue","kommune","lag","legacyArticle","mediaLink","page","partiet","person"]},"types":{"media.tag":{"weights":{"name.source":0}},"article":{"weights":{"title":10,"slug.source":0}},"blogArticle":{"weights":{"title":10,"slug.source":0}},"bydel":{"weights":{"name":10,"slug.source":0}},"documentFile":{"weights":{"title":10}},"frontPage":{"weights":{"title":10}},"fylke":{"weights":{"name":10,"slug.source":0}},"issue":{"weights":{"title":10,"slug.source":0}},"kommune":{"weights":{"name":10,"slug.source":0}},"lag":{"weights":{"title":10,"slug.source":0}},"legacyArticle":{"weights":{"title":10}},"mediaLink":{"weights":{"title":10}},"page":{"weights":{"title":10,"slug.source":0}},"partiet":{"weights":{"title":10}}},"order":[{"attribute":"_updatedAt","direction":"desc"}],"includeAttributes":["_id","_type"],"limit":1000}'

[juice49]

@jorngeorg that's great, thank you!

@Ojay thank you reporting this. Do you know whether a draft has existed for that document—and whether a draft of it has been published—in the past day?

[Ojay]

I'll find out from the boss man for you @juice49 - I think it was published though.

I'm experiencing similar to @jorngeorg too. Searching but for the work of kahanamoku, we would all have perished returns the item containing that in the title as the 18th result...

Here's the request copied as curl for you. Thanks so much for digging into this!

curl 'https://f1hjpcr4.api.sanity.io/v2024-03-12/data/textsearch/production?tag=sanity.studio.search.global' \
  -H 'accept: application/json' \
  -H 'accept-language: en-GB,en;q=0.6' \
  -H 'content-type: application/json' \
  -H 'origin: http://localhost:3333' \
  -H 'sec-ch-ua: "Brave";v="123", "Not:A-Brand";v="8", "Chromium";v="123"' \
  -H 'sec-ch-ua-mobile: ?0' \
  -H 'sec-ch-ua-platform: "macOS"' \
  -H 'sec-fetch-dest: empty' \
  -H 'sec-fetch-mode: cors' \
  -H 'sec-fetch-site: cross-site' \
  -H 'sec-gpc: 1' \
  -H 'user-agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/123.0.0.0 Safari/537.36' \
  --data-raw '{"query":{"string":"but for the work of kahanamoku, we would all have perished*"},"filter":"_type in $__types","params":{"__types":["media.tag","eos","hos","hos_chapter","contest","interview","video","books","surfboard","page","page-donors","subscribePage","feature","joint","images","author","latest","surfer","subscriptions","aboutPage","contactPage","contact","donation","gift","todo","headerSettings","generalSettings","gridSettings","latestSettings","homepageSettings"]},"types":{"media.tag":{"weights":{"name.source":0}},"eos":{"weights":{"title":10,"slug.source":0,"meta.primaryName":0,"meta.index":0,"meta.gender":0,"meta.peopleDates.birthYear":0,"meta.peopleDates.deathYear":0,"meta.peopleDates.birthYear2":0,"meta.peopleDates.deathYear2":0,"meta.peopleDates.birthYear3":0,"meta.peopleDates.deathYear3":0,"meta.peopleDates.birthYear4":0,"meta.peopleDates.deathYear4":0,"meta.continent1":0,"meta.nation1":0,"meta.states1":0,"meta.continent2":0,"meta.nation2":0,"meta.states2":0,"media.muxPlaybackId":0,"media.quotePosition":0,"media.imageGrid[].captionPosition":0}},"hos":{"weights":{"title":10,"slug.source":0,"meta.primaryName":0,"meta.index":0,"meta.gender":0,"meta.peopleDates.birthYear":0,"meta.peopleDates.deathYear":0,"meta.peopleDates.birthYear2":0,"meta.peopleDates.deathYear2":0,"meta.peopleDates.birthYear3":0,"meta.peopleDates.deathYear3":0,"meta.peopleDates.birthYear4":0,"meta.peopleDates.deathYear4":0,"meta.continent1":0,"meta.nation1":0,"meta.states1":0,"meta.continent2":0,"meta.nation2":0,"meta.states2":0,"media.muxPlaybackId":0,"media.quotePosition":0,"media.imageGrid[].captionPosition":0}},"hos_chapter":{"weights":{"title":10,"slug.source":0,"meta.primaryName":0,"meta.index":0,"meta.gender":0,"meta.peopleDates.birthYear":0,"meta.peopleDates.deathYear":0,"meta.peopleDates.birthYear2":0,"meta.peopleDates.deathYear2":0,"meta.peopleDates.birthYear3":0,"meta.peopleDates.deathYear3":0,"meta.peopleDates.birthYear4":0,"meta.peopleDates.deathYear4":0,"meta.continent1":0,"meta.nation1":0,"meta.states1":0,"meta.continent2":0,"meta.nation2":0,"meta.states2":0}},"contest":{"weights":{"title":10,"slug.source":0,"meta.primaryName":0,"meta.index":0,"meta.gender":0,"meta.peopleDates.birthYear":0,"meta.peopleDates.deathYear":0,"meta.peopleDates.birthYear2":0,"meta.peopleDates.deathYear2":0,"meta.peopleDates.birthYear3":0,"meta.peopleDates.deathYear3":0,"meta.peopleDates.birthYear4":0,"meta.peopleDates.deathYear4":0,"meta.continent1":0,"meta.nation1":0,"meta.states1":0,"meta.continent2":0,"meta.nation2":0,"meta.states2":0,"media.muxPlaybackId":0,"media.quotePosition":0,"media.imageGrid[].captionPosition":0}},"interview":{"weights":{"title":10,"slug.source":0,"meta.primaryName":0,"meta.index":0,"meta.gender":0,"meta.peopleDates.birthYear":0,"meta.peopleDates.deathYear":0,"meta.peopleDates.birthYear2":0,"meta.peopleDates.deathYear2":0,"meta.peopleDates.birthYear3":0,"meta.peopleDates.deathYear3":0,"meta.peopleDates.birthYear4":0,"meta.peopleDates.deathYear4":0,"meta.continent1":0,"meta.nation1":0,"meta.states1":0,"meta.continent2":0,"meta.nation2":0,"meta.states2":0,"media.muxPlaybackId":0,"media.quotePosition":0,"media.imageGrid[].captionPosition":0}},"video":{"weights":{"title":10,"slug.source":0,"meta.primaryName":0,"meta.index":0,"meta.gender":0,"meta.peopleDates.birthYear":0,"meta.peopleDates.deathYear":0,"meta.peopleDates.birthYear2":0,"meta.peopleDates.deathYear2":0,"meta.peopleDates.birthYear3":0,"meta.peopleDates.deathYear3":0,"meta.peopleDates.birthYear4":0,"meta.peopleDates.deathYear4":0,"meta.continent1":0,"meta.nation1":0,"meta.states1":0,"meta.continent2":0,"meta.nation2":0,"meta.states2":0,"media.muxPlaybackId":0,"media.quotePosition":0,"media.imageGrid[].captionPosition":0}},"books":{"weights":{"title":10,"slug.source":0,"meta.primaryName":0,"meta.index":0,"meta.gender":0,"meta.peopleDates.birthYear":0,"meta.peopleDates.deathYear":0,"meta.peopleDates.birthYear2":0,"meta.peopleDates.deathYear2":0,"meta.peopleDates.birthYear3":0,"meta.peopleDates.deathYear3":0,"meta.peopleDates.birthYear4":0,"meta.peopleDates.deathYear4":0,"meta.continent1":0,"meta.nation1":0,"meta.states1":0,"meta.continent2":0,"meta.nation2":0,"meta.states2":0,"media.muxPlaybackId":0,"media.quotePosition":0,"media.imageGrid[].captionPosition":0}},"surfboard":{"weights":{"title":10,"slug.source":0,"meta.primaryName":0,"meta.index":0,"meta.gender":0,"meta.peopleDates.birthYear":0,"meta.peopleDates.deathYear":0,"meta.peopleDates.birthYear2":0,"meta.peopleDates.deathYear2":0,"meta.peopleDates.birthYear3":0,"meta.peopleDates.deathYear3":0,"meta.peopleDates.birthYear4":0,"meta.peopleDates.deathYear4":0,"meta.continent1":0,"meta.nation1":0,"meta.states1":0,"meta.continent2":0,"meta.nation2":0,"meta.states2":0,"media.muxPlaybackId":0,"media.quotePosition":0,"media.imageGrid[].captionPosition":0}},"page":{"weights":{"title":10,"slug.source":0,"content[].image.captionPosition":0}},"page-donors":{"weights":{"title":10,"slug.source":0,"content[].image.captionPosition":0}},"subscribePage":{"weights":{"title":10}},"feature":{"weights":{"title":10,"slug.source":0,"meta.primaryName":0,"meta.index":0,"meta.gender":0,"meta.peopleDates.birthYear":0,"meta.peopleDates.deathYear":0,"meta.peopleDates.birthYear2":0,"meta.peopleDates.deathYear2":0,"meta.peopleDates.birthYear3":0,"meta.peopleDates.deathYear3":0,"meta.peopleDates.birthYear4":0,"meta.peopleDates.deathYear4":0,"meta.continent1":0,"meta.nation1":0,"meta.states1":0,"meta.continent2":0,"meta.nation2":0,"meta.states2":0,"media.muxPlaybackId":0,"media.quotePosition":0,"media.imageGrid[].captionPosition":0}},"joint":{"weights":{"title":10,"slug.source":0,"meta.primaryName":0,"meta.index":0,"meta.gender":0,"meta.peopleDates.birthYear":0,"meta.peopleDates.deathYear":0,"meta.peopleDates.birthYear2":0,"meta.peopleDates.deathYear2":0,"meta.peopleDates.birthYear3":0,"meta.peopleDates.deathYear3":0,"meta.peopleDates.birthYear4":0,"meta.peopleDates.deathYear4":0,"meta.continent1":0,"meta.nation1":0,"meta.states1":0,"meta.continent2":0,"meta.nation2":0,"meta.states2":0,"media.muxPlaybackId":0,"media.quotePosition":0,"media.imageGrid[].captionPosition":0}},"images":{"weights":{"slug.source":0}},"author":{"weights":{"name":10,"slug.source":0}},"latest":{"weights":{"title":10}},"surfer":{"weights":{"name":10}},"subscriptions":{"weights":{"title":10}},"aboutPage":{"weights":{"title":10,"staff[].headshot.captionPosition":0,"creators[].headshot.captionPosition":0}},"contactPage":{"weights":{"title":10,"support[].headshot.captionPosition":0}},"contact":{"weights":{"title":10}},"donation":{"weights":{"title":10,"slug.source":0}},"gift":{"weights":{"title":10,"slug.source":0}},"todo":{"weights":{"title":10}}},"order":[{"attribute":"_updatedAt","direction":"desc"}],"includeAttributes":["_id","_type"],"limit":1000}'```

[juice49]

@Ojay thank you for providing an example!