What is the recommended hardware to run highly available production setup about sharp

[lovell]

with 8 cores each

This looks like a CPU with 4 physical cores.

pre-processing each image in 5 different versions, all images are in avif

The bottleneck will almost certainly be AVIF encoding (via libaom).

At a guess the problem may be too many (hyper-)threads. You'll need to provide a complete, standalone code sample and images that allows someone else to both reproduce and profile if you'd like help optimising this further.

[woss]

Uh, a common language mistake about the cores.

Unfortunately, I cannot extract the example from the source code, it is not public. But the fact that the avif might be the slow part is already a start.

I assume jpg is faster?

At a guess the problem may be too many (hyper-)threads

What do you mean by this?

[lovell]

I assume jpg is faster?

Yes, usually significantly, the degree to which can depend on the type of images you're processing.

At a guess the problem may be too many (hyper-)threads

What do you mean by this?

Without sample code and images to reproduce it's hard to say, but it's possible the overhead of thread management and/or CPU cache thrashing might be slowing things. Sometimes reducing concurrency and/or parallelism can help improve throughput.