It appears that in-progress workers can be killed when scaling down. E.g., imagine there are 5 jobs and 5 workers processing them. Each of the 5 workers picks up a job, and now there are none left in the queue. If these jobs take some time, but one job finishes finishes before the rest, it will run its after_perform_scale_down method and see there are no jobs remaining. It does not take into account the running jobs, so Scaler.workers is set to 0. This kills the other 4 jobs that are in progress.
It appears that we need to take into account the count of running jobs as well as the ones in the queue. But, I don't think heroku is smart enough to kill idle workers before working ones. So it appears you can only really reduce the number of workers without data loss when the number of active workers is 1. That worker would be the one that is in the after_perform_scale_down method, so its work is already done.