Backup fails with 'file content does not match hash' about rest-server

[MichaelEischer]

file content does not match hash

That error is an integrity check to ensure that pack files uploaded by restic are not corrupted (totally unrelated to NTP). It can occur if either the backup client encounters a bitflip, the data is corrupted while transferring it to the rest-server or the rest-server encounters a bitflip while verifying the pack file. Bitflips are usually caused by a bit flips in memory or in the CPU while processing data.

the client returns 'Fatal: unable to save snapshot: server response unexpected: 400 Bad Request (400)'

Could you provide the log output from the client, in particular the retrying after lines?

Judging from the debug log, I'd expect the client to have retried the failed upload multiple times, however, without the client logs this is just a guess.

Assuming my guess is correct, this means that the bitflip likely occurred at the client which tried to upload the corrupted data multiple times. If this is just a one time event, you can probably ignore it. However, you must run restic check --read-data to ensure that the repository has not been corrupted.

[adamklaff]

Thanks, that makes sense. I wonder if I need to do some other integrity checks on the client. This happens with almost every backup attempt.

Here's output from the client during a failure:

Save(<data/992e4e7839>) returned error, retrying after 552.330144ms: server response unexpected: 400 Bad Request (400)
Save(<data/992e4e7839>) returned error, retrying after 1.080381816s: server response unexpected: 400 Bad Request (400)
Save(<data/992e4e7839>) returned error, retrying after 1.31013006s: server response unexpected: 400 Bad Request (400)
Save(<data/992e4e7839>) returned error, retrying after 1.582392691s: server response unexpected: 400 Bad Request (400)
Save(<data/992e4e7839>) returned error, retrying after 2.340488664s: server response unexpected: 400 Bad Request (400)
Save(<data/992e4e7839>) returned error, retrying after 4.506218855s: server response unexpected: 400 Bad Request (400)
Save(<data/992e4e7839>) returned error, retrying after 3.221479586s: server response unexpected: 400 Bad Request (400)
Save(<data/992e4e7839>) returned error, retrying after 5.608623477s: server response unexpected: 400 Bad Request (400)
Save(<data/992e4e7839>) returned error, retrying after 7.649837917s: server response unexpected: 400 Bad Request (400)
Save(<data/992e4e7839>) returned error, retrying after 15.394871241s: server response unexpected: 400 Bad Request (400)
Fatal: unable to save snapshot: server response unexpected: 400 Bad Request (400)

Here is the server output:

POST /backup/data/eacb1db8d6fb6fc83ab30e2e4e1e369d29a53970312465a8103e7b72903f5364
saveBlob()
POST /backup/data/992e4e783954618e70d415aa8e7420ccda42f11c29898bd9200c1020d4c97d0c
saveBlob()
POST /backup/data/2c8e4b6e70a82a10b00bc7cf8edec82957a141c4a47a9ce79a3d9c96633e7ed3
saveBlob()
POST /backup/data/a8f6daa5def316dec67a9c0a2b9b534f0394308b5e1ef92991b35f891f7206f7
saveBlob()
POST /backup/data/a2e0f1e183c4815568ff29ff8d9128f87529bfadbf1d45b6c1a99453c41f41cd
saveBlob()
file content does not match hash
DELETE /backup/data/992e4e783954618e70d415aa8e7420ccda42f11c29898bd9200c1020d4c97d0c
deleteBlob()
DELETE /backup/locks/6ccd40e283d859deef7e2412075624db975bf40d589f8777da9775cf478d7f7d
deleteBlob()
unexpected EOF

[MichaelEischer]

Please run some stress tests like memtest86 / prime95 to make sure the system is working correctly. That frequent data corruption usually indicates a hardware problem.

[MichaelEischer]

Closing as I don't think there's anything we can do here on the rest-server side.