andreacorbellini / rust-atomic-write-file

Atomic Write File

This is a Rust crate that offers functionality to write and overwrite files atomically, that is: without leaving the file in an intermediate state. Either the new contents of the files are written to the filesystem, or the old contents (if any) are preserved.

This crate implements two main structs: AtomicWriteFile and OpenOptions, which mimic the standard std::fs::File and std::fs::OpenOptions as much as possible.

This crate supports all major platforms, including: Unix systems, Windows, and WASI.

Motivation and Example

Consider the following snippet of code to write a configuration file in JSON format:

use std::io::Write;
use std::fs::File;

let mut file = File::options()
                    .write(true)
                    .create(true)
                    .open("config.json")?;

writeln!(file, "{{")?;
writeln!(file, "  \"key1\": \"value1\",")?;
writeln!(file, "  \"key2\": \"value2\"")?;
writeln!(file, "}}")?;

This code opens a file named config.json, truncates its contents (if the file already existed), and writes the JSON content line-by-line.

If the code is interrupted before all of the writeln! calls are completed (because of a panic, or a signal is received, or the process is killed, or a filesystem error occurs), then the file will be left in a broken state: it will not contain valid JSON data, and the original contents (if any) will be lost.

AtomicWriteFile solves this problem by placing the new contents into the destination file only after it has been completely written to the filesystem. The snippet above can be rewritten using AtomicWriteFile instead of File as follows:

use std::io::Write;
use atomic_write_file::AtomicWriteFile;

let mut file = AtomicWriteFile::options()
                               .open("config.json")?;

writeln!(file, "{{")?;
writeln!(file, "  \"key1\": \"value1\",")?;
writeln!(file, "  \"key2\": \"value2\"")?;
writeln!(file, "}}")?;

file.commit()?;

Note that this code is almost the same as the original, except that it now uses AtomicWriteFile instead of File and there's an additional call to commit().

If the code is interrupted early, before the call to commit(), the original file config.json will be left untouched. Only if the new contents are fully written to the filesystem, config.json will get them.

Documentation

Reference, examples, internal details, and limitations are all available on docs.rs/atomic-write-file.

rust-atomic-write-file's Issues

Automatically fall back to different behavior when `O_TMPFILE` doesn't work

Apparently overlayfs, used by container software, doesn't support unnamed tempfiles: launchbadge/sqlx#2908

It would be nice to be able to easily use O_TMPFILE when it is supported, and fall back to a different behavior when it isn't. Currently, there's just the Cargo feature (and a library can't be sure that it's disabled, since some other package in an application's dependency tree might activate the it).

Rename failures are not propagated to caller of .commit()

1. make a file manually echo foo > filename
2. make it immutable chattr +i filename
3. try to write over it with this library

    	let mut file = AtomicWriteFile::options().open("filename")?;
	writeln!(file, "{}", contents)?;
	file.commit()?;

4. exit without error
5. tmp file still exists and file did not get overwritten

fsync parent also after rename

After reading more I found it's recommended to fsync parent also to be sure data is persisted ok.
You can see here more https://www.quora.com/Linux/When-should-you-fsync-the-containing-directory-in-addition-to-the-file-itself
Also I see other implementations are doing it https://github.com/untitaker/rust-atomicwrites/blob/master/src/lib.rs#L200

It's not a bit deal, I can fsync parent after commit but would be nice to have this included

Documentation link in readme is broken

Hey, thanks for building this awesome crate! I have been using NamedTempFile from the tempfile crate for this purpose before but this seems more convenient and robust from the how-it-works description in the docs.

There's a small problem with the readme, the docs badge link is broken.

`os error 5` when open `AtomicWriteFile` with `tempfile`

It returns an error when commit, opening with tempfile::NamedTempFile.path().
Access is denied. (os error 5)

dependencies

├── anyhow v1.0.81
├── atomic-write-file v0.1.3
│   └── rand v0.8.5
│       ├── rand_chacha v0.3.1
│       │   ├── ppv-lite86 v0.2.17
│       │   └── rand_core v0.6.4
│       │       └── getrandom v0.2.14
│       │           └── cfg-if v1.0.0
│       └── rand_core v0.6.4 (*)
└── tempfile v3.10.1
    ├── cfg-if v1.0.0
    ├── fastrand v2.0.2
    └── windows-sys v0.52.0
        └── windows-targets v0.52.4
            └── windows_x86_64_msvc v0.52.4

Platform

Windows

Code

use anyhow::Result;
use atomic_write_file::AtomicWriteFile;
use tempfile::NamedTempFile;

fn main() -> Result<()> {
    let file = NamedTempFile::new()?;
    let path = file.path();
    let atomic = AtomicWriteFile::open(path)?;
    atomic.commit()?;
    Ok(())
}

Output

Error: 拒绝访问。 (os error 5)

Missing fsync() call after linkat()

rust-atomic-write-file/src/imp/unix/linux.rs

Line 47 in 1713ccc

match linkat(

A call to fsync() on a O_TMPFILE does nothing. Now, the anonymous file is given a temporary name then renamed to its final name, without an fsync() call in between. A power failure following the final rename may result in data loss -- the target file now contains no data, filesize is zero.

Take a look at the btrfs source code in the kernel, where the no-op behavior of fsync() is described: https://github.com/torvalds/linux/blob/026e680b0a08a62b1d948e5a8ca78700bfac0e6e/fs/btrfs/tree-log.c#L2422