apple / swift-system

Low-level system calls and types for Swift

Swift System

Swift System provides idiomatic interfaces to system calls and low-level currency types. Our vision is for System to act as the single home for low-level system interfaces for all supported Swift platforms.

Multi-platform not Cross-platform

System is a multi-platform library, not a cross-platform one. It provides a separate set of APIs and behaviors on every supported platform, closely reflecting the underlying OS interfaces. A single import will pull in the native platform interfaces specific for the targeted OS.

Our immediate goal is to simplify building cross-platform libraries and applications such as SwiftNIO and SwiftPM. System does not eliminate the need for #if os() conditionals to implement cross-platform abstractions, but it does make it safer and more expressive to fill out the platform-specific parts.

Usage

import SystemPackage

let message: String = "Hello, world!" + "\n"
let path: FilePath = "/tmp/log"
let fd = try FileDescriptor.open(
  path, .writeOnly, options: [.append, .create], permissions: .ownerReadWrite)
try fd.closeAfter {
  _ = try fd.writeAll(message.utf8)
}

API documentation

Adding SystemPackage as a Dependency

To use the SystemPackage library in a SwiftPM project, add the following line to the dependencies in your Package.swift file:

.package(url: "https://github.com/apple/swift-system", from: "1.0.0"),

Finally, include "SystemPackage" as a dependency for your executable target:

let package = Package(
    // name, platforms, products, etc.
    dependencies: [
        .package(url: "https://github.com/apple/swift-system", from: "1.0.0"),
        // other dependencies
    ],
    targets: [
        .target(name: "MyTarget", dependencies: [
            .product(name: "SystemPackage", package: "swift-system"),
        ]),
        // other targets
    ]
)

Source Stability

The Swift System package is source stable. The version numbers follow Semantic Versioning -- source breaking changes to public API can only land in a new major version.

Contributing

Before contributing, please read CONTRIBUTING.md.

LICENSE

See LICENSE for license information.

swift-system's Issues

API Request: Splitting `POSIXPath` and `WindowsPath` out of `FilePath`

FilePath is good, but we sometimes need to process POSIX paths or Windows paths specifically, regardless of the current platform. eg. when we create our own file system, we may want it to stick to POSIX paths for various reasons: cross-platform consistency, simplicity, tool compatibility, etc.

The biggest problem appears to be, FilePath is implemented as a whole and the Windows and POSIX parts of implementation are highly coupled. Here is the ideal layout:

/// Unify `FilePath` APIs
public protocol FilePathProtocol { … }

/// Default implementation
extension FilePathProtocol { … }

/// POSIX path struct
public struct POSIXPath: FilePathProtocol { … }

/// Windows path struct
public struct WindowsPath: FilePathProtocol { … }

/// The current `FilePath`
#if os(Windows)
public typealias FilePath = WindowsPath
#else
public typealias FilePath = POSIXPath
#endif

However, these two new types do need to rely on the same set of Root and Component, then we may also need:

public protocol FilePathRootProtocol { … }
public extension FilePathRootProtocol { … }
public extension POSIXPath {
    public struct Root: FilePathRootProtocol { … }
}
public extension WindowsPath {
    public struct Root: FilePathRootProtocol { … }
}

public protocol FilePathComponentProtocol { … }
public extension FilePathComponentProtocol { … }
public extension POSIXPath {
    public struct Component: FilePathComponentProtocol { … }
}
public extension WindowsPath {
    public struct Component: FilePathComponentProtocol { … }
}

public protocol FilePathProtocol {
    associatedtype Root: FilePathRootProtocol
    associatedtype Component: FilePathComponentProtocol
}

I’d like to collect some feedback for this change & ideas on how to implement this in the correct way.

FileDescriptor creation do not respect Group Write permissions

When open file descriptor for write and set option .create created file does not have group write permission. Checked with all group permissions .groupWrite, .groupReadWrite, .groupReadWriteExecute

In case below, expect to have permissions 0020, but get 0000

FileDescriptor.open(filePath,
                                  .writeOnly,
                                  options: [.create],
                                  permissions: [.groupWrite])

System: MacOS 10.15.7
Swift toolchain: Swift Development Snapshot 2021-02-16
SwiftPackage revision 2e9c1a71185c828416751283b40697725da550b6

Does not compile for visionOS

Compiling for visionOS in Xcode 15.2 on 23C71 yields an enormous number of build errors (180+)

Build fails in release with: ld: Undefined symbols: static (extension in SystemPackage):SystemPackage._StrSlice.== infix(A, A) -> Swift.Bool

Any package which depends on swift-system and uses the == on FilePath.Component will fail to build in release mode (debug is fine).

Consider this example program

import SystemPackage

print(FilePath("/x").lastComponent! == FilePath("/x").lastComponent!)

with the following dependencies/targets

    dependencies: [
        .package(url: "https://github.com/apple/swift-system.git", from: "1.0.0"),
    ],
    targets: [
        .executableTarget(
            name: "repro",
            dependencies: [
                .product(name: "SystemPackage", package: "swift-system"),
            ]
        ),
    ]

debug works

$ swift run 
Building for debugging...
Build complete! (0.34s)
true

release fails

$ swift run -c release
Building for production...
error: link command failed with exit code 1 (use -v to see invocation)
ld: Undefined symbols:
  static (extension in SystemPackage):SystemPackage._StrSlice.== infix(A, A) -> Swift.Bool, referenced from:
      _repro_main in main.swift.o
clang: error: linker command failed with exit code 1 (use -v to see invocation)
error: fatalError
[0/1] Linking repro

Also reproes in Xcode.

This is Xcode 15A221 but also reproduces on anything else I tried.

Frequently Asked Questions

Why import SystemPackage instead of import System?

It's a workaround for current compiler limitations. If you have a binary System module available, as we do on Darwin and might have on other platforms in the future, then a System module from a package would conflict with that.

In the future the compiler might support module namespaces, which would allow us to name the module System in a package namespace distinct from one in a SDK or toolchain namespace. Alternatively or additionally, we might ship a binary System module for all platforms, in which case the package build would exist strictly for backwards deployment or prototyping.

Why are @available directives commented out?

This is due to language/tooling issues. We'd like the ability to add availability based on conditional compilation for ABI-stable builds of Swift System while not adding availability for package builds (i.e. a source dependency). For now, we include, but comment out, the availability declarations matching what has already been shipped in binary releases.

Why so many struct and no enums?

Swift struct gives us control over the layout (binary representation) of our types to an extent that enums do not. This is why types such as Errno and FileDescriptor.SeekOrigin are nominally structs, even though their API is more enum-like.

Swift enums are also exhaustive. This allows for exhaustive switching in user code, but there is no way to retroactively add, rename, alias, or fix cases. Swift’s open enums allows for extensibility, but can no longer be exhaustively switched over, providing effectively the same API experience as our enum-like nominal structs. Without layout control, there are fewer tools to change or extend the API/ABI surface with either enum or open enum, since the cases are the representation.

Isolate Windows codebase from UN*X systems

We keep breaking the Windows port whenever we land a new feature for Darwin and/or Linux. This needs to stop.

There is absolutely no reason the Windows port of swift-system needs to be built from the same sources (or provide the same APIs) as UN*X systems; in fact, this is detrimental to the objective of this project and only serves to make it more difficult/error-prone to maintain this package. To a lesser extent, this is also true of Linux vs Darwin.

• Set up platform-specific directories under Sources/ and Tests/. Explicitly exclude source files that wrap POSIX/Darwin/Linux syscalls from the Windows build, and vice versa.
• For cases where code sharing across platforms actually makes sense (i.e. some POSIX calls on Linux & Darwin), set up a POSIX/ platform directory, and include the sources therein in both the Darwin and Linux builds. If we want to keep some parts of the implementation of FilePath shared across all platforms, then move those parts into a shared/ or common/ subdirectory.
• Just like the actual build, tests are platform specific, and they need to be configured accordingly. The test suite in this package is currently completely broken on Windows.
• Never, ever, pretend that Windows implements POSIX system calls.

Consider escaping invalid Unicode in `FilePath.debugDescription`

FilePath.debugDescription does encoding correction, but we could instead just escape invalid Unicode.

Foundation additions

Should the open-source Swift System have a new module, with the same Foundation additions as in the closed-source Swift System?

// MARK: - Foundation Additions

import Foundation

// Available when Foundation is imported with System
@available(macOS 11.0, iOS 14.0, watchOS 7.0, tvOS 14.0, *)
extension FilePath {

    /// Creates a file path from a URL
    ///
    /// The result is nil if `url` doesn't refer to a local file.
    public init?(_ url: URL)
}

// Available when Foundation is imported with System
@available(macOS 11.0, iOS 14.0, watchOS 7.0, tvOS 14.0, *)
extension URL {

    /// Creates a URL from a file path by by copying and validating its UTF-8 contents.
    ///
    /// This initializer does not try to repair ill-formed UTF-8 code unit
    /// sequences. If any are found, the result of the initializer is `nil`.
    ///
    public init?(_ path: FilePath)

    /// Creates a URL from a file path by by copying and validating its UTF-8 contents.
    ///
    /// This initializer does not try to repair ill-formed UTF-8 code unit
    /// sequences. If any are found, the result of the initializer is `nil`.
    ///
    public init?(_ path: FilePath, isDirectory: Bool)
}

Should `FileDescriptor.closeAfter` provide a `FileDescriptor` to the invoked closure?

I've been wanting to write something like

let foo = try FileDescriptor
  .open("my-file", .readOnly)
  .closeAfter { fileDescriptor in
    /// do stuff with fileDescriptor
  }

Could you please remove the v-prefix from the version 0.0.2 tag?

The tag for version 0.0.2 now has a prefix of v, unlike the previous tag, which is now v0.0.2.

This will affect automatic updates of Renovate (https://github.com/apps/renovate).
Renovate will automatically create a PR from the GitHub tag as follows

-.package(url: "https://github.com/apple/swift-system", from: "0.0.1"),
+.package(url: "https://github.com/apple/swift-system", from: "v0.0.2"),

This causes an error in the Swift Package Manager.

...
[5/5] RUN cd ./swiftfiddle.com/_Packages/ && swift build -c release:
#8 1.044 /swiftfiddle.com/_Packages: error: manifest parse error(s):
#8 1.044 Invalid semantic version string 'v0.0.2'
...

If it's not important to add the v prefix, could you please revert to the previous format (like 0.0.2)?

Double-free with FilePath in Xcode 13 beta

Originally reported as https://bugs.swift.org/browse/SR-14895 by @Frizlab.

FilePath crashes (double-free) with the following code

import Foundation
import System

if #available(macOS 12.0, *) {
	let relative = FilePath("Versions")
	_ = FilePath("/a/b/c").appending(relative.components)
	print(relative)
}

Demo repo: https://gitlab.com/frizlab-demo-projects/pathcrash

It does not crash when using SystemPackage version 0.0.2 instead of System (either in release or debug compilation mode).

Configuration: macOS 12.0 Beta (21A5268h) with Xcode 13.0 beta (13A5155e)

FilePath is Decodable, but not from a String

it looks like FilePath has a conformance to Codable, but this is a synthesized conformance, and expects a keyed container with a property called _storage.

i don’t know if this is intentional, but it doesn’t seem very useful. FilePath should just be Decodable from a String.

Add TextOutputStream to FileDescriptor?

Would it make sense to add TextOutputStream conformance to FileDescriptor to support writing something like this:

print("An error occurred", to: FileDescriptor.standardError)

Hmm, I tried doing this:

print("File exists.", to: &FileDescriptor.standardError)       //  Cannot pass immutable value as inout argument: 'standardError' is a get-only property

extension
FileDescriptor : TextOutputStream
{
    public mutating func write(_ inString: String) {
        _ = try? FileDescriptor.standardError.writeAll(inString.utf8)
    }
}

Perhaps those would be better as public static members?

Better Way Of Defining Linux System Constants

So the Linux constants may vary from one architecture to another. Why not get the actual values from the Linux system itself? Incorporate a script like this into your build process for Linux, by including its output in the generated source file:

#/usr/bin/python3
#+
# Better way of defining constants for this
# <https://github.com/apple/swift-system/blob/main/Sources/System/LinuxPlatformConstants.swift>
# source file.
#-

import sys
import os
import subprocess
import tempfile
import shutil

consts_to_define = \
    (
        {
            "symbols" :
                [
                    "EPERM", "ENOENT", "ESRCH", "EINTR", "EIO", "ENXIO",
                    "E2BIG", "ENOEXEC", "EBADF", "ECHILD", "EAGAIN", "ENOMEM",
                    "EACCES", "EFAULT", "ENOTBLK", "EBUSY", "EEXIST", "EXDEV",
                    "ENODEV", "ENOTDIR", "EISDIR", "EINVAL", "ENFILE", "EMFILE",
                    "ENOTTY", "ETXTBSY", "EFBIG", "ENOSPC", "ESPIPE", "EROFS",
                    "EMLINK", "EPIPE", "EDOM", "ERANGE", "EDEADLK", "ENAMETOOLONG",
                    "ENOLCK", "ENOSYS", "ENOTEMPTY", "ELOOP", "EWOULDBLOCK", "ENOMSG",
                    "EIDRM", "ECHRNG", "EL2NSYNC", "EL3HLT", "EL3RST", "ELNRNG",
                    "EUNATCH", "ENOCSI", "EL2HLT", "EBADE", "EBADR", "EXFULL",
                    "ENOANO", "EBADRQC", "EBADSLT", "EDEADLOCK", "EBFONT", "ENOSTR",
                    "ENODATA", "ETIME", "ENOSR", "ENONET", "ENOPKG", "EREMOTE",
                    "ENOLINK", "EADV", "ESRMNT", "ECOMM", "EPROTO", "EMULTIHOP",
                    "EDOTDOT", "EBADMSG", "EOVERFLOW", "ENOTUNIQ", "EBADFD", "EREMCHG",
                    "ELIBACC", "ELIBBAD", "ELIBSCN", "ELIBMAX", "ELIBEXEC", "EILSEQ",
                    "ERESTART", "ESTRPIPE", "EUSERS", "ENOTSOCK", "EDESTADDRREQ", "EMSGSIZE",
                    "EPROTOTYPE", "ENOPROTOOPT", "EPROTONOSUPPORT", "ESOCKTNOSUPPORT",
                    "EOPNOTSUPP", "ENOTSUP", "EPFNOSUPPORT", "EAFNOSUPPORT", "EADDRINUSE",
                    "EADDRNOTAVAIL", "ENETDOWN", "ENETUNREACH", "ENETRESET", "ECONNABORTED",
                    "ECONNRESET", "ENOBUFS", "EISCONN", "ENOTCONN", "ESHUTDOWN", "ETOOMANYREFS",
                    "ETIMEDOUT", "ECONNREFUSED", "EHOSTDOWN", "EHOSTUNREACH", "EALREADY",
                    "EINPROGRESS", "ESTALE", "EUCLEAN", "ENOTNAM", "ENAVAIL", "EISNAM",
                    "EREMOTEIO", "EDQUOT", "ENOMEDIUM", "EMEDIUMTYPE", "ECANCELED",
                    "ENOKEY", "EKEYEXPIRED", "EKEYREVOKED", "EKEYREJECTED", "EOWNERDEAD",
                    "ENOTRECOVERABLE", "ERFKILL", "EHWPOISON",
                ],
            "defines" : [],
            "includes" : ["errno.h"],
            "mark" : "errno",
        },
        {
            "symbols" :
                [
                    "O_ACCMODE", "O_RDONLY", "O_WRONLY", "O_RDWR",
                    "O_CREAT", "O_EXCL", "O_NOCTTY", "O_TRUNC",
                    "O_APPEND", "O_NONBLOCK", "O_DSYNC", "FASYNC",
                    "O_DIRECT", "O_LARGEFILE", "O_DIRECTORY",
                    "O_NOFOLLOW", "O_NOATIME", "O_CLOEXEC",
                    "SEEK_SET", "SEEK_CUR", "SEEK_END",
                ],
            "defines" : ["_GNU_SOURCE", "_LARGEFILE64_SOURCE"],
            "includes" : ["sys/types.h", "sys/stat.h", "fcntl.h"],
            "mark" : "File Operations",
        },
    )

sys.stdout.write("#if os(Linux)\n")
for consts_entry in consts_to_define :
    tempdir = tempfile.mkdtemp(prefix = "swift-consts-")
    srcfilename = "defconsts.c"
    exefilename = "defconsts"
    srcfile = open(os.path.join(tempdir, srcfilename), "wt")
    for define in consts_entry["defines"] :
        srcfile.write("#define %s\n" % define)
    #end for
    for include in consts_entry["includes"] :
        srcfile.write("#include <%s>\n" % include)
    #end for
    srcfile.write \
      (
        "#include <stdio.h>\n"
        "\n"
        "int main(void)\n"
        "  {\n"
      )
    srcfile.write \
      (
            "    fputs(\"\\n// MARK: %s\\n\", stdout);\n"
        %
            consts_entry["mark"]
      )
    for symbol in consts_entry["symbols"] :
        srcfile.write("    fputs(\"\\n@_alwaysEmitIntoClient\\n\", stdout);\n")
        srcfile.write \
          (
                "    fprintf(stdout, \"internal var _%(symbol)s: CInt { %%d }\\n\", %(symbol)s);\n"
            %
                {"symbol" : symbol}
          )
    #end for
    srcfile.write \
      (
        "    return 0;\n"
        "  } /*main*/\n"
      )
    srcfile.close()
    subprocess.check_call \
      (
        args = ("gcc", "-o", exefilename, srcfilename),
        cwd = tempdir
      )
    sys.stdout.write \
      (
        subprocess.check_output
          (
            args = ("./" + exefilename,),
            cwd = tempdir,
            text = True
          )
      )
    shutil.rmtree(tempdir, ignore_errors = True)
#end for
sys.stdout.write("\n#endif\n")

I’m sure you can see how to extend it to add other groups of constants as needed.

Why `import SystemPackage` instead of `import System`?

Why is the target named SystemPackage instead of System?

Question: why are @available directives commented out?

Like e.g. here:

swift-system/Sources/System/FileDescriptor.swift

Line 17 in 8ffa04c

// @available(macOS 10.16, iOS 14.0, watchOS 7.0, tvOS 14.0, *)

Are these supposed to be commented out, and if so, why?

Update README for 1.4.0

• Update README for version 1.3.0
• Update copyright notices for files changed since 1.2.1

Add `FileDescriptor.openPipe`

I had mentally lumped that under process types and calls, but it's not actually blocked on any design work there. We can add this anytime and might make a good starter task.

Originally posted by @milseman in #16 (comment)

One thing that isn't completely clear to me is whether or not it would be useful to introduce a Pipe type (or FileDescriptor.Pipe so as not to cloud the top level namespace). I'm not sure how big of a deal this is, but this could prevent bugs like let (write, read) = try FileDescriptor.openPipe(), as well as allow us to implement close and closeAfter for both file descriptors at once. (Not sure how this affects Windows, @compnerd)

You are correct, this shouldn't be too difficult, at least on the Linux/macOS side of things. I'd be happy to take a pass if we agree on whether or not to include Pipe.

Assertion failure in FilePathParsing.swift

Assertion Failure: https://github.com/apple/swift-system/blob/main/Sources/System/FilePath/FilePathParsing.swift#L226
Test case: https://github.com/apple/swift-system/blob/main/Tests/SystemTests/FilePathTests/FilePathComponentsTest.swift#L247

> [IO.Path]::GetFullPath("//foo///bar/baz/")
\\foo\bar\baz\
> [IO.Path]::GetPathRoot("//foo///bar/baz/")
\\foo\bar
> [IO.Path]::GetPathRoot([IO.Path]::GetFullPath("//foo///bar/baz/"))
\\foo\bar

This also matches the Microsoft documentation on path normalization:

Normally, any path passed to a Windows API is (effectively) passed to the GetFullPathName function and normalized. There is one important exception: a device path that begins with a question mark instead of a period. Unless the path starts exactly with \?\ (note the use of the canonical backslash), it is normalized.

Originally posted by @compnerd in #101 (comment)

Add System.FilePath <-> SystemPackage.FilePath conversion?

It's quite awkward to work on an application where both System.FilePath and SystemPackage.FilePath are floating around.

I love that this package is available, but as things stand, I'm having to add the following code to my libraries to facilitate type conversions:

#if canImport(System)
  import System
  import SystemPackage

  @available(macOS 11, iOS 14, tvOS 14, watchOS 7, *)
  extension System.FilePath {

    /// Creates a `FilePath` (from the platform's `System` framework) from a `FilePath` (from the `swift-system` package).
    ///
    public init(_ packageFilePath: SystemPackage.FilePath) {
      self = packageFilePath.withCString { .init(cString: $0) }
    }
  }

  @available(macOS 11, iOS 14, tvOS 14, watchOS 7, *)
  extension SystemPackage.FilePath {

    /// Creates a `FilePath` (from the `swift-system` package) from a `FilePath` (from the platform's `System` framework).
    ///
    public init(_ sdkFilePath: System.FilePath) {
      self = sdkFilePath.withCString { .init(cString: $0) }
    }
  }
#endif

But this isn't really the kind of code I feel my library should be responsible for. It's really none of my business to bridge between these 2 libraries. Would it be acceptable to add these to the package distribution?

I understand that the compiler may provide better tools for dealing with this, and if/when that happens, these functions could be deprecated and removed at the next SemVer-major release. Until that time, adding these functions (and perhaps others for FileDescriptor) would make using the package distribution much more ergonomic.

Deprecated APIs are lacking availability information

I have a function which makes use of FilePath(cString:), so it can be used as far back as macOS 11/iOS 14.

Now, I'm getting a warning that that initializer has been deprecated (in favour of platformString:). That's fine, but I'd still like to make use of the old initializer on those older OSes. Unfortunately, even when I only use the deprecated initializer on a fallback path, I still get warnings:

I believe the reason for this is that the @available annotations do not include version information. It is possible to write something like:

@available(macOS, introduced: 11, deprecated: 12)

API Roadmap

This is meant to be a continuously-updated list of API candidates for System.

Criteria for Inclusion

System's primary aim is to be the place developers go to access native platform interfaces, presented in as Swifty a way as feasible. This positions System as a successor to Darwin/GlibC/SwiftWin32 and a “bedrock” upon which to enable better systems-level programming in Swift.

System is not aiming to be a cross-platform abstraction layer; there will be platform differences reflected in System's API. Of course, if a concept can be expressed the same way across platforms without harm or significant compromise to the developer's native experience on that platform, that's great. But, System's API should feel natural and native for the target platform.

System aims to provide API in 3 areas:

1. As-Swifty-as-we-reasonably-can direct access to system calls and types

If a system interface exists, that’s strong justification for providing access to it in System as faithfully as we can.

2. Common types for library API at the systems level

System hosts common types for systems level programming, enabling libraries built on top of System to use the same, well-crafted types in their API.

For example, many API take locations in the file system, for which FilePath provides a common expression.

3. High-value utility functionality and abstractions, where appropriate

System provides common utilities that users of System would otherwise find themselves having to reinvent. These usually have some combination of being pervasive, having obvious desired behavior, and being difficult/onerous to implement correctly.

For example, ensuring that a file descriptor is closed under all (non-pathological) conditions can be complex in the presence of error-handling, so System provides FileDescriptor.closeAfter.

This area has a higher bar for contribution, requiring more justification and thought. What is "obvious desired behavior" and "correct" can differ amongst users of System.

Roadmap

Released

• 0.0.1
  • syscalls: read, write, lseek, pread, pwrite, open, close
  • types: FilePath, FileDescriptor, Errno, FilePermissions
  • helpers: closeAfter, writeAll
• 0.0.2
  • FilePath Syntactic Operations
    • decompose a path, analyze a path, components view, mutate a path, etc
  • Standard file descriptors
    • static FileDescriptor.standardInput, FileDescriptor.standardOutput, FileDescriptor.standardError
  • dup/dup2
    • FileDescriptor.duplicate, with optional as target.

Merged

In Progress

• Strongly-typed process and signal wrappers
  • ProcessID, Signal, SignalSet, ProcessID.TaskInfo, ProcessID.ResourceUsageInfo
  • TODO: posix_spawn and execv functionality and interfaces
• Initial socket support
• Pipes

Sketches

Sketches are merge-worthy, quick-but-complete presentations of systems interfaces. They help to surface unknown-unknowns and can be easily adapted into a proposal. They are not necessarily named or expressed in their final form.

• fcntl
• sysconf, pathconf, fpathconf
• openat, fsync, sync

Starter tasks

• Add support for pipe

Near Term

Anything here could be added to System in the near term. Most of the functionality can be added by following existing API design patterns, though some will need new patterns.

• sleep capabilities (e.g. usleep)
• FilePath semantic (i.e. file-system interacting) operations
  • stat, chown, directory iteration, etc.
  • temporary files
• pthread low-level interfaces
• Further networking and sockets functionality
  • getaddrinfo, gethostent, etc.
• Environment variables
  • FilePath: ~ expansion, currentWorkingDirectory
• OSString-like abstraction
  • Null-terminated bytes on Unix, wchar or bytes on Windows (pending investigation)
• I/O events
  • kqueue/kevent for Darwin, epoll for Linux, APC (or something similar) for Windows
• exit handling
  • exit, atexit, etc
• tty
  • ioctl, etc
• Fleshing out FileDescriptor more
  • chmod, umask, etc.

Long Term (vague "blue-sky" hopes)

• moveonly File type
• high level Process type
• io_uring on Linux

Windows build is broken

D:\a\swift-url\swift-url\.build\checkouts\swift-system\Sources\System\FileOperations.swift:445:7: error: cannot find 'system_ftruncate' in scope

😐

I don't like it when my builds break because swift-system published a release without even testing it. Please never do that again.

Divine Intervention

Release package version 1.4.0

• Include the Sendable conformances (#115)
• Include Mach.Port (#129)
• Bump up toolchain requirement to 5.6, which supports concurrency (#141)
  • Clean up any conditional compilation branches that are no longer needed.
  • Move the Sendable conformances to the base type declarations.

Trunk compilation is broken with the Swift 5.7 compiler on linux

I reported this earlier on the offending pull, but filing an issue to make sure it is handled.

The error:

Sources/System/MachPort.swift:75:5: error: deinitializers may only be declared within a class or actor
    deinit {
    ^

strerror is not (guaranteed) to be thread safe

This package's Errno type is using strerror to produce error strings. For unknown error numbers strerror writes a string into a static buffer and returns it, but whether this static buffer is thread local is unspecified. Using strerror_r would guarantee thread safety even for unknown error codes.

FileDescriptor.open(_:_:options:permissions:retryOnInterrupt:) should print an error message when trapping on unspecified permissions

FileDescriptor.open(_:_:options:permissions:retryOnInterrupt:) traps if options.contains(.create) and permissions is nil. since its callers can throw, why not throw an error instead?

alternatively, if this isn’t appropriate for throws, it should be a fatalError instead of a precondition, so users don’t have to hunt through source code to figure out why FileDescriptor is crashing…

Compiler Crashes When Building For Windows or Linux

Compiler Crashes When Building For Windows or Linux

Table of Contents

• Compiler Crashes When Building For Windows or Linux
  • Reproduction Steps
  • Compilation Output
    • Linux Compilation Output
    • Windows Compilation Output

Created by gh-md-toc

• Swift Version: 5.3, 5.4-dev (Windows)
• MacOS Version: 11.1, 10.15 (GitHub Actions)
• Linux Version: Ubuntu 20.04, 18.04 (Docker)

Windows Swift version:

compnerd.org Swift version 5.4-dev (LLVM e2976fe639d1f50, Swift 769cfce6ed904e0)
Target: x86_64-unknown-windows-msvc

Mac Swift version:

Apple Swift version 5.3.2 (swiftlang-1200.0.45 clang-1200.0.32.28)
Target: x86_64-apple-darwin20.2.0

Reproduction Steps

Steps to reproduce:

• Install Docker to test on Linux
• Clone main branch of apple/swift-system
• Run swift build on Mac and note that compilation succeeds
• Run docker run -v $(pwd):/src -w /src -it swift:latest swift build to build on Linux and note that compilation fails

Compilation Output

Linux Compilation Output

Linux compilation output:

$ docker run -v $(pwd):/src -w /src -it swift:latest swift build
swift: /home/buildnode/jenkins/workspace/oss-swift-5.3-package-linux-ubuntu-18_04/swift/lib/SILGen/SILGenApply.cpp:4102: bool swift::Lowering::SILGenModule::isNonMutatingSelfIndirect(swift::SILDeclRef): Assertion `method->isNonMutating()' failed.
Stack dump:
0.	Program arguments: /usr/bin/swift -frontend -c /src/Sources/System/Errno.swift /src/Sources/System/FileDescriptor.swift /src/Sources/System/FileHelpers.swift /src/Sources/System/FileOperations.swift /src/Sources/System/FilePath/FilePath.swift /src/Sources/System/FilePath/FilePathComponentView.swift /src/Sources/System/FilePath/FilePathComponents.swift /src/Sources/System/FilePath/FilePathParsing.swift /src/Sources/System/FilePath/FilePathString.swift -primary-file /src/Sources/System/FilePath/FilePathSyntax.swift -primary-file /src/Sources/System/FilePath/FilePathWindows.swift -primary-file /src/Sources/System/FilePermissions.swift /src/Sources/System/Platform/DarwinPlatformConstants.swift /src/Sources/System/Platform/LinuxPlatformConstants.swift /src/Sources/System/Platform/Platform.swift /src/Sources/System/Platform/PlatformString.swift /src/Sources/System/Platform/WindowsPlatformConstants.swift /src/Sources/System/SystemString.swift /src/Sources/System/Util.swift /src/Sources/System/UtilConsumers.swift -emit-module-path /src/.build/x86_64-unknown-linux-gnu/debug/SystemPackage.build/FilePath/FilePathSyntax~partial.swiftmodule -emit-module-path /src/.build/x86_64-unknown-linux-gnu/debug/SystemPackage.build/FilePath/FilePathWindows~partial.swiftmodule -emit-module-path /src/.build/x86_64-unknown-linux-gnu/debug/SystemPackage.build/FilePermissions~partial.swiftmodule -emit-module-doc-path /src/.build/x86_64-unknown-linux-gnu/debug/SystemPackage.build/FilePath/FilePathSyntax~partial.swiftdoc -emit-module-doc-path /src/.build/x86_64-unknown-linux-gnu/debug/SystemPackage.build/FilePath/FilePathWindows~partial.swiftdoc -emit-module-doc-path /src/.build/x86_64-unknown-linux-gnu/debug/SystemPackage.build/FilePermissions~partial.swiftdoc -emit-module-source-info-path /src/.build/x86_64-unknown-linux-gnu/debug/SystemPackage.build/FilePath/FilePathSyntax~partial.swiftsourceinfo -emit-module-source-info-path /src/.build/x86_64-unknown-linux-gnu/debug/SystemPackage.build/FilePath/FilePathWindows~partial.swiftsourceinfo -emit-module-source-info-path /src/.build/x86_64-unknown-linux-gnu/debug/SystemPackage.build/FilePermissions~partial.swiftsourceinfo -emit-dependencies-path /src/.build/x86_64-unknown-linux-gnu/debug/SystemPackage.build/FilePath/FilePathSyntax.d -emit-dependencies-path /src/.build/x86_64-unknown-linux-gnu/debug/SystemPackage.build/FilePath/FilePathWindows.d -emit-dependencies-path /src/.build/x86_64-unknown-linux-gnu/debug/SystemPackage.build/FilePermissions.d -emit-reference-dependencies-path /src/.build/x86_64-unknown-linux-gnu/debug/SystemPackage.build/FilePath/FilePathSyntax.swiftdeps -emit-reference-dependencies-path /src/.build/x86_64-unknown-linux-gnu/debug/SystemPackage.build/FilePath/FilePathWindows.swiftdeps -emit-reference-dependencies-path /src/.build/x86_64-unknown-linux-gnu/debug/SystemPackage.build/FilePermissions.swiftdeps -target x86_64-unknown-linux-gnu -disable-objc-interop -I /src/.build/x86_64-unknown-linux-gnu/debug -I /src/Sources/CSystem/include -color-diagnostics -enable-testing -g -module-cache-path /src/.build/x86_64-unknown-linux-gnu/debug/ModuleCache -swift-version 5 -Onone -D SWIFT_PACKAGE -D DEBUG -enable-anonymous-context-mangled-names -Xcc -fmodule-map-file=/src/.build/x86_64-unknown-linux-gnu/debug/CSystem.build/module.modulemap -parse-as-library -module-name SystemPackage -o /src/.build/x86_64-unknown-linux-gnu/debug/SystemPackage.build/FilePath/FilePathSyntax.swift.o -o /src/.build/x86_64-unknown-linux-gnu/debug/SystemPackage.build/FilePath/FilePathWindows.swift.o -o /src/.build/x86_64-unknown-linux-gnu/debug/SystemPackage.build/FilePermissions.swift.o -index-store-path /src/.build/x86_64-unknown-linux-gnu/debug/index/store -index-system-modules
1.	Swift version 5.3.2 (swift-5.3.2-RELEASE)
2.	While evaluating request SILGenSourceFileRequest(SIL Generation for file "/src/Sources/System/FilePath/FilePathSyntax.swift")
3.	While emitting SIL for 'starts(with:)' (at /src/Sources/System/FilePath/FilePathSyntax.swift:70:10)
4.	While silgen emitFunction SIL function "@$s13SystemPackage8FilePathV6starts4withSbAC_tF".
 for 'starts(with:)' (at /src/Sources/System/FilePath/FilePathSyntax.swift:70:10)
5.	While silgen closureexpr SIL function "@$s13SystemPackage8FilePathV6starts4withSbAC_tFSbyKXEfu_".
 for expression at [/src/Sources/System/FilePath/FilePathSyntax.swift:72:39 - line:73:29] RangeText="components.starts(
	  with: other.components"
/usr/bin/swift[0x51fa1c4]
/usr/bin/swift[0x51f7dbe]
/usr/bin/swift[0x51fa49c]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x12980)[0x7f56e772b980]
/lib/x86_64-linux-gnu/libc.so.6(gsignal+0xc7)[0x7f56e5d96fb7]
/lib/x86_64-linux-gnu/libc.so.6(abort+0x141)[0x7f56e5d98921]
/lib/x86_64-linux-gnu/libc.so.6(+0x3048a)[0x7f56e5d8848a]
/lib/x86_64-linux-gnu/libc.so.6(+0x30502)[0x7f56e5d88502]
/usr/bin/swift[0xaa6617]
/usr/bin/swift[0xa45ff7]
/usr/bin/swift[0xa3e1ea]
/usr/bin/swift[0xa3a12c]
/usr/bin/swift[0xa39e42]
/usr/bin/swift[0xa11e97]
/usr/bin/swift[0xa02968]
/usr/bin/swift[0xabe8fb]
/usr/bin/swift[0xaa8982]
/usr/bin/swift[0xabd577]
/usr/bin/swift[0xac7d19]
/usr/bin/swift[0xaadc3a]
/usr/bin/swift[0xaaadf2]
/usr/bin/swift[0xa0e720]
/usr/bin/swift[0x9ff5d4]
/usr/bin/swift[0xa7f75e]
/usr/bin/swift[0xa2af9e]
/usr/bin/swift[0x9baaff]
/usr/bin/swift[0xa1db99]
/usr/bin/swift[0xa0e73a]
/usr/bin/swift[0xa02968]
/usr/bin/swift[0xabe502]
/usr/bin/swift[0xaa8982]
/usr/bin/swift[0xabd577]
/usr/bin/swift[0xac7d19]
/usr/bin/swift[0xaadc3a]
/usr/bin/swift[0xaaadf2]
/usr/bin/swift[0xa0e720]
/usr/bin/swift[0x9ff5d4]
/usr/bin/swift[0xa7f75e]
/usr/bin/swift[0xa7c632]
/usr/bin/swift[0xa7a708]
/usr/bin/swift[0xa7a5fd]
/usr/bin/swift[0xa2a6e3]
/usr/bin/swift[0x9c2b57]
/usr/bin/swift[0x9b80fc]
/usr/bin/swift[0xa8c039]
/usr/bin/swift[0xa8713b]
/usr/bin/swift[0xa870fd]
/usr/bin/swift[0x9bfb3c]
/usr/bin/swift[0x9bf682]
/usr/bin/swift[0xa79fec]
/usr/bin/swift[0x9c5a96]
/usr/bin/swift[0x9c126e]
/usr/bin/swift[0x9c1174]
/usr/bin/swift[0x561d74]
/usr/bin/swift[0x55f069]
/usr/bin/swift[0x4e8ce8]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xe7)[0x7f56e5d79bf7]
/usr/bin/swift[0x4e888a]

Windows Compilation Output

Windows compilation output:

PS C:\Users\user\test\swift-system> swift build -Xswiftc -sdk -Xswiftc $env:SDKROOT
[1/1] Compiling CSystem shims.c
[1/1] Compiling CSystem shims.c

* Build Completed!
<module-includes>:2:10: note: in file included from <module-includes>:2:

#include "C:\Users\user\test\swift-system\Sources\CSystem\include\CSystemWindows.h"

		 ^

C:\Users\user\test\swift-system\Sources\CSystem\include\CSystemWindows.h:15:10: note: in file included from C:\Users\user\test\swift-system\Sources\CSystem\include\CSystemWindows.h:15:

#include <Windows.h>

		 ^

C:\Program Files (x86)\Windows Kits\10\include\10.0.19041.0\um/windows.h:172:10: note: in file included from C:\Program Files (x86)\Windows Kits\10\include\10.0.19041.0\um/windows.h:172:

#include <winbase.h>

		 ^

C:\Program Files (x86)\Windows Kits\10\include\10.0.19041.0\um\winbase.h:5656:29: error: missing '#include <winbase.h>'; '_COPYFILE2_MESSAGE_TYPE' must be defined before it is used

	COPYFILE2_MESSAGE_TYPE  Type;

							^

C:\Program Files (x86)\Windows Kits\10\include\10.0.19041.0\um\winbase.h:5621:14: note: definition here is not reachable

typedef enum _COPYFILE2_MESSAGE_TYPE {

			 ^

Assertion failed: !D->isUnconditionallyVisible() && "expected a hidden declaration", file D:\a\1\s\llvm-project\clang\lib\Serialization\ASTWriter.cpp, line 6051

Please submit a bug report (https://swift.org/contributing/#reporting-bugs) and include the project and the crash backtrace.

Stack dump:

0.	C:\Program Files (x86)\Windows Kits\10\include\10.0.19041.0\um\winbase.h:5723:3: current parser token 'COPYFILE2_MESSAGE'

 #0 0x00007ff6db48b815 (C:\Library\Developer\Toolchains\unknown-Asserts-development.xctoolchain\usr\bin\swiftc.exe+0x454b815)

 #1 0x00007ff94fc71891 (C:\Windows\System32\ucrtbase.dll+0x71891)

 #2 0x00007ff94fc72861 (C:\Windows\System32\ucrtbase.dll+0x72861)

 #3 0x00007ff94fc741c5 (C:\Windows\System32\ucrtbase.dll+0x741c5)

 #4 0x00007ff94fc74501 (C:\Windows\System32\ucrtbase.dll+0x74501)

 #5 0x00007ff6da131a53 (C:\Library\Developer\Toolchains\unknown-Asserts-development.xctoolchain\usr\bin\swiftc.exe+0x31f1a53)

 #6 0x00007ff6d9e9ff12 (C:\Library\Developer\Toolchains\unknown-Asserts-development.xctoolchain\usr\bin\swiftc.exe+0x2f5ff12)

 #7 0x00007ff6da9fd00f (C:\Library\Developer\Toolchains\unknown-Asserts-development.xctoolchain\usr\bin\swiftc.exe+0x3abd00f)

 #8 0x00007ff6da1edcf7 (C:\Library\Developer\Toolchains\unknown-Asserts-development.xctoolchain\usr\bin\swiftc.exe+0x32adcf7)

 #9 0x00007ff6da278767 (C:\Library\Developer\Toolchains\unknown-Asserts-development.xctoolchain\usr\bin\swiftc.exe+0x3338767)

#10 0x00007ff6da089882 (C:\Library\Developer\Toolchains\unknown-Asserts-development.xctoolchain\usr\bin\swiftc.exe+0x3149882)

#11 0x00007ff6da021fc2 (C:\Library\Developer\Toolchains\unknown-Asserts-development.xctoolchain\usr\bin\swiftc.exe+0x30e1fc2)

#12 0x00007ff6da02ede9 (C:\Library\Developer\Toolchains\unknown-Asserts-development.xctoolchain\usr\bin\swiftc.exe+0x30eede9)

#13 0x00007ff6da01e439 (C:\Library\Developer\Toolchains\unknown-Asserts-development.xctoolchain\usr\bin\swiftc.exe+0x30de439)

#14 0x00007ff6d9ffca38 (C:\Library\Developer\Toolchains\unknown-Asserts-development.xctoolchain\usr\bin\swiftc.exe+0x30bca38)

#15 0x00007ff6da00130e (C:\Library\Developer\Toolchains\unknown-Asserts-development.xctoolchain\usr\bin\swiftc.exe+0x30c130e)

#16 0x00007ff6da005883 (C:\Library\Developer\Toolchains\unknown-Asserts-development.xctoolchain\usr\bin\swiftc.exe+0x30c5883)

#17 0x00007ff6d9e86e06 (C:\Library\Developer\Toolchains\unknown-Asserts-development.xctoolchain\usr\bin\swiftc.exe+0x2f46e06)

#18 0x00007ff6d9e86c28 (C:\Library\Developer\Toolchains\unknown-Asserts-development.xctoolchain\usr\bin\swiftc.exe+0x2f46c28)

#19 0x00007ff6d9e46103 (C:\Library\Developer\Toolchains\unknown-Asserts-development.xctoolchain\usr\bin\swiftc.exe+0x2f06103)

#20 0x00007ff6d9e41dca (C:\Library\Developer\Toolchains\unknown-Asserts-development.xctoolchain\usr\bin\swiftc.exe+0x2f01dca)

#21 0x00007ff6db4706b4 (C:\Library\Developer\Toolchains\unknown-Asserts-development.xctoolchain\usr\bin\swiftc.exe+0x45306b4)

#22 0x00007ff6db47083f (C:\Library\Developer\Toolchains\unknown-Asserts-development.xctoolchain\usr\bin\swiftc.exe+0x453083f)

#23 0x00007ff6db48c2fd (C:\Library\Developer\Toolchains\unknown-Asserts-development.xctoolchain\usr\bin\swiftc.exe+0x454c2fd)

#24 0x00007ff94fc214c2 (C:\Windows\System32\ucrtbase.dll+0x214c2)

#25 0x00007ff951306fd4 (C:\Windows\System32\KERNEL32.DLL+0x16fd4)

#26 0x00007ff951d9cec1 (C:\Windows\SYSTEM32\ntdll.dll+0x4cec1)

On GitHub Actions, with this workflow file, the Swift compiler ends the build with a very helpful message: error: fatalError. (You can view the full output here)

The Swift Package Index Linux build shows the same error.

O_DIRECTORY is not exposed

As the title suggests O_DIRECTORY is not exposed in any public API, which makes it annoying to open a FileDescriptor for a path only if it refers to a directory.

Fix `SystemString`'s `Collection` conformance

Swift 6.0 has an improved Array.replaceSubrange() implementation which is stricter than before with its Collection parameters.

SystemString’s withContiguousStorageIfAvailable() passes an UnsafeBufferPointer whose count is larger than itself, to accommodate a trailing null byte.

Array.replaceSubrange() does not like that at all, and swift-system’s tests crash with pre-release Swift 6 toolchains.
(See the re-implementation in apple/swift#66160.)

rdar://124102547

POSIX Spawn APIs

Is anyone actively working on this? I added just-enough-for-my-use-case API in my own project here: https://github.com/GeorgeLyon/Shwift/blob/main/Sources/Shell/Support/Posix%20Spawn.swift but I'd be happy to flesh it out and contribute.

Maybe this should wait on #20, but I can also probably just add the ProcessID type the same way it is proposed in that PR.

Release Testing for 1.4.0

• Swift 5.6.3 on macOS

• Swift 5.7.3 on macOS

• Swift 5.8.1 on macOS

• Swift 5.9 on macOS

• Swift 5.6.3 on Ubuntu 20.04 (x64)

• Swift 5.7.3 on Ubuntu 22.04 (x64)

• Swift 5.8.1 on Ubuntu 22.04 (x64)

• Swift 5.9 on Ubuntu 22.04

• Swift 5.8.1 on Windows