apple / swift-collections-benchmark

A benchmarking tool for Swift Collection algorithms

Swift Collections Benchmark

This package lets you collect and easily visualize performance data about data structure implementations and collection algorithms. It was created to help develop the Swift Collections package, but it's useful for so much more!

This project primarily concentrates on benchmarking Swift code, but it can also be used to run benchmarks (and, especially, to analyze benchmark results) in other languages, too.

Example

Here is a short benchmark, measuring the performance of Array.sorted() and Set.contains(_:):

import CollectionsBenchmark

var benchmark = Benchmark(title: "Demo Benchmark")

benchmark.addSimple(
  title: "Array<Int> sorted",
  input: [Int].self
) { input in
  blackHole(input.sorted())
}

benchmark.add(
  title: "Set<Int> contains",
  input: ([Int], [Int]).self
) { input, lookups in
  let set = Set(input)
  return { timer in
    for value in lookups {
      precondition(set.contains(value))
    }
  }
}

benchmark.main()

Here is how you run it:

$ swift run -c release benchmark run results --cycles 5
Running 2 tasks on 76 sizes from 1 to 1M:
  Array<Int> sorted
  Set<Int> contains
Output file: /Users/klorentey/Projects/swift-collections-benchmark-demo/Demo/results
Appending to existing data (if any) for these tasks/sizes.

Collecting data:
  1.2.4...8...16...32...64...128...256...512...1k...2k...4k...8k...16k...32k...64k...128k...256k...512k...1M -- 5.31s
  1.2.4...8...16...32...64...128...256...512...1k...2k...4k...8k...16k...32k...64k...128k...256k...512k...1M -- 5.35s
  1.2.4...8...16...32...64...128...256...512...1k...2k...4k...8k...16k...32k...64k...128k...256k...512k...1M -- 5.29s
  1.2.4...8...16...32...64...128...256...512...1k...2k...4k...8k...16k...32k...64k...128k...256k...512k...1M -- 5.3s
  1.2.4...8...16...32...64...128...256...512...1k...2k...4k...8k...16k...32k...64k...128k...256k...512k...1M -- 5.34s
Finished in 26.6s
$ swift run -c release benchmark render results chart.png
$ open chart.png

And this is what you get:

Today I learned that sorting 20 integers in an Array takes about as much time as looking at all items in a 20-member Set. Fascinating! 🤓

Documentation

For a tour of the features provided by this library, please be sure check out our Getting Started Guide!

Project Status

Swift Collections Benchmark is intended primarily as a developer tool, rather than something that would be used in production apps.

It exposes a source-level API and a command line interface, neither of which are technically stable yet. After an initial period of experimentation, we expect to stabilize both interfaces. Until then, new releases may sometimes come with changes that might break existing benchmark definitions, or that may change the command line interface in ways that could break existing scripts. We'll try our best to keep this to a minimum, though (or mitigate with a multi-release deprecation period), even during this chaotic initial period.

Adding Swift Collections Benchmark as a Dependency

To use this package in a SwiftPM project, add the following line to the dependencies in your Package.swift file:

.package(url: "https://github.com/apple/swift-collections-benchmark", from: "0.0.1"),

In the typical case, you'll want to set up a standalone executable target that is dedicated to benchmarking:

// swift-tools-version:5.3
import PackageDescription

let package = Package(
  name: "MyPackage",
  products: [
    .executable(name: "my-benchmark", targets: ["MyBenchmark"]),
  ],
  dependencies: [
    .package(url: "https://github.com/apple/swift-collections-benchmark", from: "0.0.1"),
    // ... other dependencies ...
  ],
  targets: [
    // ... other targets ...
    .target(
      name: "MyBenchmark",
      dependencies: [
        .product(name: "CollectionsBenchmark", package: "swift-collections-benchmark"),
      ]),
  ]
)

Contributing to Swift Collections Benchmark

Asking questions

We can use the Swift Collections Forum to ask and answer questions on how to use or work on this package. It's also a great place to discuss its evolution.

Reporting a bug

If you find something that looks like a bug, please open a Bug Report! Fill out as many details as you can.

Fixing a bug or making a small improvement

1. Submit a PR with your change. If there is an existing issue for the bug you're fixing, please include a reference to it.
2. Make sure to add test coverage for whatever changes you are making (if possible).

(Note: The package doesn't currently come with many tests, reflecting its origins as a supporting project -- we strive to improve that!)

Proposing a small enhancement

1. Raise a Feature Request. Discuss why it would be important to implement it.
2. Submit a PR with your implementation, participate in the review discussion.
3. When there is a consensus that the feature is desirable, and the implementation works well, it will be merged.
4. Rejoice!

Proposing a larger feature

1. Raise a Feature Request, or start a topic on the forum. Discuss why it would be important to implement it, and potential implementation strategies.
2. Submit a PR with your implementation, and participate in the review discussion. Sometimes we may need to go through several revisions! This is fine -- it makes the end result that much better.
3. When there is a consensus that the feature is desirable, and the implementation works well, it will be merged.
4. Celebrate!

Licensing

By submitting a pull request, you represent that you have the right to license your contribution to Apple and the community, and agree by submitting the patch that your contributions are licensed under the Swift License, a copy of which is provided in this repository.

Code of Conduct

Like all Swift.org projects, we would like the Swift Collections Benchmark project to foster a diverse and friendly community. We expect contributors to adhere to the Swift.org Code of Conduct. A copy of this document is available in this repository.

Contacting the maintainers

The current code owner of this package is Karoy Lorentey (@lorentey). You can contact him on the Swift forums, or by writing an email to klorentey at apple dot com. (Please keep it related to this project.)

In case of moderation issues, you can also directly contact a member of the Swift Core Team.

swift-collections-benchmark's Issues

Support Swift concurrency

A common benchmarking need is to benchmark asynchronous work. With the advent of Swift concurrency, I might suggest adding support for async closures.

With the current implementation, there are challenges:

1. One has to use dispatch group/semaphore wait to make asynchronous work behave in the synchronous environment of this package. (See my note below for example of my clumsy workaround.)
2. This package’s existing Task type is exceeding confusing for users who might attempt to use the now-pervasive Swift concurrency Task, getting non-standard autocompletion. One has to resort to import Concurrency and _Concurrency.Task.detached {…} (or whatever) when dealing with Swift concurrency Task.

I am happy to take a pass at a PR for this, but a few questions:

1. How to handle this package’s existing Task type:

   • I might be inclined to rename this as BenchmarkTask or Benchmark.Test to avoid confusion. But that would be a breaking change. Is there a reason this is public and/or named Task? Thoughts regarding the renaming/namespacing?
   • Is there a reason that this Task was exposed rather than being internal/private?

2. Preferences regarding the interface for renditions that take async closures?

   • Would you like an entirely separate async API? E.g., addSimpleAsync (or withSimple, or whatever) for async rendition of addSimple? Ditto with blackHole, etc. What naming convention would you like to adopt? How would you like to tackle this?

Conceptually, this does not seem outlandishly complicated to make this support concurrency, and the biggest question in my mind is how to do so elegantly in a non-breaking manner. If you would like me to attempt a PR, a little guidance would be appreciated. Of course, if you would rather tackle this yourself, that would be great. I’m happy either way, but was hesitant to spend a lot of time on this without a little direction and/or guidelines.

FWIW, here is an example of my benchmarking of asynchronous work. I had to wrap my asynchronous test in something that used a dispatch group (or I could have used a semaphore, too) to wait for the asynchronous work to finish. Using group/semaphore wait is clumsy and an anti-pattern in Swift concurrency. And the use of _Concurrency.Task.detached feels awkward.

Guide on writing benchmarking tasks

We need a chapter 2 of the documentation that's all about writing benchmarks.

The addSimple method demonstrated in chapter 1 has no support for omitting parts of the code from the benchmarking scope, so only a fraction of benchmarks can use it.

The real add method has support for adding setup code that isn't measured (both code that needs to be only run once per each input, and code that needs to be repeated before each measurement), but it is way too subtle to use without adequate documentation.

Remove “collections” from the repo name

This package is the “Swift Collections Benchmark”. But why tie it to “collections” at all? I understand that this was the impetus for this package, but it has uses well beyond collections. I would suggest renaming this the “Swift Benchmarks” (or something like that) and lose any implied suggestion that this benchmarking is for collections only.

For example, we might use this for testing any computationally intensive process, e.g., the benchmarking of pixel buffer processing, or benchmarking to determining the appropriate striding factor for a particular parallel calculation. The list is endless.

This package should start using the stdlib's Clock and Duration types

Subject says it all -- this package should stop using platform-specific clock APIs (or delegating to Foundation.Date).

It's been a while since we added ContiguousClock/SuspendingClock/Duration to the stdlib, and the restricted availability of these types is less of an issue with every passing day, even on Apple platforms.

The primary difficulty is in making sure we preserve custom formatting of duration values, and that we don't break statistical calculations. Duration is a fixed-width integer count of attoseconds, which I expect to be just as good as or better than the current plain Double for doing the simple statistical calculations in this package -- but this needs to be verified in practice. (Some things could be tricky to express without relying on Int128...)

The format of saved benchmark results will likely need to change to reflect the change in duration representations. That is fine -- this package makes no promises on compatibility of previously saved results.

Update swift-system dependency to 0.0.2

Swift System 0.0.1 is really old and doesn't build on Android.

Information

• Package version: main
• Platform version: Android 11
• Swift version: 5.4

Checklist

• If possible, I've reproduced the issue using the main branch of this package.
• I've searched for existing GitHub issues.

swift-collections-benchmark often fails on M1

In attempting to replicate the graphs from the announcement blog, I've found swift-collections-benchmark to be unreliable on an M1 Mac mini. I've only been able to run a single cycle so far, as any attempt at more than 1 always fails.

Information

• Package version: main
• Platform version: macOS 11.2.3
• Swift version: Xcode 12.5b3, Apple Swift version 5.4 (swiftlang-1205.0.26.4 clang-1205.0.19.54)

Checklist

• If possible, I've reproduced the issue using the main branch of this package.
• I've searched for existing GitHub issues.

Steps to Reproduce

./generate-results.sh
+ ../../Utils/run-benchmarks.sh library run results.json --library Library.json --max-size 16K --cycles 1 --mode replace-all
Running 19 tasks on 52 sizes from 1 to 16k:
  Deque<Int> subscript get, random offsets (discontiguous)
  Array<Int> subscript get, random offsets
  std::deque<intptr_t> at, random offsets
  Deque<Int> prepend, reserving capacity
  Array<Int> prepend, reserving capacity
  std::deque<intptr_t> push_front
  OrderedSet<Int> successful contains
  Set<Int> successful contains
  std::unordered_set<intptr_t> successful find
  Array<Int> successful contains
  OrderedSet<Int> append, reserving capacity
  Set<Int> insert, reserving capacity
  std::unordered_set<intptr_t> insert, reserving capacity
  OrderedDictionary<Int, Int> subscript, successful lookups
  Dictionary<Int, Int> subscript, successful lookups
  std::unordered_map<intptr_t, intptr_t> successful find
  OrderedDictionary<Int, Int> subscript, append, reserving capacity
  Dictionary<Int, Int> subscript, insert, reserving capacity
  std::unordered_map<intptr_t, intptr_t> insert, reserving capacity
Output file: /Users/jshier/Desktop/Code/swift-collections/Documentation/Announcement-benchmarks/results.json
Discarding all existing data.

Collecting data:
  1.2.../../Utils/run-benchmarks.sh: line 11: 40819 Trace/BPT trap: 5       swift run --package-path "$srcroot" $flags swift-collections-benchmark "$@"

Expected behavior

Successful benchmark run, as I did on my Intel iMac.

Actual behavior

1.2.../../Utils/run-benchmarks.sh: line 11: 40819 Trace/BPT trap: 5 swift run --package-path "$srcroot" $flags swift-collections-benchmark "$@"

Build fails off macOS due to missing AffineTransform in Foundation module

When running the build for all platforms I found it fails due to AffineTransform only being available on macOS. See the docs below.

• AffineTransform

The error shows in CGTypes.swift in block which includes iOS, tvOS and watchOS which do not have AffineTransform in Foundation. The build target is set to build for all platform and this package is a dependency of Swift Collections which will also need to build across Apple's platforms and ideally Linux and Windows.

Information

• version 0.0.1
• latest stable platform version
• Swift 5.4

Checklist

• If possible, I've reproduced the issue using the main branch of this package.
• I've searched for existing GitHub issues.

Steps to Reproduce

• Build for iOS
• See error in CGTypes.swift

Expected behavior

There should be an equivalent type which can be used off macOS on the other platforms. I tried CoreGraphics.CGAffineTransform but it does not work quite the same.

Actual behavior

Due to the missing type the build fails for iOS. It appears it would only build for macOS.

Example code crashes on linux

Example code from README.md crashes on linux. The message I'm getting after installing swift-backtrace is
"Swift runtime failure: Double value cannot be converted to Int because it is either infinite or NaN"

Project files without the ".build" directory ThisIsBenchmark.zip (if I included .build it would be too big for github)

Information

• Package version: main branch
• Platform version: Ubuntu 20.04.2 LTS running in a VM
• Swift version: Swift version 5.5-dev (LLVM 8c37fc45eae308f, Swift 63b62320eb022d8) Target: x86_64-unknown-linux-gnu

Checklist

• If possible, I've reproduced the issue using the main branch of this package.
• I've searched for existing GitHub issues.

Steps to Reproduce

Replace this paragraph with an explanation of how to reproduce the incorrect behavior.
Include a simple code example, if possible.

• create new executable package
• add swift-collections-benchmark dependency
• copy-paste example benchmarks from README.md to main.swift
• optionally add swift-backtrace to show more info on crash
• run swift run -c release ThisIsBenchmark run out in terminal

Expected behavior

package is compiled, and example benchmark is executed

Actual behavior

package is compiled with a "remark: Incremental compilation has been disabled: it is not compatible with whole module optimization", but crashes at runtime. Without the backtrace package the error doesn't say anything, but with the backtrace package it looks like this:

Running 2 tasks on 76 sizes from 1 to 1M:
  Array<Int> sorted
  Set<Int> contains
Output file: /home/cukier/Desktop/ThisIsBenchmark/out
Appending to existing data (if any) for these tasks/sizes.

Collecting data:
  10x7f3f4a3653bf
0x55b70ba8f75d, Swift runtime failure: Double value cannot be converted to Int because it is either infinite or NaN at /home/cukier/Desktop/ThisIsBenchmark/.build/checkouts/swift-collections-benchmark/Sources/CollectionsBenchmark/Benchmark/Task.swift:0
0x55b70ba8f75d, CollectionsBenchmark.Task.measure(size: CollectionsBenchmark.Size, input: A, options: CollectionsBenchmark.Benchmark.Options) -> Swift.Optional<CollectionsBenchmark.Time> at /home/cukier/Desktop/ThisIsBenchmark/.build/checkouts/swift-collections-benchmark/Sources/CollectionsBenchmark/Benchmark/Task.swift:121
0x55b70ba60956, CollectionsBenchmark._ConcreteTask.measure(size: CollectionsBenchmark.Size, input: Any, options: CollectionsBenchmark.Benchmark.Options) -> Swift.Optional<CollectionsBenchmark.Time> at /home/cukier/Desktop/ThisIsBenchmark/.build/checkouts/swift-collections-benchmark/Sources/CollectionsBenchmark/Benchmark/AnyTask.swift:119
0x55b70ba78101, CollectionsBenchmark.AnyTask.measure(size: CollectionsBenchmark.Size, input: Any, options: CollectionsBenchmark.Benchmark.Options) -> Swift.Optional<CollectionsBenchmark.Time> at .build/checkouts/swift-collections-benchmark/Sources/CollectionsBenchmark/Benchmark/AnyTask.swift:49
0x55b70ba78101, function signature specialization <Arg[3] = [Closure Propagated : closure #2 (CollectionsBenchmark.Benchmark.Event) throws -> () in CollectionsBenchmark._Document.run(benchmark: CollectionsBenchmark.Benchmark, options: CollectionsBenchmark.Benchmark.Options) throws -> (), Argument Types : [Swift.BoolCollectionsBenchmark._Document]> of CollectionsBenchmark.Benchmark.measureOneCycle(tasks: Swift.Array<CollectionsBenchmark.AnyTask>, sizes: Swift.Array<CollectionsBenchmark.Size>, options: CollectionsBenchmark.Benchmark.Options, delegate: (CollectionsBenchmark.Benchmark.Event) throws -> ()) throws -> () at /home/cukier/Desktop/ThisIsBenchmark/.build/checkouts/swift-collections-benchmark/Sources/CollectionsBenchmark/Benchmark/Benchmark+RunOptions.swift:40
0x55b70ba7963d, function signature specialization <Arg[1] = [Closure Propagated : closure #2 (CollectionsBenchmark.Benchmark.Event) throws -> () in CollectionsBenchmark._Document.run(benchmark: CollectionsBenchmark.Benchmark, options: CollectionsBenchmark.Benchmark.Options) throws -> (), Argument Types : [Swift.BoolCollectionsBenchmark._Document]> of CollectionsBenchmark.Benchmark.run(options: CollectionsBenchmark.Benchmark.Options, delegate: (CollectionsBenchmark.Benchmark.Event) throws -> ()) throws -> () at /home/cukier/Desktop/ThisIsBenchmark/.build/checkouts/swift-collections-benchmark/Sources/CollectionsBenchmark/Benchmark/Benchmark+RunOptions.swift:73
0x55b70bae0a41, CollectionsBenchmark._Document.run(benchmark: CollectionsBenchmark.Benchmark, options: CollectionsBenchmark.Benchmark.Options) throws -> () at /home/cukier/Desktop/ThisIsBenchmark/.build/checkouts/swift-collections-benchmark/Sources/CollectionsBenchmark/BenchmarkCLI/_Document.swift:149
0x55b70badb00b, CollectionsBenchmark._BenchmarkCLI.Run.run(benchmark: CollectionsBenchmark.Benchmark) throws -> () at /home/cukier/Desktop/ThisIsBenchmark/.build/checkouts/swift-collections-benchmark/Sources/CollectionsBenchmark/BenchmarkCLI/BenchmarkCLI+Run.swift:76
0x55b70badb23f, protocol witness for CollectionsBenchmark._BenchmarkCommand.run(benchmark: CollectionsBenchmark.Benchmark) throws -> () in conformance CollectionsBenchmark._BenchmarkCLI.Run : CollectionsBenchmark._BenchmarkCommand in CollectionsBenchmark at /home/cukier/Desktop/ThisIsBenchmark/<compiler-generated>:0
0x55b70ba6a9fc, CollectionsBenchmark.Benchmark.main() -> () at /home/cukier/Desktop/ThisIsBenchmark/.build/checkouts/swift-collections-benchmark/Sources/CollectionsBenchmark/Benchmark/Benchmark+Main.swift:32
0x55b70bb316f2, ThisIsBenchmark_main at /home/cukier/Desktop/ThisIsBenchmark/Sources/ThisIsBenchmark/main.swift:27
0x7f3f498d60b2
0x55b70b9da6dd
0xffffffffffffffff
Illegal instruction (core dumped)