GithubHelp home page GithubHelp logo

act's Introduction

Aerospike Certification Tool (ACT)

This project is maintained by Aerospike

Overview


ACT provides a pair of programs for testing and certifying flash/SSD devices' performance for Aerospike Database data and index storage. ACT measures latency during a mixed load of read and write operations while modeling the Aerospike Database server's I/O pattern as closely as practical.

ACT allows you to test a single device or multiple devices, using your actual connector/controller hardware.

There are two programs: act_storage models Aeropike Database data storage I/O patterns, and act_index models Aeropike Database index storage I/O patterns for Aerospike Database's "All Flash" mode.

The purpose of this certification is:

  1. Determine if an SSD device(s) will stand up to the demands of a high-speed real-time database.
  2. Evaluate the upper limits of the throughput you can expect from a device(s).

Not all SSDs can handle the high volume of transactions required by high performance real-time databases like Aerospike Database. Many SSDs are rated for 100K+ reads/writes per second, but in production the actual load they can withstand for sustained periods of time is generally much lower. In the process of testing many common SSDs in high-throughput tests, Aerospike developed this certification tool, ACT, that you can use to test/certify an SSD for yourself.

We have found performance – especially latency – of SSDs to be highly dependent on the write load the SSD is subjected to. Over the first few hours of a test, performance can be excellent, but past the 4- to 12-hour mark (depending on the device), performance can suffer.

The ACT tool allows you to test an SSD device(s) for yourself. In addition, Aerospike has tested a variety of SSDs and has specific recommendations. For more information, visit the Aerospike Database documentation at: http://www.aerospike.com/docs/operations/plan/ssd/ssd_certification.html.

What ACT Does


By default, act_storage performs a combination of large (128K) block reads and writes and small (1.5K) block reads, simulating standard real-time Aerospike Database data read/write and defragmentation loads.

By default, act_index performs a mixture of 4K reads and writes, simulating standard real-time Aerospike Database "All Flash" index device loads.

Latencies are measured for a long enough period of time (typically 24 hours) to evaluate device stability and overall performance.

Traffic/Loading

You can simulate:

  • "Nx" load - 1x load (2000 reads/sec and 1000 writes/sec per device) times N
  • any other stress load or high-performance load (custom configurable)

Latency Rate Analysis

ACT's output shows latency broken into intervals of 2^n ms: 1, 2, 4, 8 ... ms (analysis program's display intervals are configurable).

For example, a test might indicate that 0.25% of requests failed to complete in 1 ms or less and 0.01% of requests failed to complete in 8 ms or less.

Methodology for act_storage

The small read operations model client read requests. Requests are done at the specified rate by a number of service threads.

The large-block read and write operations model the Aerospike server's write requests and defragmentation process. The operations occur at a rate determined by the specified write request rate, and are executed from one dedicated large-block read thread and one dedicated large-block write thread per device.

Methodology for act_index

The 4K device reads model index element access that occurs during client read and write requests, and defragmentation. One device read is executed on service threads for each client read, and for each client write. In addition, more reads are executed in "cache threads" to model index element access during defragmentation.

The "cache threads" also execute all the 4k device writes, which model index element changes due to client write requests and defragmentation.

Unlike the Aerospike Database "All Flash" mode, act_index does not mmap files in mounted directories on the devices - it models the raw device I/O pattern, assuming no caching benefit from mmap. Therefore to configure act_index we simply specify the devices.

Process for Certifying Device(s) for 30x Performance


In general, we recommend that you certify a device for 30x performance. Many devices do not pass the 30x certification. If you do not have a high-volume application, you may find that a 10x or 20x certification will be sufficient. The instructions below describe the 30x certification process, but you may need to adjust the test based on your requirements.

To certify a device(s) for 30x performance with Aerospike Database requires two stages:

  1. Test a single device to determine performance using the hardware configuration and connectors. The single-device certification will help you determine individual device performance.
  2. If you will be using multiple devices, you can then run ACT to test multiple devices to see how the results will be affected by the capacity of the bus or the throughput of the RAID controller that is managing your devices.

The test process with ACT is the same for both stages, but in the first stage you are testing a device and in the second stage, you are testing the linearity/scalability of your connector with multiple devices installed.

The single-device stage takes 48 hours. The multi-device stage takes an additional 48 hours.

The first stage is to certify a single device, to test the device itself and the connection.

Begin by installing your SSD device. Our website has more details about installing SSDs in different environments and configurations at http://www.aerospike.com/docs/operations/plan/ssd/ssd_setup.html.

Test 1: Test under high loads

Run ACT for 24 hrs using the 30x test (60000 reads/sec and 30000 writes/sec). The device passes this test if less than 5% of operations fail to complete in 1 ms or less.

Many devices fail this test and are unsuitable for use with Aerospike Database.

Test 2: Stress test to ensure the device does not fail under excessive loads

Run a 60x test for 24 hrs (120000 reads/sec and 60000 writes/sec). The device passes this test if ACT runs to completion, regardless of the error rate.

If you are testing a single device, then the device is certified when it passes Test 1 and Test 2.

The second stage is to certify multiple devices, to make sure that performance scales linearly when you add devices.

Install the additional SSDs to be tested. Our website has more details about installing SSDs in different environments and configurations at http://www.aerospike.com/docs/operations/plan/ssd/ssd_setup.html.

Test 3: Repeat Test 1, with all devices installed: Test under high loads

Run ACT for 24 hrs using the 30x test (60000 reads/sec and 30000 writes/sec per device). The devices pass this test if less than 5% of operations fail to complete in 1 ms or less.

Test 4: Repeat Test 2, with all devices installed: Stress test to ensure the devices do not fail under excessive loads

Run a 60x test for 24 hrs (120000 reads/sec and 60000 writes/sec per device). The devices pass this test if ACT runs to completion, regardless of the error rate.

The devices are certified if they pass Test 3 and Test 4.

Once the device(s) has been certified, the device can be used with Aerospike Database.

 

Determining Expected Performance at Higher Throughput


If your application is going to have high volumes of transactions and your device(s) passes the 30x certification, we recommend that you test your device to determine its upper limit on transaction processing latency. This will help you determine how many SSDs you will need to run your application when you are fully scaled up.

To certify a device(s) at higher levels of performance, do the certification process as described above, but use higher loads (80x, 100x, etc.). Test the device(s) at progressively higher rates until more than 5% of operations fail in 1 ms.

For example, if you test at 60x and less than 5% of operations fail to complete in 1 ms, re-run the test at 80x, etc. When the device completes the test at a particular speed with more than 5% of operations failing to complete in 1 ms (i.e., fails the test), then the device is certified at the next lower level where the device DOES have fewer than 5% of errors in under 1 ms.

If your device is testing well at higher loads, you may want to shorten the test time. Running ACT for six hours will give you a good idea whether your device can pass ACT testing at a given traffic volume. Before certifying your device at a given traffic level, we recommend a full 24-hour test.

As before, test a single device first, and then test with multiple devices to make sure that the performance scales linearly with your connector/controller.

Getting Started


Download the ACT package through git:

$ git clone https://github.com/aerospike/act.git

This creates an /act directory.

Alternately you can download the ZIP or TAR file from the links at the left. When you unpack/untar the file, it creates an /aerospike-act- directory.

Install the Required Libraries

Before you can build ACT, you need to install some libraries.

For CentOS:

$ sudo yum install make gcc

For Debian or Ubuntu:

$ sudo apt-get install make gcc libc6-dev

Build the package.

$ cd act    OR    cd /aerospike-act-<version>
$ make

This will create 3 binaries in a target/bin directory:

  • act_prep: This executable prepares a device for ACT by writing zeroes on every sector of the disk and then filling it up with random data (salting). This simulates a normal production state.

  • act_storage: The executable for modeling Aerospike Database data storage device I/O patterns.

  • act_index: The executable for modeling Aerospike Database "All Flash" mode index device I/O patterns.

Running the ACT Certification Process


To certify your device(s), first determine what certification test you will run, as described above in Process for Certifying a Drive(s) for 3x Performance or Determining Expected Performance at Higher Throughput.

For each certification test with ACT, you must perform the following steps:

  1. Prepare the device(s) with act_prep -- only the first time you test.
  2. Create the config file for your test.
  3. Run the test, sending the results to a log file.
  4. Analyze log file output using the /analysis/act_latency.py script.
  5. Determine pass/fail for the test.

The details of these steps are described in detail below.

The tests destroy all data on the devices being tested!

When preparing devices and running tests, make sure the devices are specified by name correctly.

Make sure the test device is not mounted.

1. Prepare the Drives with act_prep - First Time Only


The first time you test a device(s), you must prepare the device(s) by first cleaning them (writing zeros everywhere) and then "salting" them (writing random data everywhere) with act_prep.

act_prep takes a device name as its only command-line parameter. For a typical 240GB SSD, act_prep takes 30-60+ minutes to run. The time varies depending on the device and the capacity.

If you are testing multiple devices, you can run act_prep on all of the devices in parallel. Preparing multiple devices in parallel does not take a lot more time than preparing a single device, so this step should only take an hour or two.

For example, to clean and salt the device /dev/sdc: (over-provisioned using hdparm)

$ sudo ./act_prep /dev/sdc &

If you are using a RAID controller / over-provisioned using fdisk, make sure you specify the partition and not the raw device. If the raw device is used then ACT will wipe out the partition table and this will invalidate the test.

$ sudo ./act_prep /dev/sdc1 &

2. Create a Configuration File


Create your config file by copying the appropriate example config file in the /config directory and modifying it, as described in the ACT Configuration Reference below. The example files are for the standard 1x load (2000 reads/sec and 1000 writes/sec per device).

Copy act_storage.conf to run the normal data storage modeling tests, or copy act_index.conf to run "All Flash" mode tests for index devices.

3. Run the test


From the ACT installation directory, run:

$ sudo ./target/bin/act_storage actconfig.txt > output.txt &

where:

* actconfig.txt - path/name of your config file
* output.txt    - path/name of your log file

If running ACT from a remote terminal, it is best to run it as a background process, or within a "screen". To verify that ACT is running, tail the output text file with the -f option.

Note that if the device(s) being tested performs so badly that ACT cannot keep up with the specified load, ACT will halt before the configured test duration has elapsed. ACT may also halt prematurely if it encounters unexpected device I/O or system errors.

4. Analyze ACT Output


Run /analysis/act_latency.py to process the ACT log file and tabulate data. Note that you can run the script when the test is not yet complete, and you will see the partial results.

For example:

$ ./analysis/act_latency.py -l output.txt

where:

 -l <act output file name> - required parameter that specifies the path/name of the log file generated by ACT

and optionally:

 -h <histogram name(s)> - optional parameter specifying histogram name(s): defaults are small read latency histograms
 -t <slice duration in seconds> - optional parameter specifying slice length; default is 3600 sec (1 hour)
 -s <start threshold> - optional parameter specifying start threshold for display; default is 0 (1 ms/us)
 -n <number of thresholds> - optional parameter specifying number of thresholds to display; default is 7
 -e <display every 'e'th threshold> - optional parameter specifying display threshold frequency; default is (every) 1
 -x <display throughputs> - optional parameter indicating that throughputs should also be displayed: default is no

The Python script analyzes the ACT output in time slices as specified, and displays latency data at various verification intervals for each slice.

The example output below is for an act_storage 12-hour test (each slice is an hour), run with options -n 3 (display 3 thresholds) and -e 3 (display every 3rd threshold). The reads table shows read latencies accumulated over all devices. So for example, in the 5th hour, 1.68% of reads failed to complete in under 1 ms.

         reads
         %>(ms)
 slice        1      8     64
 -----   ------ ------ ------
     1     1.67   0.00   0.00
     2     1.38   0.00   0.00
     3     1.80   0.14   0.00
     4     1.43   0.00   0.00
     5     1.68   0.00   0.00
     6     1.37   0.00   0.00
     7     1.44   0.00   0.00
     8     1.41   0.00   0.00
     9     2.70   0.73   0.00
    10     1.54   0.00   0.00
    11     1.53   0.00   0.00
    12     1.47   0.00   0.00
 -----   ------ ------ ------
   avg     1.62   0.07   0.00
   max     2.70   0.73   0.00

The script will also echo the configuration used to generate the log file, along with other basic information, above the latency tables. (We do not show his output in the example above.)

5. Evaluate Device(s) by the Standard Pass/Fail Criteria


Passing a Performance Test

In any one-hour period of an ACT performance test, we expect that:

  • fewer than 5% of transactions fail to complete in 1 ms
  • fewer than 1% of transactions fail to complete in 8 ms
  • fewer than 0.1% of transactions fail to complete in 64 ms

The max line of the output shows the highest values observed in any single slice (hour) of time, and the values on the max line should not exceed the allowable error values specified above.

In the example output above, we show only 12 hours of results, and the device passes because the worst performance in any slice was 2.7% of transactions failing to complete within 1 ms, 0.73% of transactions failing to complete in less than 8 ms and no transactions failing to complete within 64 ms.

A device(s) which does not exceed these error thresholds in 24 hours passes the load test.

Passing a Stress Test

When doing stress testing at a level ABOVE where the device is certified, a device passes the test if ACT runs to completion, regardless of the number of errors.

ACT Configuration Reference


Modifying the Config File


This package includes two example config files, one for act_storage (/config/act_storage.conf) and one for act_index (/config/act_index.conf).

Chose the one appropriate for the test you wish to run. (This is usually act_storage. Run act_index only if you are testing devices for storing indexes when running Aerospike Database in "All Flash" mode.)

First, you must be sure to set the correct device name(s). Then you should adjust the transaction request rates.

Each example config file has the transaction request rates for a 1x load with a single device. To generate a config file for an Nx load, simply multiply those rates by N, and by the number of devices you are testing with, if using multiple devices.

For example, to generate a config file for a single-device 60x load, change read-reqs-per-sec to 120000, and write-reqs-per-sec to 60000.

Or, to generate a config file for a four-device 60x load, change read-reqs-per-sec to 480000, and write-reqs-per-sec to 240000.

You may of course run customized loads, including read-only loads (set write-reqs-per-sec to 0) or write-only loads (set read-reqs-per-sec to 0).

The other fields in the configuration files should generally not be changed, but you may do so to run highly customized tests.

Format of Lines in the Config File


All fields use a

name-token: value

format, and must be on a single line. Field order in the file is unimportant. To add comments, add a line(s) that begin with '#'.

Fields that you Must Change:

device-names Comma-separated list of device names (full path) to test. For example:

device-names: /dev/sdb,/dev/sdc

Make sure the devices named are entered correctly.

Fields that you will Almost Always Change:

read-reqs-per-sec Read transactions/second to simulate. Note that this is not per device. For 30 times (30x) the normal load for four devices, this value would be 30 x 4 x 2000 = 240000. Formula: n x number of devices x 2000.

write-reqs-per-sec Write transactions/second to simulate. For act_storage, this value along with record-bytes, large-block-op-kbytes, defrag-lwm-pct, and others, determines the rate of large-block operations. Note that this is not per device. For 30 times (30x) the normal load for four devices, this value would be 30 x 4 x 1000 = 120000. Formula: n x number of devices x 1000.

Fields that you may Sometimes Change:

test-duration-sec Duration of the entire test, in seconds. Note that it has to be a single number, e.g. use 86400, not 60 x 60 x 24. The default is one day (24 hours).

Fields that you will Rarely or Never Change:

service-threads Total number of service threads on which requests are generated and done. If a test stops with a message like "... ACT can't do requested load ...", it doesn't mean the devices failed, it just means the transaction rates specified are too high to achieve with the configured number of service threads. Try testing again with more service threads. The default service-threads is 5x the number of CPUs, detected by ACT at runtime.

cache-threads (act_index ONLY) Number of threads from which to execute all 4K writes, and 4K reads due to index access during defragmentation. These threads model the system threads that would do these device I/O operations behind mmap. The default cache-threads is 8.

report-interval-sec Interval between generating observations, in seconds. This is the smallest granularity that you can analyze. Default is 1 sec. The /analysis/act_latency.py script aggregates these observations into slices, typically hour-long groups.

microsecond-histograms Flag that specifies what time units the histogram buckets will use -- yes means use microseconds, no means use milliseconds. If this field is left out, the default is no.

record-bytes (act_storage ONLY) Size of a record in bytes. This determines the size of a read operation -- just record-bytes rounded up to a multiple of 512 bytes (or whatever the device's minimum direct op size). Along with write-reqs-per-sec, large-block-op-kbytes, and others, this item determines the rate of large-block operations. record-bytes is rounded up to a multiple of 16 bytes to model Aerospike Database storage granularity. For example, if record-bytes is 1500, write-reqs-per-sec is 1000, and large-block-op-kbytes is 128, we write (1504 x 1000) bytes per second, or (1504 x 1000) / (128 x 1024) = 11.4746 large blocks per second. With defrag-lwm-pct 50, we double this to simulate defragmentation where blocks depleted to 50%-used are re-packed, yielding a large-block write (and read) rate of 22.949 blocks per second. May not be larger than 8Mbytes.

record-bytes-range-max (act_storage ONLY) If set, simulate a range of record sizes from record-bytes up to record-bytes-range-max. Therefore if set, it must be larger than record-bytes. The simulation models a linear distribution of sizes within the range. The default record-bytes-range-max is 0, meaning no range -- model all records with size record-bytes. May not be larger than 8Mbytes.

large-block-op-kbytes (act_storage ONLY) Size written and read in each large-block write and large-block read operation respectively, in Kbytes. May not be larger than 8192 (8Mbytes).

replication-factor Simulate the device load you would see if this node was in a cluster with the specified replication-factor. Increasing replication-factor increases the write load, e.g. replication-factor 2 doubles the write load. For act_storage, this doubles the large-block read and write rates. It can also affect the record-sized internal read rate if update-pct is non-zero. The default replication-factor is 1.

update-pct (act_storage ONLY) Simulate the device load you would see if this percentage of write requests were updates, as opposed to replaces. Updates cause the current version of a record to be read before the modified version is written, while replaces do not need to read the current version. Therefore a non-zero update-pct will generate a bigger internal record-sized read rate. E.g. if read-reqs-per-sec is 2000 and write-reqs-per-sec is 1000, the internal read-req rate will be somewhere between 2000 (update-pct 0), and 2000 + 1000 = 3000 (update-pct 100). The default update-pct is 0.

defrag-lwm-pct Simulate the device load you would see if this was the defrag threshold. The lower the threshold, the emptier large blocks are when we defragment them (pack the remaining records into new blocks), and the lower the "write amplification" caused by defragmentation. E.g. if defrag-lwm-pct is 50, the write amplification will be 2x, meaning defragmentation doubles the internal effective storage write rate, which (for act_storage) is manifest as the large-block read and write rates. For act_index, defragmentation generates an extra internal index device read and write load. The default defrag-lwm-pct is 50.

no-defrag-reads (act_storage ONLY) Flag to model Aerospike 7.0+ storage-engine memory with device/file backing. This models defrag for a given write load as usual, but without the large-block reads. To simulate storage-engine memory with device backing, in addition to setting this flag, do not specify a read load (read-reqs-per-sec), and do not set the tomb-raider flag. The default no-defrag-reads is no.

compress-pct (act_storage ONLY) Generate compressible data when writing to devices. With compress-pct 100, the data is fully random (not compressible). Lower values cause runs of zeros to be interleaved with random data such that the data should be compressible to the specified percentage of original size. The compressibility of data may affect performance on some devices, especially those supporting in-line compression. The default compress-pct is 100.

disable-odsync Option to not set O_DSYNC when opening file descriptors. Don't configure this true if configuring commit-to-device. The default disable-odsync is no (i.e. O_DSYNC is set by default).

commit-to-device (act_storage ONLY) Flag to model the mode where Aerospike commits each record to device synchronously, instead of flushing large blocks full of records. This causes a device I/O load with many small, variable-sized writes. Large block writes (and reads) still occur to model defragmentation, but the rate of these is reduced. The default commit-to-device is no.

tomb-raider (act_storage ONLY) Flag to model the Aerospike tomb raider. This simply spawns a thread per device in which the device is read from beginning to end, one large block at a time. The thread sleeps for tomb-raider-sleep-usec microseconds between each block. When the end of the device is reached, we repeat, reading from the beginning. (In other words, we don't model Aerospike's tomb-raider-period.) The default tomb-raider is no.

tomb-raider-sleep-usec (act_storage ONLY) How long to sleep in each device's tomb raider thread between large-block reads. The default tomb-raider-sleep-usec is 1000, or 1 millisecond.

max-lag-sec How much the large-block operations (act_storage) or cache-thread operations (act_index) are allowed to lag behind their target rates before the ACT test fails. Also, how much the service threads that generate and do requests are allowed to lag behind their target rates before the ACT test is stopped. Note that this doesn't necessarily mean the devices failed the test - it means the transaction rates specified are too high to achieve with the configured number of service threads. Note - max-lag-sec 0 is a special value for which the test will not be stopped due to lag. The default max-lag-sec is 10.

act's People

Contributors

bbulkow avatar cricketue avatar cstivers78 avatar deric avatar gooding470 avatar helipilot50 avatar k0st1an avatar kavink avatar kportertx avatar kristi avatar mcoberly2 avatar miko avatar mtendjou avatar pvinh-spike avatar wchu-citrusleaf avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

act's Issues

ACT cannot set the scheduler mode if the device is partitioned using fdisk

Looks like a bug where ACT cannot set scheduler mode to noop if the device is a partition,

Would be nice if we can find the raw devicename in which the partition belongs and set the scheduler mode for that.

Example output of error:

Aerospike act - device IO test
Copyright 2011 by Aerospike. All rights reserved.

CIO CONFIGURATION
device-names: /dev/sde1
num-devices: 1
queue-per-device: no
num-queues: 2
threads-per-queue: 8
test-duration-sec: 86400
report-interval-sec: 1
read-reqs-per-sec: 2000
large-block-ops-per-sec: 24
read-req-num-512-blocks: 3
large-block-op-kbytes: 128
use-valloc: no
num-write-buffers: 1024
scheduler-mode: noop

ERROR: couldn't open /sys/block/sde1/queue/scheduler
/dev/sde1 size = 101146235904 bytes, 197551104 512-byte blocks, 771684 large blocks

ACT TEST1 and TEST2

In the ACT TEST 1 and TEST 2 you never mentions or provide the number threads/queue. A sample will do. Or the number thread irrelevant for the ACT?

If I am using 56 Core, 256 GB, what would be recommended number of threads.

Also for certification do I need to run "storage" and "index" or just storage will do?

Thanks

ACT doesn't distinguish between partitions and devices so setting scheduler doesn't work

since setting a scheduler is being done at the device level, providing a list of partitions results in not actually setting the scheduler at all:

ERROR: couldn't open /sys/block/nvme0n1p2/queue/scheduler errno 2 'No such file or directory'
/dev/nvme0n1p2 size = 1187558457344 bytes, 1132544 large blocks, minimum IO size = 512 bytes
ERROR: couldn't open /sys/block/nvme0n1p3/queue/scheduler errno 2 'No such file or directory'
/dev/nvme0n1p3 size = 1188632199168 bytes, 1133568 large blocks, minimum IO size = 512 bytes
ERROR: couldn't open /sys/block/nvme0n1p4/queue/scheduler errno 2 'No such file or directory'
/dev/nvme0n1p4 size = 1188632199168 bytes, 1133568 large blocks, minimum IO size = 512 bytes
ERROR: couldn't open /sys/block/nvme0n1p5/queue/scheduler errno 2 'No such file or directory'
/dev/nvme0n1p5 size = 1188632199168 bytes, 1133568 large blocks, minimum IO size = 512 bytes
ERROR: couldn't open /sys/block/nvme0n1p6/queue/scheduler errno 2 'No such file or directory'
/dev/nvme0n1p6 size = 1188632199168 bytes, 1133568 large blocks, minimum IO size = 512 bytes
ERROR: couldn't open /sys/block/nvme1n1p2/queue/scheduler errno 2 'No such file or directory'
/dev/nvme1n1p2 size = 1187558457344 bytes, 1132544 large blocks, minimum IO size = 512 bytes
ERROR: couldn't open /sys/block/nvme1n1p3/queue/scheduler errno 2 'No such file or directory'
/dev/nvme1n1p3 size = 1188632199168 bytes, 1133568 large blocks, minimum IO size = 512 bytes
ERROR: couldn't open /sys/block/nvme1n1p4/queue/scheduler errno 2 'No such file or directory'
/dev/nvme1n1p4 size = 1188632199168 bytes, 1133568 large blocks, minimum IO size = 512 bytes
ERROR: couldn't open /sys/block/nvme1n1p5/queue/scheduler errno 2 'No such file or directory'
/dev/nvme1n1p5 size = 1188632199168 bytes, 1133568 large blocks, minimum IO size = 512 bytes
ERROR: couldn't open /sys/block/nvme1n1p6/queue/scheduler errno 2 'No such file or directory'
/dev/nvme1n1p6 size = 1188632199168 bytes, 1133568 large blocks, minimum IO size = 512 bytes
# ls  /sys/block/nvme1n1/queue/scheduler
/sys/block/nvme1n1/queue/scheduler

ACT does not build on Debian 9 (GCE)

On Debian 9 (on a GCE vm-instance) I've installed the dependencies and tried make ; make -f Makesalt. I get the following error:

echo "Building act.o"
Building act.o
gcc -g -fno-common -std=gnu99 -Wall -D_REENTRANT -D_FILE_OFFSET_BITS=64 -MMD  -o act.o -c -I. -I/usr/include act.c
echo "Building histogram.o"
Building histogram.o
gcc -g -fno-common -std=gnu99 -Wall -D_REENTRANT -D_FILE_OFFSET_BITS=64 -MMD  -o histogram.o -c -I. -I/usr/include histogram.c
histogram.c: In function ‘histogram_create’:
histogram.c:101:9: warning: passing argument 1 of ‘memset’ discards ‘volatile’ qualifier from pointer target type [-Wdiscarded-array-qualifiers]
  memset(&h->counts, 0, sizeof(h->counts));
         ^
In file included from histogram.c:37:0:
/usr/include/string.h:62:14: note: expected ‘void *’ but argument is of type ‘volatile cf_atomic64 (*)[65] {aka volatile long unsigned int (*)[65]}’
 extern void *memset (void *__s, int __c, size_t __n) __THROW __nonnull ((1));
              ^~~~~~
echo "Building queue.o"
Building queue.o
gcc -g -fno-common -std=gnu99 -Wall -D_REENTRANT -D_FILE_OFFSET_BITS=64 -MMD  -o queue.o -c -I. -I/usr/include queue.c
echo "Building random.o"
Building random.o
gcc -g -fno-common -std=gnu99 -Wall -D_REENTRANT -D_FILE_OFFSET_BITS=64 -MMD  -o random.o -c -I. -I/usr/include random.c
echo "Linking act"
Linking act
gcc -g -fno-common -std=gnu99 -Wall -D_REENTRANT -D_FILE_OFFSET_BITS=64 -MMD -o act  ./act.o ./histogram.o ./queue.o ./random.o -lpthread -lrt -lcrypto -lssl -lz
/usr/bin/ld: cannot find -lz
collect2: error: ld returned 1 exit status
Makefile:29: recipe for target 'act' failed
make: *** [act] Error 1
echo "Building salt.o"
Building salt.o
gcc -g -fno-common -std=gnu99 -Wall -D_REENTRANT -D_FILE_OFFSET_BITS=64 -MMD  -o salt.o -c -I. -I/usr/include salt.c
echo "Linking actprep"
Linking actprep
gcc -g -fno-common -std=gnu99 -Wall -D_REENTRANT -D_FILE_OFFSET_BITS=64 -MMD -o actprep  ./salt.o ./random.o -lpthread -lrt -lcrypto -lssl -lz
/usr/bin/ld: cannot find -lz
collect2: error: ld returned 1 exit status
Makesalt:29: recipe for target 'actprep' failed
make: *** [actprep] Error 1

actprep build error

I am receiving the error below when trying to build actprep.

matt@matt-work:~/source/act$ make -f Makesalt
echo "Building salt.o"
Building salt.o
gcc -g -fno-common -std=gnu99 -Wall -D_REENTRANT -D_FILE_OFFSET_BITS=64 -MMD -o salt.o -c -I. -I/usr/include salt.c
echo "Linking actprep"
Linking actprep
gcc -g -fno-common -std=gnu99 -Wall -D_REENTRANT -D_FILE_OFFSET_BITS=64 -MMD -o actprep ./salt.o -lpthread -lrt -lssl
./salt.o: In function rand_fill': /home/matt/source/act/salt.c:429: undefined reference toRAND_bytes'
./salt.o: In function rand_seed': /home/matt/source/act/salt.c:458: undefined reference toRAND_seed'
collect2: error: ld returned 1 exit status
make: *** [actprep] Error 1

act_prep take 10x longer in 5.0

Hi,
I started using act 5.0 recently and noticed act 5.0 salting (14MB/sec) take much longer compared to 4.0 (400MB/sec).
After a bit digging, difference seems like caused by O_DSYNC flag. I am wondering will this invalidate my test results if I patch out O_DSYNC flag during salting.
Thanks!

Latency differences between ACT 5.2 and 6.4 versions

I am running ACT benchmark on an NVMe SSD, and seeing odd results between different versions of ACT. Specifically, after act_prep has run, and the drive has reached a steady state, running 40x scaling load via ACT version 5.2 vs. version 6.4 produce very different latency results, running each on the same system and disk. Here is a snippet of the latency results between the two versions. I am using the default config for the specific version in each run, changing only the target read-reqs-per-sec and write-reqs-per-sec to match the target load.

6.4 ACT Results @40x

        reads
        %>(ms)
slice        1      2      4      8     16     32     64
-----   ------ ------ ------ ------ ------ ------ ------
    1    70.96  28.44   3.76   0.05   0.00   0.00   0.00
    2    71.34  28.79   3.93   0.06   0.00   0.00   0.00
    3    71.18  28.79   3.87   0.05   0.00   0.00   0.00
    4    71.13  28.71   3.84   0.05   0.00   0.00   0.00
    5    71.25  28.90   3.99   0.08   0.00   0.00   0.00
    6    71.16  28.69   3.88   0.08   0.01   0.00   0.00
    7    71.17  28.63   3.76   0.05   0.00   0.00   0.00
    8    71.27  28.79   3.88   0.05   0.00   0.00   0.00
    9    71.26  28.94   4.01   0.06   0.00   0.00   0.00
   10    71.19  28.80   3.94   0.06   0.00   0.00   0.00
-----   ------ ------ ------ ------ ------ ------ ------
  avg    71.19  28.75   3.89   0.06   0.00   0.00   0.00
  max    71.34  28.94   4.01   0.08   0.01   0.00   0.00

5.2 ACT Results @40x

        reads                                              device-reads
        %>(ms)                                             %>(ms)
slice        1      2      4      8     16     32     64        1      2      4      8     16     32     64
-----   ------ ------ ------ ------ ------ ------ ------   ------ ------ ------ ------ ------ ------ ------
    1     0.33   0.02   0.01   0.00   0.00   0.00   0.00     0.07   0.01   0.01   0.00   0.00   0.00   0.00
    2     0.33   0.02   0.01   0.00   0.00   0.00   0.00     0.08   0.02   0.01   0.00   0.00   0.00   0.00
    3     0.26   0.00   0.00   0.00   0.00   0.00   0.00     0.04   0.00   0.00   0.00   0.00   0.00   0.00
    4     0.28   0.02   0.01   0.00   0.00   0.00   0.00     0.06   0.02   0.01   0.00   0.00   0.00   0.00
    5     0.27   0.01   0.01   0.00   0.00   0.00   0.00     0.05   0.01   0.01   0.00   0.00   0.00   0.00
    6     0.29   0.02   0.01   0.00   0.00   0.00   0.00     0.07   0.02   0.01   0.00   0.00   0.00   0.00
    7     0.28   0.00   0.00   0.00   0.00   0.00   0.00     0.05   0.00   0.00   0.00   0.00   0.00   0.00
    8     0.29   0.01   0.01   0.00   0.00   0.00   0.00     0.07   0.01   0.00   0.00   0.00   0.00   0.00
    9     0.30   0.02   0.01   0.00   0.00   0.00   0.00     0.07   0.02   0.01   0.00   0.00   0.00   0.00
   10     0.27   0.01   0.00   0.00   0.00   0.00   0.00     0.05   0.01   0.00   0.00   0.00   0.00   0.00
-----   ------ ------ ------ ------ ------ ------ ------   ------ ------ ------ ------ ------ ------ ------
  avg     0.29   0.01   0.01   0.00   0.00   0.00   0.00     0.06   0.01   0.01   0.00   0.00   0.00   0.00
  max     0.33   0.02   0.01   0.00   0.00   0.00   0.00     0.08   0.02   0.01   0.00   0.00   0.00   0.00

I have attached the full results output for both the 5.2 and 6.4 run. I am looking for some pointers on what might be the cause for the significant difference in latency being reported between the two test runs on the versions noted.
ACT-5.2-40x-results.txt
ACT-6.4-40x-results.txt

Stack blowed

In thread subroutine run_service, it uses stack space as the buffer (see line 453 and 465). Given the default number of service thread being 5 times of the cpu cores on the host and the read_size seeming to be a random value, how likely is it to exhaust the stack space so that actually few numbers of threads would be created (and few I/O would be executed than specified, resulting in a superficially higher score) ? Using thread stack this way could be dangerous. Could we change it to use heap memory as the buffer?

partitions disappear on running act_prep script

Followed the instructions and created partitions on my nvme drives. as per this link https://github.com/aerospike/act
As soon as I run act_prep script, all the partitions disappear.
`[root@PCI27 ~]# ls -1 /dev/nvme*
/dev/nvme0
/dev/nvme0n1
/dev/nvme0n1p1
/dev/nvme0n1p10
/dev/nvme0n1p11
/dev/nvme0n1p12
/dev/nvme0n1p13
/dev/nvme0n1p14
/dev/nvme0n1p15
/dev/nvme0n1p16
/dev/nvme0n1p17
/dev/nvme0n1p18
/dev/nvme0n1p19
/dev/nvme0n1p2
/dev/nvme0n1p20
/dev/nvme0n1p21
/dev/nvme0n1p22
/dev/nvme0n1p23
/dev/nvme0n1p24
/dev/nvme0n1p25
/dev/nvme0n1p26
/dev/nvme0n1p27
/dev/nvme0n1p28
/dev/nvme0n1p29
/dev/nvme0n1p3
/dev/nvme0n1p30
/dev/nvme0n1p31
/dev/nvme0n1p32
/dev/nvme0n1p4
/dev/nvme0n1p5
/dev/nvme0n1p6
/dev/nvme0n1p7
/dev/nvme0n1p8
/dev/nvme0n1p9
/dev/nvme1
/dev/nvme1n1
/dev/nvme1n1p1
/dev/nvme1n1p10
/dev/nvme1n1p11
/dev/nvme1n1p12
/dev/nvme1n1p13
/dev/nvme1n1p14
/dev/nvme1n1p15
/dev/nvme1n1p16
/dev/nvme1n1p17
/dev/nvme1n1p18
/dev/nvme1n1p19
/dev/nvme1n1p2
/dev/nvme1n1p20
/dev/nvme1n1p21
/dev/nvme1n1p22
/dev/nvme1n1p23
/dev/nvme1n1p24
/dev/nvme1n1p25
/dev/nvme1n1p26
/dev/nvme1n1p27
/dev/nvme1n1p28
/dev/nvme1n1p29
/dev/nvme1n1p3
/dev/nvme1n1p30
/dev/nvme1n1p31
/dev/nvme1n1p32
/dev/nvme1n1p4
/dev/nvme1n1p5
/dev/nvme1n1p6
/dev/nvme1n1p7
/dev/nvme1n1p8
/dev/nvme1n1p9
/dev/nvme2
/dev/nvme2n1
/dev/nvme2n1p1
/dev/nvme2n1p10
/dev/nvme2n1p11
/dev/nvme2n1p12
/dev/nvme2n1p13
/dev/nvme2n1p14
/dev/nvme2n1p15
/dev/nvme2n1p16
/dev/nvme2n1p17
/dev/nvme2n1p18
/dev/nvme2n1p19
/dev/nvme2n1p2
/dev/nvme2n1p20
/dev/nvme2n1p21
/dev/nvme2n1p22
/dev/nvme2n1p23
/dev/nvme2n1p24
/dev/nvme2n1p25
/dev/nvme2n1p26
/dev/nvme2n1p27
/dev/nvme2n1p28
/dev/nvme2n1p29
/dev/nvme2n1p3
/dev/nvme2n1p30
/dev/nvme2n1p31
/dev/nvme2n1p32
/dev/nvme2n1p4
/dev/nvme2n1p5
/dev/nvme2n1p6
/dev/nvme2n1p7
/dev/nvme2n1p8
/dev/nvme2n1p9
/dev/nvme3
/dev/nvme3n1
/dev/nvme3n1p1
/dev/nvme3n1p10
/dev/nvme3n1p11
/dev/nvme3n1p12
/dev/nvme3n1p13
/dev/nvme3n1p14
/dev/nvme3n1p15
/dev/nvme3n1p16
/dev/nvme3n1p17
/dev/nvme3n1p18
/dev/nvme3n1p19
/dev/nvme3n1p2
/dev/nvme3n1p20
/dev/nvme3n1p21
/dev/nvme3n1p22
/dev/nvme3n1p23
/dev/nvme3n1p24
/dev/nvme3n1p25
/dev/nvme3n1p26
/dev/nvme3n1p27
/dev/nvme3n1p28
/dev/nvme3n1p29
/dev/nvme3n1p3
/dev/nvme3n1p30
/dev/nvme3n1p31
/dev/nvme3n1p32
/dev/nvme3n1p4
/dev/nvme3n1p5
/dev/nvme3n1p6
/dev/nvme3n1p7
/dev/nvme3n1p8
/dev/nvme3n1p9
[root@FM21V213-PCI27 ~]# cd act/target/bin/
[root@FM21V213-PCI27 bin]# sudo ./act_prep /dev/nvme0
nvme0 nvme0n1p13 nvme0n1p19 nvme0n1p24 nvme0n1p3 nvme0n1p6
nvme0n1 nvme0n1p14 nvme0n1p2 nvme0n1p25 nvme0n1p30 nvme0n1p7
nvme0n1p1 nvme0n1p15 nvme0n1p20 nvme0n1p26 nvme0n1p31 nvme0n1p8
nvme0n1p10 nvme0n1p16 nvme0n1p21 nvme0n1p27 nvme0n1p32 nvme0n1p9
nvme0n1p11 nvme0n1p17 nvme0n1p22 nvme0n1p28 nvme0n1p4
nvme0n1p12 nvme0n1p18 nvme0n1p23 nvme0n1p29 nvme0n1p5
[root@FM21V213-PCI27 bin]# sudo ./act_prep /dev/nvme0n1 &
[1] 8590
[root@PCI27 bin]# /dev/nvme0n1 size = 7681501126656 bytes, 58605202 large blocks
cleaning device /dev/nvme0n1
................

##Ran the below steps on all my 4 drives
sudo ./act_prep /dev/nvme0n1 &
sudo ./act_prep /dev/nvme1n1 &
sudo ./act_prep /dev/nvme2n1 &
sudo ./act_prep /dev/nvme3n1 &

Then I check for partitions again and this is what I see

[root@PCI27 bin]# ls -1 /dev/nvme*
/dev/nvme0
/dev/nvme0n1
/dev/nvme1
/dev/nvme1n1
/dev/nvme2
/dev/nvme2n1
/dev/nvme3
/dev/nvme3n1
[root@PCI27 bin]#`

Act_prep and compression

Hi all!

I want to run test act_storage with compression 2 (50%). But act_prep clean and then filling my device without compression. How I can run act_prep with compression?

Incorrect values in analysis histogram

There appears to be a regression in the analysis script after 2cca411 which results in incorrect latency reported in the first histogram bucket.

Steps to reproduce:

  1. Run ACT to produce some output to analyze
  2. Checkout current commit db9961f
  3. Run /analysis/act_latency.py and notice large values in first bucket (bug)
        reads                  device-reads           /dev/nvme0n1-reads     large-block-writes     large-block-reads
        %>(ms)                 %>(ms)                 %>(ms)                 %>(ms)                 %>(ms)
slice        1      2      4        1      2      4        1      2      4        1      2      4        1      2      4
-----   ------ ------ ------   ------ ------ ------   ------ ------ ------   ------ ------ ------   ------ ------ ------
    1    17.50   0.10   0.00     0.03   0.00   0.00     0.03   0.00   0.00   100.00 100.00 100.00     0.00   0.00   0.00
  1. Checkout earlier commit 2cca411
  2. Run /analysis/act_latency.py again and notice correct values in first bucket
        reads                  device-reads           /dev/nvme0n1-reads     large-block-writes     large-block-reads
        %>(ms)                 %>(ms)                 %>(ms)                 %>(ms)                 %>(ms)
slice        1      2      4        1      2      4        1      2      4        1      2      4        1      2      4
-----   ------ ------ ------   ------ ------ ------   ------ ------ ------   ------ ------ ------   ------ ------ ------
    1     0.20   0.03   0.01     0.13   0.02   0.01     0.13   0.02   0.01   100.00 100.00 100.00     0.06   0.03   0.01

ACT Test result is far lower than offical webside

My Samsung 840 Pro 256GB SSD ACT Test result is far lower than offical webside.
Is it config file issue?

My config file:

mandatory non-zero, ignored if queue-per-device is yes:

num-queues: 8

mandatory non-zero:

threads-per-queue: 8
test-duration-sec: 86400
report-interval-sec: 1
read-reqs-per-sec: 6000
large-block-ops-per-sec: 71
read-req-num-512-blocks: 3
large-block-op-kbytes: 128

My test system:
CPU: i7-4770, RAM:DDR3 16GB

Test rsult: > 1 ms (% total / ssd only) = 28.5 / 27.75

ACT profile characteristics

Quoting from https://www.aerospike.com/docs/operations/plan/ssd/ssd_certification.html#pcie-nvme-based-flash These devices were tested at the specified speed with a 67% read/33% write ratio of 1.5 KB objects over 24 hours.

How to determine or specify the read/write ratio in the the config file? does the read/write ratio include both reads and large-block-reads?

1x load has 2000 reads/sec and 1000 writes/sec per device, what would be the read-write ratio here?

Thanks.

[UX] Actprep silently fails if a partition is still mounted from the target device

Running sudo ./actprep /dev/sdN will fail if the disk was previously in-use and had a partition that might still be mounted by accident (and therefore, dd wouldn't fully zero-ize it). It would enhance usability if there was a note hinting to unmount first or atleast a general, noisy error message would show up.

Using ubuntu-server 14.04.3 (LTS)

Should ACT generate a warning if irqbalance disabled?

Wasted a bunch of time testing some instances and eventually gave up on figuring out why the results looked weird compared to other similar instances.. The vendor finally came back and said it was because their OS image had "irqbalance" disabled by default, which caused the IO to stack up on cpu0 and make the ACT results look bad. After two weeks of back and forth, finally came to that conclusion and with irqbalance enabled things look sane.

Because having irqbalance can badly hurt IO performance, can we add a warning to ACT to tell the end user if irqbalance is disabled? Obviously this could be intended but a warning might be a good idea.

Certification question

I've been reading the text on the wiki and I had a question about the act certification criteria for single drives. In the readme I find encountered this line:

Run ACT for 24 hrs using the 3x test (6000 reads/sec and 3000 writes/sec). The drive passes this test if less than 5% of operations fail to complete in 1 ms or less.

Later in the document I encountered these lines:

In any one-hour period of an ACT performance test, we expect that:

fewer than 5% of transactions fail to complete in 1 ms
fewer than 1% of transactions fail to complete in 8 ms
fewer than 0.1% of transactions fail to complete in 64 ms
The max line of the output shows the highest values observed in any single slice (hour) of time and the values on the max line should not exceed the allowable error values specified above.

My question is which of the above is correct? Does the drive pass if " if less than 5% of operations fail to complete in 1 ms or less" for the total number of operations or is it per one hour time period as described in the second quote above?

Let's say I have these results from a drive:

     trans                  device
     %>(ms)                 %>(ms)

slice 1 8 64 1 8 64


 1     1.67   0.00   0.00     1.63   0.00   0.00
 2     1.38   0.00   0.00     1.32   0.00   0.00
 3     1.80   0.14   0.00     1.56   0.08   0.00
 4     6.43   0.00   0.00     1.39   0.00   0.00
 5     1.68   0.00   0.00     1.65   0.00   0.00
 6     1.37   0.00   0.00     1.33   0.00   0.00
 7     1.44   0.00   0.00     1.41   0.00   0.00
 8     1.41   0.00   0.00     1.35   0.00   0.00
 9     2.70   0.73   0.00     1.91   0.08   0.00
10     1.54   0.00   0.00     1.51   0.00   0.00
11     1.53   0.00   0.00     1.48   0.00   0.00
12     1.47   0.00   0.00     1.43   0.00   0.00
13     1.67   0.00   0.00     1.63   0.00   0.00
14     1.38   0.00   0.00     1.32   0.00   0.00
15     1.80   0.14   0.00     1.56   0.08   0.00
16     2.43   0.00   0.00     1.39   0.00   0.00
17     1.68   0.00   0.00     1.65   0.00   0.00
18     1.37   0.00   0.00     1.33   0.00   0.00
19     1.44   0.00   0.00     1.41   0.00   0.00
20     1.41   0.00   0.00     1.35   0.00   0.00
21     2.70   0.73   0.00     1.91   0.08   0.00
22     1.54   0.00   0.00     1.51   0.00   0.00
23     1.53   0.00   0.00     1.48   0.00   0.00
24     1.47   0.00   0.00     1.43   0.00   0.00

avg 1.62 0.07 0.00 1.50 0.01 0.00
max 6.43 0.73 0.00 1.91 0.08 0.00

As we can see at hour 4 the percentage of operations that failed to complete in under 1ms is greater than 5% and thus the drive failed for that time period. As it fails for that one time period then it fails certification at that workload, correct?

Error reporting is too noisy

I was running ACT tests in a cloud provider over the weekend.. Today I noticed they are out of disk.

#:~/act/config# grep -c 'Too many open files' act_250x.conf.out
58379616
#:~/act/config# grep -c '^ERROR' act_250x.conf.out
58433002

-rw-r--r-- 1 root root 3.8G Sep 30 16:36 act_250x.conf.out

Okay, I know error logging is important - but this is getting out of hand.

As far as I know, this error isn't helpful. This happens almost every time we push a system to its limits, especially on cloud systems even with us running as root with no file limit set to 1MM.

# ulimit -Hn
1048576

This also causes problems with the latency interpreter, making it run very slow and sometimes giving bad results (I've seen it but don't have replication steps for the issue).

Can we request some changes here? I'm thinking

  1. have a flag to ignore open files limit? Or just ignore by default. Is it valuable?
  2. Possibly write errors to a separate log file?
  3. Could it aggregate errors instead of printing each individual error line?
  4. Maybe limit the number of errors reported, like 'repeated 23457x bajillion times' instead of logging each individual problem.

I know the #2/#3 point would make it difficult to troubleshoot ACT so I think just having a flag to ignore this kind of error would be a good thing.

Act not quitting properly on Ubuntu 14.04 LTS ('Device can't keep up' message not logged, zombie process)

We ran a few benchmarking rounds by now. On our initial setup we used Ubuntu 14.04 (LTS) and now that we switched to CentOS we noticed that Act under CentOS fails properly / user friendly while Act under Ubuntu would leave only question marks behind.

We noticed that at some point the act process would simply stop dumping into the output file (checked with tail) while the process kept running as normal (using same cpu resources as before!) without any hint what happened. The output didn't contain a device can't keep up message (tried to grep). Process couldn't be killed with SIGTERM or SIGKILL (as root), hinting at some kind of an "waiting for I/O"-deadlock. Out of 4-5x this happened, it only exited smoothly 1x. With CentOS we haven't experience any unsmooth exit so far.

It was only strace that would bring down the process and hint at an output into stdout with the message "device can't keep up". Unfortunately we didn't save the strace output. Can anybody confirm that this is a general issue? Is it supposed/unavoidable to die like this? If not, we are ready to reinstall ubuntu and re-produce the situation, if further input is necessary.

On both OS we started act with an &-operator.

Licence

You've listed this is open source in a blog post, but you don't have a licence included. Which one are you using for this project, and could you include it in the source?

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.