COSBench is a benchmarking tool to measure the performance of Cloud Object Storage services. Object storage is an emerging technology that is different from traditional file systems (e.g., NFS) or block device systems (e.g., iSCSI). Amazon S3 and Openstack* swift are well-known object storage solutions.
COSBench now supports OpenStack* Swift and Amplidata v2.3, 2.5 and 3.1, as well as custom adaptors.
As part of our project we implemented a simple prefetching system for Ceph. We use COSBench to evaluate the performance of Ceph with and without the prefetching system. The prefetching is triggered when there is a prefetch header introduced on GET request for an object on or when there is range request in bytes for an object in S3. COSBench does not support range request or sending customized headers on S3. So, we modified COSBench to meet our needs.
Populate the config attribute of the storage element with is_range_request, file_length and chunk_length. Here is an example for a workstage:
<workstage>
<work>
<storage type="s3" config="is_range_request=true;file_length=<file_length>;chunk_length=<chunk_length>;path_style_access=true" />
<operation type="read" />
</work>
</workstage>
- is_range_request=true makes usual READ operation not to read whole file instead it only reads the chunk of the file
- file_length is long number which indicated the size of the file in bytes
- chunk_length is long number where you indicate how many bytes you want to read from each file in the buckets
chunk_length < file_length, so what algorithm does is it will randomly request for chunk_length bytes between 0 and file_length - chunk_length
Please refer to s3-config-range-sample.xml for a complete workload configuration. In this example, we are creating 1 bucket, 10 objects of size 15MB. We are running RANGE workload for 5MB.
Populate the config attribute of the storage element with is_range_request, file_length and chunk_length. Here is an example for a workstage:
<workstage>
<work>
<storage type="s3" config="is_prefetch=true;path_style_access=true" />
<operation type="read" />
</work>
</workstage>
- is_prefetch=true makes usual READ operation not to read file at all. This is only related to our project, so prefetch would send the user 200 OK status and start copying the object to Rados Gateway
Please refer to s3-config-prefetch-sample.xml for a complete workload configuration. In this example, we are creating 1 bucket, 10 objects of size 4MB. We are running PREFETCH workload for 4MB.
- Make sure to add the flag -Dcom.amazonaws.services.s3.disableGetObjectMD5Validation=true to cosbench-start.sh. Otherwise, you will face the following error when trying to add an object to the bucket com.intel.cosbench.api.storage.StorageException: com.amazonaws.services.s3.model.AmazonS3Exception: null (Service: Amazon S3; Status Code: 403; Error Code: SignatureDoesNotMatch;
- Make sure path_style_access=true is introduced in the config attribute. This also might lead to erros when working with S3 PUT operations.
- Obtain installation package for pre-release v0.4.2.c4 from github (Releases) and unzip it under the /home directory on the node.
wget https://github.com/intel-cloud/cosbench/archive/v0.4.2.c4.zip unzip 0.4.2.c4.zip cd 0.4.2.c4 chmod +x *.sh- The latest release v0.4.2 is broken
- If v0.4.2 is used, there will be an error in controller-boot.log: "Could not find or load main class org.eclipse.equinox.launcher.Main"
- ./start-all.sh will run forever printing ".Ncat: Connection refused." repeatedly which is open issue #386
- The suggested solution at #240 changing TOOL_PARAMS="-i 0" to TOOL_PARAMS="" in cosbench-start.sh does not solve the issue
- Install dependencies
sudo yum install -y java-1.8.0-openjdk sudo yum install nc
- Start COSBench controller and COSBench driver
./start-all.sh - Start COSBench Drivers
./start-driver.sh - Start COSBench Controller
./start-controller.sh
Please refer to COSBench installation for additional instructions. Please refer to "COSBenchUserGuide.pdf" for details.
- Submitting a workload
./cli.sh submit <path_to_xml_file> - path_to_xml_file is the path for the configuration file
-
Download Eclipse Installer
-
Install Eclipse IDE for Enterprise Java Developers, not Eclipse IDE for Java Developers
-
Download COSBench as mentioned above and import to Eclipse according to the instructions below
- Eclipse IDE for Java does not have Plug-in Development in Preferences
COSBench is composed of independent projects. So, if you want to make a specific change for some project like in our case (we only modified cosbench-s3 project)
- Make sure that your project can be built and there is no errors
- Select the project and click "Export -> Plug-in Development -> Deployable plugins and fragments"
- Set the "Directory" to "dist\osgi" folder
- At "dist\osgi\plugins" folder, there will be generated a .jar file
- Stop COSBench ./stop-all.sh
- Copy the generated .jar file to the node where COSBench installed to the directory osgi/plugins and replace it
- Run COSBench ./start-all.sh to test the changes you made
Please refer to Setting Up Development Environment for additional instructions. If a build from source code is needed, please refer to BUILD.md for details.
a) Intel source code is being released under the Apache 2.0 license.
b) Additional libraries used with COSBench have their own licensing; refer to 3rd-party-licenses.pdf for details.
Please refer to "DISTRIBUTIONS.md" to get the link for distribution packages.
If needed, adaptors can be developed for new storage services; please refer to "COSBenchAdaptorDevGuide.pdf" for details.
Wiki: (https://github.com/intel-cloud/cosbench/wiki)
Issue tracking: (https://github.com/intel-cloud/cosbench/issues)
Mailing list: (http://cosbench.1094679.n5.nabble.com/)
COSBench-Workload-Generator: (https://github.com/giteshnandre/COSBench-Workload-Generator)
COSBench-Plot: (https://github.com/icclab/cosbench-plot)
React
Typescript
Django
Laravel
Facebook
Microsoft
Google
Alibaba
D3
Tencent