Support for building and running on Joyent SmartOS about hibari

[tatsuya6502]

A great reading material

Examining File System Latency in Production - Joyent SmartOS

[tatsuya6502]

I tried to build, run and dtrace Hibari v0.3-dev on a SmartOS guest with Erlang/OTP R15B installed. It was just a short run but I had no problem to build and run Hibari on SmartOS. I'll run a lengthy test with Basho Bench after I cut a RC (release candidate) of Hibari v0.3.

However, I found a major issue on Erlang DTrace support on SmartOS (#18 (comment)). It's not a show stopper and I hope it's fixable. I'll investigate it when I have more time.

[tatsuya6502]

However, I found a major issue on Erlang DTrace support on SmartOS (#18). It's not a show stopper and I hope it's fixable. I'll investigate it when I have more time.

This issue has been resolved and DTrace works fine on SmartOS. See this comment for the details.

[tatsuya6502]

I tried to build, run and dtrace Hibari v0.3-dev on a SmartOS guest with Erlang/OTP R15B installed. It was just a short run but I had no problem to build and run Hibari on SmartOS. I'll run a lengthy test with Basho Bench after I cut a RC (release candidate) of Hibari v0.3.

I built Hibari v0.3.0 RC on SmartOS and ran a couple of 8-hour stability tests. I found no issues on running Hibari.

However, I found that repo command fails while downloading Hibari source files. I'm not a Python expert and I may not have enough time to investigate this before v0.3 release. So for now, we'll have to workaround this by downloading source files on a different machine with different OS.

% sm-summary 
* Gathering SmartMachine summary..
SM UUID             78c849af-be16-4463-a90a-4e85bbdef5eb
SM ID               1
Hostname            hibari-smartos1
SmartOS build       joyent_20130321T213641Z 
Image               base64 1.9.1
Base Image          NA
Documentation       http://wiki.joyent.com/jpc2/SmartMachine+Base
Pkgsrc              http://pkgsrc.joyent.com/packages/SmartOS/2012Q4/x86_64/All
Processes           24
Memory (RSS) Cap    2048M
Memory (RSS) Used   57M
Memory (RSS) Free   1991M
Swap Cap            2048M
Swap Used           37M
/tmp Used           12K
Disk Quota          21G
% Disk Used         22%

% python --version
Python 2.7.3

% ./priv/scripts/x-build.sh
...

+ ../repo init -u git://github.com/hibari/manifests.git -m hibari-default.xml -b release-0.3
Get https://gerrit.googlesource.com/git-repo
...

 * [new tag]         v1.9.6     -> v1.9.6
Get git://github.com/hibari/manifests.git
remote: Counting objects: 261, done.
remote: Compressing objects: 100% (175/175), done.
remote: Total 261 (delta 162), reused 185 (delta 86)
Receiving objects: 100% (261/261), 440.00 KiB | 183 KiB/s, done.
Resolving deltas: 100% (162/162), done.
From git://github.com/hibari/manifests
 * [new branch]      dev        -> origin/dev
 * [new branch]      master     -> origin/master
 * [new branch]      release-0.3 -> origin/release-0.3
 * [new branch]      release-v0.1.10 -> origin/release-v0.1.10
 * [new tag]         v0.1.0     -> v0.1.0
 * [new tag]         v0.1.1     -> v0.1.1
 * [new tag]         v0.1.10    -> v0.1.10
 * [new tag]         v0.1.2     -> v0.1.2
 * [new tag]         v0.1.4     -> v0.1.4
 * [new tag]         v0.1.5     -> v0.1.5
 * [new tag]         v0.1.6     -> v0.1.6
 * [new tag]         v0.1.7     -> v0.1.7
 * [new tag]         v0.1.8     -> v0.1.8
 * [new tag]         v0.1.9     -> v0.1.9
Traceback (most recent call last):
  File "/home/tatsuya/hibari-qa/hibari/.repo/repo/main.py", line 414, in <module>
    _Main(sys.argv[1:])
  File "/home/tatsuya/hibari-qa/hibari/.repo/repo/main.py", line 390, in _Main
    result = repo._Run(argv) or 0
  File "/home/tatsuya/hibari-qa/hibari/.repo/repo/main.py", line 138, in _Run
    result = cmd.Execute(copts, cargs)
  File "/home/tatsuya/hibari-qa/hibari/.repo/repo/subcmds/init.py", line 348, in Execute
    self._LinkManifest(opt.manifest_name)
  File "/home/tatsuya/hibari-qa/hibari/.repo/repo/subcmds/init.py", line 217, in _LinkManifest
    self.manifest.Link(name)
  File "/home/tatsuya/hibari-qa/hibari/.repo/repo/manifest_xml.py", line 126, in Link
    self.Override(name)
  File "/home/tatsuya/hibari-qa/hibari/.repo/repo/manifest_xml.py", line 119, in Override
    self._Load()
  File "/home/tatsuya/hibari-qa/hibari/.repo/repo/manifest_xml.py", line 334, in _Load
    self.manifestProject.worktree))
  File "/home/tatsuya/hibari-qa/hibari/.repo/repo/manifest_xml.py", line 369, in _ParseManifestXml
    except (OSError, xml.parsers.expat.ExpatError) as e:
AttributeError: 'module' object has no attribute 'expat'

[tatsuya6502]

I tried to build, run and dtrace Hibari v0.3-dev on a SmartOS guest with Erlang/OTP R15B installed. It was just a short run but I had no problem to build and run Hibari on SmartOS. I'll run a lengthy test with Basho Bench after I cut a RC (release candidate) of Hibari v0.3.

I built Hibari v0.3.0 RC on SmartOS and ran a couple of 8-hour stability tests. I found no issues on running Hibari.

However, I found that repo command fails while downloading Hibari source files. I'm not a Python expert and I may not have enough time to investigate this before v0.3 release. So for now, we'll have to workaround this by downloading source files on a different machine with different OS.

In past two years, Joyent world has changed. Now SmartOS is able to run unmodified Linux 64-bit executables in a Solaris zone called lx-brand (Linux brand). Also their open-sourced Triton platform (formerly Smart Data Center, or SDC) built on SmartOS now supports Docker API, so you can run unmodified Linux Docker containers on Triton.

This is amazing because lx-brand containers runs on illumos kernel (instead of Linux kernel) thus we can take every advantages from whole illumos stack; ZFS, DTrace and Crossbow (software-defined network) while being more secure than Linux Docker.

Because of this big change, I no longer have to fix repo for joyent-branded zone (the SmartOS native zone), because Hibari seems just work in lx-branded zone. These brands should not have any differences in features and performance, so there is no reason to use joyent-brand.

Here is some terminal log from my SmarOS-based home server. I used CentOS 6.7 20151005 image on SmartOS 20151112T052215Z. See the outputs from uname and /native/usr/bin/uname. Executables are running on illumos kernel but they think they are running on Linux kernel.

% ssh -A [email protected]
- SmartOS Live Image v0.147+ build: 20151112T052215Z
...
[root@28-92-4a-38-fd-6b ~]# vmadm list -s alias
UUID                                  TYPE  RAM      STATE             ALIAS
...
8bbccada-85d0-4cd7-b178-10473feb6e99  LX    4096     running           lx-centos-6c
...
[root@28-92-4a-38-fd-6b ~]# zlogin 8bbccada-85d0-4cd7-b178-10473feb6e99 
[Connected to zone '8bbccada-85d0-4cd7-b178-10473feb6e99' pts/2]
Last login: Tue Nov 24 09:01:59 from zone:global
   __        .                   .
 _|  |_      | .-. .  . .-. :--. |-
|_    _|     ;|   ||  |(.-' |  | |
  |__|   `--'  `-' `;-| `-' '  ' `-'
                   /  ;  Instance (CentOS 6.7 20151005)
                   `-'   https://docs.joyent.com/images/container-native-linux

[root@lx-centos-6c ~]# su - hibari

[hibari@lx-centos-6c ~]$ uname -a       
Linux lx-centos-6c 3.13.0 BrandZ virtual linux x86_64 x86_64 x86_64 GNU/Linux

[hibari@lx-centos-6c ~]$ /native/usr/bin/uname -a
SunOS lx-centos-6c 5.11 joyent_20151112T052215Z i86pc i386 i86pc

[hibari@lx-centos-6c ~]$ source /usr/local/erlang/18.1.4_hipe/activate 
[hibari@lx-centos-6c ~]$ erl
Erlang/OTP 18 [erts-7.1] [source] [64-bit] [smp:2:2] [async-threads:10] [hipe] [kernel-poll:false]

Eshell V7.1  (abort with ^G)
1> 
User switch command
 --> q

[hibari@lx-centos-6c ~]$ mkdir hibari
[hibari@lx-centos-6c ~]$ cd hibari
[hibari@lx-centos-6c hibari]$ /usr/local/bin/repo init \
    -u git://github.com/hibari/manifests.git -m hibari-default.xml \
    -b gbrick-gh17-redesign-disk-storage
...
Get https://github.com/android/tools_repo.git
...
Get git://github.com/hibari/manifests.git
...

Your identity is: Tatsuya Kawano <[email protected]>
If you want to change this, please re-run 'repo init' with --config-name
...
repo has been initialized in /home/hibari/hibari

[hibari@lx-centos-6c hibari]$ /usr/local/bin/repo sync
...
Fetching projects: 100% (25/25), done.  
Syncing work tree: 100% (25/25), done.  

[hibari@lx-centos-6c hibari]$ cd hibari
[hibari@lx-centos-6c hibari]$ repo forall -c "git branch dev origin/dev"
[hibari@lx-centos-6c hibari]$ repo forall -c "git checkout dev"
[hibari@lx-centos-6c hibari]$ git checkout gbrick-gh17-redesign-disk-storage
[hibari@lx-centos-6c hibari]$ (cd lib/gdss_admin && git checkout gbrick-gh17-redesign-disk-storage)
[hibari@lx-centos-6c hibari]$ (cd lib/gdss_brick && git checkout gbrick-gh17-redesign-disk-storage)
[hibari@lx-centos-6c hibari]$ (cd lib/h2leveldb && git checkout develop)
[hibari@lx-centos-6c hibari]$ (cd lib/HyperLevelDB && git checkout master)
[hibari@lx-centos-6c hibari]$ (cd lib/snappy && git checkout master)
[hibari@lx-centos-6c hibari]$ repo status
project hibari-doc/                             branch dev
project hibari/                                 branch gbrick-gh17-redesign-disk-storage
project hibari/lib/asciiedoc/                   branch dev
project hibari/lib/HyperLevelDB/                branch master
project hibari/lib/bear/                        branch dev
project hibari/lib/cluster_info/                branch dev
project hibari/lib/congestion_watcher/          branch dev
project hibari/lib/edown/                       branch dev
project hibari/lib/folsom/                      branch dev
project hibari/lib/gdss_admin/                  branch gbrick-gh17-redesign-disk-storage
project hibari/lib/gdss_brick/                  branch gbrick-gh17-redesign-disk-storage
project hibari/lib/gdss_client/                 branch dev
project hibari/lib/gdss_ubf_proto/              branch dev
project hibari/lib/gmt_util/                    branch dev
project hibari/lib/h2leveldb/                   branch develop
project hibari/lib/lager/                       branch dev
project hibari/lib/meck/                        branch dev
project hibari/lib/partition_detector/          branch dev
project hibari/lib/pmod_transform/              branch dev
project hibari/lib/qc/                          branch dev
project hibari/lib/sext/                        branch dev
project hibari/lib/snappy/                      branch master
project hibari/lib/ubf/                         branch dev
project hibari/lib/ubf_thrift/                  branch dev
project patches/                                branch dev

[hibari@lx-centos-6c hibari]$ make check-package
cleaning: hibari-0.3.0-dev-x86_64-unknown-linux-gnu-64 ...
./rebar -r clean
...
compiling: hibari-0.3.0-dev-x86_64-unknown-linux-gnu-64 ...
./rebar compile
...
generating: hibari-0.3.0-dev-x86_64-unknown-linux-gnu-64 ...
./rebar generate
==> rel (generate)
WARN:  'generate' command does not apply to directory /home/hibari/hibari/hibari
packaging: hibari-0.3.0-dev-x86_64-unknown-linux-gnu-64 ...
f4e1a68ab2a6d6a51f22b8e1817d80fb  hibari-0.3.0-dev-x86_64-unknown-linux-gnu-64.tgz
-rw-rw-r-- 1 hibari hibari       83 Nov 25 01:00 hibari-0.3.0-dev-x86_64-unknown-linux-gnu-64-md5sum.txt
-rw-rw-r-- 1 hibari hibari 21311054 Nov 25 01:00 hibari-0.3.0-dev-x86_64-unknown-linux-gnu-64.tgz
bootstrapping package: hibari-0.3.0-dev-x86_64-unknown-linux-gnu-64 ...
make: [bootstrap-dirty-package] Error 127 (ignored)
tar -C ./tmp -xzf ../hibari-0.3.0-dev-x86_64-unknown-linux-gnu-64.tgz
./tmp/hibari/bin/hibari start
./tmp/hibari/bin/hibari-admin bootstrap
ok
checking package: hibari-0.3.0-dev-x86_64-unknown-linux-gnu-64 ...
./tmp/hibari/bin/hibari-admin client-add [email protected]
ok
./tmp/hibari/bin/hibari-admin client-list
['[email protected]']
./tmp/hibari/bin/hibari-admin client-delete [email protected]
ok
./tmp/hibari/bin/hibari stop
ok
./tmp/hibari/bin/hibari start
./tmp/hibari/bin/hibari ping
pong
./tmp/hibari/bin/hibari stop
ok
[hibari@lx-centos-6c hibari]$ 

I am going to close this issue. I removed the release notes entry (1e66a0d).