support for Centos 7 (Scientific Unix 7) about aerospike-client-hhvm

[rbotzer]

Hi Andrew. We intend to support all 64-bit Linux variants, but we'll probably do it once HHVM does so as well. Once HHVM reaches the next LTS (3.9.0) we will push to get wider support, and until then we intend to continue testing with resent releases on Ubuntu 14.04. For example, HHVM 3.8.0 was released just a couple of hours ago.

[andrewmilkowski]

thank you @rbotzer , it is imperative for my org, that all interfaces are as stable as they possibly can as we intend to use this in high volume production environment

put a watch on this topic (and facebook HHVM), meantime re-pulling latest master from here and attempting HHVM 3.8.0 assembly

thanks for your fast feedback on this, we look forward to speed gains provided by this client...

[andrewmilkowski]

@rbotzer HHVM 3.8.0 assembly fails on Centos 7 so this is pretty much a blocker for us.. for now

[rbotzer]

Did you look at their instructions for using other pre-built packages or building HHVM manually?

[andrewmilkowski]

yes, git master succeeds... then git checkout HHVM-3.8 and below and error

cmake -DCMAKE_INSTALL_PREFIX=/opt/local/src/git/facebook/hhvm/dist .
make -j$(($(nproc)+1)) # make with CORE_COUNT+1 threads, that would be fast. You may run out of RAM

./hphp/hhvm/hhvm --version
make install

let me do fresh check out fresh new copy and re-assembly as sanity check

[andrewmilkowski]

@rbotzer fresh new git clone (and git checkout HHVM-3.8 from within), did the trick... HVVM-3.8 is assembling (pass 48 step error) then I will tie it into this areospike client and repeat the test.. (must have been old artifacts) appreciate it ...

[andrewmilkowski]

• actually hhvm does not start file (daemon mode) *... hold a sec investigating

@rbotzer , using HHVM-3.8 on Centos 7, so HHVM starts up fine: below is command line process list and php.ini used

1. /opt/local/src/git/facebook/hhvm/dist/bin/hhvm --config /etc/hhvm/server.ini --mode daemon -vServer.Type=fastcgi -vServer.Port=9000
2. /etc/hhvm/server.ini :

pid = /var/run/hhvm/hhvm.pid

hhvm.server.port = 9000
hhvm.server.type = fastcgi
hhvm.server.ip = 127.0.0.1
hhvm.server.default_document = index.php
hhvm.log.use_log_file = true
hhvm.log.header = true
hhvm.log.file = /var/log/hhvm/error.log
hhvm.repo.central.path = /var/run/hhvm/hhvm.hhbc
hhvm.server.thread_count = 40
hhvm.server.expose_hphp = false
hhvm.server.request_timeout_seconds = 8
hhvm.repo.authoritative = true
hhvm.extensions[] = /opt/local/src/git/aerospike/aerospike-client-hhvm/dist/lib64/hhvm/extensions/20150212/aerospike-hhvm.so

[amilkowski@localhost aerospike]$ ps aux | grep hhvm
root 101539 0.0 0.0 210744 3448 pts/1 S+ 18:30 0:00 sudo tail -f /var/log/hhvm/error.log
root 101542 0.0 0.0 107932 616 pts/1 S+ 18:30 0:00 tail -f /var/log/hhvm/error.log
amilkow+ 106403 0.0 0.0 112648 972 pts/2 S+ 18:34 0:00 grep --color=auto hhvm

then doing below causes "Seg Fault"

[amilkowski@localhost aerospike]$ pwd
/opt/local/src/git/aerospike/aerospike-client-hhvm/src/aerospike
[amilkowski@localhost aerospike]$ cp tests/aerospike.inc tests/aerospike.local.inc
[amilkowski@localhost aerospike]$ scripts/test.sh tests/phpt/Put
Core dumped: Segmentation fault
Stack trace in /tmp/stacktrace.105296.log
scripts/test.sh: line 48: 105296 Segmentation fault (core dumped) $TEST_PHP_EXECUTABLE --php -z ${AEROSPIKE_EXT_PATH}/aerospike-hhvm.so ${AEROSPIKE_EXT_PATH}/run-tests.php $@
[amilkowski@localhost aerospike]$

[andrewmilkowski]

@rbotzer , ok sorry about.. have the "use case"

HHVM by itself works perfect on Centos 7 with fastcgi, sample php info page displays "PHP Version 5.6.99-hhvm"

iff/when I add aerospike extension such as below, segment fault results, so we can not use Centos 7 with this solution.. (for now) is there any other help I can yield to trace Centos 7 issue @rbotzer , such as dump analysis , otherwise it is unfortunate.. speed ups would be great for this client/driver...

hhvm.extensions[] = /opt/local/src/git/aerospike/aerospike-client-hhvm/dist/lib64/hhvm/extensions/20150212/aerospike-hhvm.so

[Mon Jul 13 21:03:08 2015] [hphp] [20646:7f8969c34a40:0:000007] [] loading static content took 0'00" (1 us) wall time
[Mon Jul 13 21:03:08 2015] [hphp] [20646:7f8969c34a40:0:000008] [] Core dumped: Segmentation fault
[Mon Jul 13 21:03:08 2015] [hphp] [20646:7f8969c34a40:0:000009] [] Stack trace in /tmp/stacktrace.20646.log

[rbotzer]

Please echo that stack trace file - that will help.

[andrewmilkowski]

empty, nothing following "PHP Stacktrace"

[amilkowski@localhost aerospike]$ sudo tail -f /tmp/stacktrace.20646.log

PHP Stacktrace:

[andrewmilkowski]

also for reference apache2 server is running with fastcgi (installed via these simplest possible instruction, sorry for Spanish. but it was simplest bare bone I could find: https://gist.github.com/marciopaiva/5c8087e53691d50a046b)

command line was:

/opt/local/src/git/facebook/hhvm/dist/bin/hhvm --config /etc/hhvm/server.ini --mode daemon -vServer.Type=fastcgi -vServer.Port=9000

below server.ini file:

pid = /var/run/hhvm/hhvm.pid

hhvm.server.port = 9000
hhvm.server.type = fastcgi
hhvm.server.ip = 127.0.0.1
#hhvm.server.default_document = index.php
hhvm.log.use_log_file = true
hhvm.log.header = true
hhvm.log.file = /var/log/hhvm/error.log
#hhvm.repo.central.path = /var/run/hhvm/hhvm.hhbc
hhvm.server.thread_count = 40
hhvm.server.expose_hphp = false
hhvm.server.request_timeout_seconds = 8
#hhvm.repo.authoritative = true
hhvm.extensions[] = /opt/local/src/git/aerospike/aerospike-client-hhvm/dist/lib64/hhvm/extensions/20150212/aerospike-hhvm.so <----- causes segment fault and empty stack trace

[andrewmilkowski]

HHVM itself supports Centos 7

https://github.com/facebook/hhvm/wiki/Building-and-installing-hhvm-on-CentOS-7.x

---

just checked hhvm version, still on the wrong one, HipHop VM 3.9.0-dev, making sure it is 3.8.0 and retesting

[root@localhost ccpp-2015-07-14-06:30:05-6546]# /opt/local/src/git/facebook/hhvm/dist/bin/hhvm --version
HipHop VM 3.9.0-dev (rel)
Compiler: heads/master-0-g9e8ee235b09f8e11bb66fe936d61116299d7131b
Repo schema: 691afd7259988ddbd73599b525e4c3c287075676

---

following aerospike c client was linked in

[amilkowski@localhost aerospike-client-c]$ git branch -a

• (detached from 3.1.16)
  master
  remotes/origin/HEAD -> origin/master
  remotes/origin/master

all tests pass against aerospike installed from aerospike-server-community-3.5.14-el6.tgz

make test

..
..

SUMMARY

[✔] key_basics: 13/13 tests passed.
[✔] key_apply: 7/7 tests passed.
[✔] key_apply2: 14/14 tests passed.
[✔] key_operate: 3/3 tests passed.
[✔] info_basics: 2/2 tests passed.
[✔] udf_basics: 1/1 tests passed.
[✔] udf_types: 10/10 tests passed.
[✔] udf_record: 3/3 tests passed.
[✔] index_basics: 2/2 tests passed.
[✔] query_foreach: 8/8 tests passed.
[✔] scan_basics: 10/10 tests passed.
[✔] batch_get: 5/5 tests passed.
[✔] policy_read: 5/5 tests passed.
[✔] policy_scan: 5/5 tests passed.
[✔] ldt_lmap: 1/1 tests passed.

89 tests: 89 passed, 0 failed

---

same "segment fault" w/ HipHop VM 3.8.1-dev (hhvm with fastcgi by itself works just fine)

/opt/local/src/git/facebook/hhvm/dist/bin/hhvm --version
HipHop VM 3.8.1-dev (rel)
Compiler: heads/HHVM-3.8-0-gff62432c080a51d2bcd54dfcfe9aca81bba758a8
Repo schema: 57d3d7404b0fae47066d233aba33a64e9a198635

---

/var/log/messages last trace before SIGINT

Jul 13 12:57:40 localhost kernel: aerospike-hhvm.[71053]: segfault at 1 ip 0000000000000001 sp 00007fffe48e6608 error 14 in aerospike-hhvm.so[7fd1f6b99000+35000]

---

core_backtrace casing SIGINT

"signal": 11
, "executable": "/opt/local/src/git/facebook/hhvm/dist/bin/hhvm"
, "stacktrace":
[ { "crash_thread": true
, "frames":
[ { "address": 140522706258196
, "build_id": "bfe527c0930746301f12f0b174ab129096f38771"
, "build_id_offset": 104724
, "function_name": "HPHP::AerospikeExtension::moduleInit()"
, "file_name": "/opt/local/src/git/aerospike/aerospike-client-hhvm/dist/lib64/hhvm/extensions/20150212/aerospike-hhvm.so"
}
, { "address": 22929353
, "build_id": "f668c88368667cbf59e4e6c4ac1122a98502ce10"
, "build_id_offset": 18735049
, "function_name": "HPHP::ExtensionRegistry::moduleInit()"
, "file_name": "/opt/local/src/git/facebook/hhvm/dist/bin/hhvm"
}
, { "address": 12956309
, "build_id": "f668c88368667cbf59e4e6c4ac1122a98502ce10"
, "build_id_offset": 8762005
, "function_name": "HPHP::hphp_process_init()"
, "file_name": "/opt/local/src/git/facebook/hhvm/dist/bin/hhvm"
}
, { "address": 14853995
, "build_id": "f668c88368667cbf59e4e6c4ac1122a98502ce10"
, "build_id_offset": 10659691
, "function_name": "HPHP::HttpServer::HttpServer()"
, "file_name": "/opt/local/src/git/facebook/hhvm/dist/bin/hhvm"
}
, { "address": 13046752
, "build_id": "f668c88368667cbf59e4e6c4ac1122a98502ce10"
, "build_id_offset": 8852448
, "function_name": "HPHP::execute_program_impl(int, char**)"
, "file_name": "/opt/local/src/git/facebook/hhvm/dist/bin/hhvm"
}
, { "address": 13053551
, "build_id": "f668c88368667cbf59e4e6c4ac1122a98502ce10"
, "build_id_offset": 8859247
, "function_name": "HPHP::execute_program(int, char**)"
, "file_name": "/opt/local/src/git/facebook/hhvm/dist/bin/hhvm"
}
, { "address": 11299585
, "build_id": "f668c88368667cbf59e4e6c4ac1122a98502ce10"
, "build_id_offset": 7105281
, "function_name": "main"
, "file_name": "/opt/local/src/git/facebook/hhvm/dist/bin/hhvm"
} ]
} ]
}

[andrewmilkowski]

isolated, SIGINT to these lines of code (commented them out) extension is loaded, obviously you need these registrations, but hopefully helps troubleshooting @rbotzer

something very likely connects to "EXTENSION_CONSTANTS_SIZE" being incorrect

class AerospikeExtension : public Extension {
    public:
        AerospikeExtension(): Extension("aerospike", "1.0") {}
        void moduleInit() override
        {

/*
for (uint32_t i = 0; i <= EXTENSION_CONSTANTS_SIZE; i++) {
Native::registerClassConstant(s_Aerospike.get(),
StaticString(extension_constants[i].constant_name).get(),
extension_constants[i].constant_no);
}
*/

[rbotzer]

@andrewmilkowski it does help, thanks.

[andrewmilkowski]

appreciate it!, this extension you wrote would be great performance boost for us as you can imagine, anything else you need shoot message immediate, will respond right back

thanks so much

[andrewmilkowski]

just dumped EXTENSION_CONSTANTS_SIZE : 96

off by one 95 succeeds, rusty a bit with C++ , so could be the array size

shouldn't the line read

for (uint32_t i = 0; i < EXTENSION_CONSTANTS_SIZE; i++) (instead off <=)

above loads the extension, trying basic aerospike operation next (like connect)

what is strange is how this worked for others... regardless of OS/g++

[andrewmilkowski]

running test suite

[amilkowski@localhost aerospike]$ pwd
/opt/local/src/git/aerospike/aerospike-client-hhvm/src/aerospike

scripts/test.sh tests/phpt/Put

still does not PASS , regardless of the "fix" (on the first error, something else is critically wrong)

[amilkowski@localhost aerospike]$ pwd
/opt/local/src/git/aerospike/aerospike-client-hhvm/src/aerospike
[amilkowski@localhost aerospike]$ vi run-local-test.sh
[amilkowski@localhost aerospike]$ ./run-local-test.sh
*** EXTENSION_CONSTANTS_SIZE : 96
sh: /usr/bin/hhvm: No such file or directory

PHP : /opt/local/src/git/facebook/hhvm/dist/bin/hhvm
PHP_SAPI : cli
PHP_VERSION : 5.6.99-hhvm
ZEND_VERSION: 2.4.99
HHVM_VERSION:
PHP_OS : Linux - Linux localhost.localdomain 3.10.63-11.el7.x86_64 #1 SMP Tue Apr 7 16:47:43 UTC 2015 x86_64
INI actual : /opt/local/src/git/aerospike/aerospike-client-hhvm/src/aerospike
More .INIs :
CWD : /opt/local/src/git/aerospike/aerospike-client-hhvm/src/aerospike
Extra dirs :
VALGRIND : Not used

Running selected tests.
FAIL Put - Nested List with default options. [tests/phpt/Put/ChecNestedList.phpt]

---

0 passed but no SIGINT

Number of tests : 77 77
Tests skipped : 0 ( 0.0%) --------
Tests warned : 0 ( 0.0%) ( 0.0%)
Tests failed : 66 ( 85.7%) ( 85.7%)
Expected fail : 11 ( 14.3%) ( 14.3%)
Tests passed : 0 ( 0.0%) ( 0.0%)

Time taken : 18 seconds

[andrewmilkowski]

@rbotzer

yay... below succeeds with this extension Centos 7 and HHVM 3.8.1-dev (from git branch), connect/put/get (with the for (uint32_t i = 0; i < EXTENSION_CONSTANTS_SIZE; i++) (instead off <=)

so its just test failures, we also nee to be able to run php as "standalone" not tied to the webserver...

array(array("addr"=>$HOST_ADDR, "port"=>$HOST_PORT))); $db = new Aerospike($config); $db = new Aerospike($config, false); if (!$db->isConnected()) { echo "Could not connect to host $HOST_ADDR:$HOST_PORT [{$db->errorno()}]: {$db->error()}"; exit(1); } echo "connection established successfully
"; echo "Adding a bin to the record if Aerospike::POLICY_GEN_EQ
"; $key = $db->initKey("test", "characters", 1); $start = **LINE**; $current_gen = $meta["generation"]; $new_bin = array("eyesight"=>"bad"); $status = $db->put($key, $new_bin, 0, array(Aerospike::OPT_POLICY_GEN=>array(Aerospike::POLICY_GEN_EQ, $current_gen))); if ($status == Aerospike::OK) { echo "success"; } elseif ($status == Aerospike::ERR_RECORD_GENERATION) { echo "The record at PK={$key['key']} has been altered and has a newer generation"; } else { echo "fail $db "; } echo "Getting the recordget($key, $record); if ($status == Aerospike::OK) { echo "success
"; var_dump($record); } elseif ($status == Aerospike::ERR_RECORD_NOT_FOUND) { echo "Could not find a user with PK={$key['key']} in the set test.characters"; } else { echo standard_fail($db); } ?>

[rbotzer]

That's good news. Usually the webserver context is the more complex one than the CLI, so I expect we can figure it out soon.

[andrewmilkowski]

perfect... thank you Ronen, @rbotzer so this "Please use a standard build (without the -l flag), as a debug build will cause the tests to fail." I enabled by the following (still failing on tests but just wanted to confirm)

did not see "-I" flag I did this though

#set(CMAKE_BUILD_TYPE Debug)
set(CMAKE_BUILD_TYPE Release)

(actually we want to assemble final builds as "release" builds for the ultimate performance

vi src/aerospike/config.cmake

HHVM_EXTENSION(aerospike-hhvm
main/ext_aerospike.cpp
main/conversions.cpp
main/policy.cpp
main/helper.cpp
main/batch_op_manager.cpp
main/scan_operation.cpp)
HHVM_DEFINE(aerospike-hhvm -DAEROSPIKE_C_CHECK)
include_directories(include)
#target_link_libraries(aerospike-hhvm /usr/lib/libaerospike.so)
target_link_libraries(aerospike-hhvm /opt/local/src/git/aerospike/aerospike-client-c/target/Linux-x86_64/lib/libaerospike.so)
#include_directories(/usr/include/aerospike)
include_directories(/opt/local/src/git/aerospike/aerospike-client-c/target/Linux-x86_64/include)
set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -g -O0")
#set(CMAKE_BUILD_TYPE Debug)
set(CMAKE_BUILD_TYPE Release)
HHVM_SYSTEMLIB(aerospike-hhvm ext_aerospike.php)
set(CMAKE_INSTALL_PREFIX /opt/local/src/git/aerospike/aerospike-client-hhvm/dist)

[andrewmilkowski]

made successful command run with

$TEST_PHP_EXECUTABLE --php -z /opt/local/src/git/aerospike/aerospike-client-hhvm/src/aerospike/aerospike-hhvm.so aerospike-test.php

(-z pases the extension to HHVM)

so the only thing remaining are failing PHP test cases which is very likely due to extension (.so) not being passed to the interpreter (in the run-tests.php)

this error "Fatal error: Uncaught exception 'ReflectionException' with message 'Class Aerospike does not exist' in :" tells me so

what I am really interested in @rbotzer is "asynchronous" high performance API from within this project

[rbotzer]

I appreciate any pull request you may have. If not we'll go over all your comments before we package up the next release.

With regards to asynchronous, I can refer you to the blog post that I wrote recently: http://www.aerospike.com/blog/jit-down-aerospike-hhvm/ . Right now the plan is to follow a similar path to that of the HHVM team, and first get parity with the PHP API, then extend with Hack's async capabilities.

[andrewmilkowski]

@rbotzer pull request is bit tough because my primary language is not C++

please do review "run-tests.php" and how it resolves the ".so" extension for test, it could be you have environmental variables that "works" and passes test (but not on the clean machine)

besides async capability interest also is in the recent change from libevent (but thats outside of this project scope) and into proxygen, PHP 7 differentials are small between this solution , focus is what aerospike HHVM can provide above and beyond what PHP 7 can already do (async call outs is one, but there could be others)

[rbotzer]

Thanks again for all the information. It's very useful. 👍

PHP 7 compatibility is a short distance from our current PHP client. As you pointed out, it will not be seeing async support, while our HHVM client will take advantage of the async capabilities of HHVM, once we first achieve API parity.

Even before an async capability is added you get the acceleration of your own PHP code through HHVM, and Aerospike as a database provides speed and scalability that is not matched. I am happy to chat with you offline about what the server has to offer.

[andrewmilkowski]

@rbotzer , no problem at all (also will take you upon your offer to chat offline), temporarily adjusted php interpreter in run-tests.php (overrode $php variable and added the extension)

$php = '/opt/local/src/git/facebook/hhvm/dist/bin/hhvm --php -z /opt/local/src/git/aerospike/aerospike-client-hhvm/src/aerospike/aerospike-hhvm.so';

there 3 test failures, (thanks btw for extensive test suite)

Number of tests : 326 312
Tests skipped : 14 ( 4.3%) --------
Tests warned : 0 ( 0.0%) ( 0.0%)
Tests failed : 3 ( 0.9%) ( 1.0%)
Expected fail : 51 ( 15.6%) ( 16.3%)
Tests passed : 258 ( 79.1%) ( 82.7%)

Time taken : 102 seconds

---

there is also a problem starting hhvm with the extension as user "nobody" (will follow with separate ticket)

---

FAILED TEST SUMMARY

Connection - Check instantion with config argument that is not an array [tests/phpt/Connection-test/CheckConfigNotArray.phpt]
Connection - Check instantion with config argument that is not an array [tests/phpt/Connection/CheckConfigNotArray.phpt]
Reconnect - Check for Basic Reconnect after close for a non-persistent connection. [tests/phpt/Reconnect/TestReconnectNonPersistentConnectionPositive.phpt]

---

first 2 errors carry the same error stack trace (very likely related to aerospike version)

---- EXPECTED OUTPUT
ERR_PARAM
---- ACTUAL OUTPUT

[0] __construct() expects parameter 1 to be array, string given

Fatal error: Uncaught exception 'Exception' with message '__construct() expects parameter 1 to be array, string given' in :
Stack trace:
#0 /opt/local/src/git/aerospike/aerospike-client-hhvm/src/aerospike/tests/Connection.inc(56): Aerospike->__construct()
#1 (): Connection->testConfigNotArray()
#2 /opt/local/src/git/aerospike/aerospike-client-hhvm/src/aerospike/tests/astestframework/ASTestFramework.inc(197): ReflectionMethod->invoke()
#3 /opt/local/src/git/aerospike/aerospike-client-hhvm/src/aerospike/tests/astestframework/astest-phpt-loader.inc(28): ASTestFramework->runSingleTest()
#4 /opt/local/src/git/aerospike/aerospike-client-hhvm/src/aerospike/tests/phpt/Connection-test/CheckConfigNotArray.php(3): aerospike_phpt_runtest()
#5 {main}
---- FAILED

---

last error relates to undefined variable

---- EXPECTED OUTPUT
OK
---- ACTUAL OUTPUT

[8] Use of undefined constant AEROSPIKE_CONFIG_NAME2 - assumed 'AEROSPIKE_CONFIG_NAME2'

Fatal error: Uncaught exception 'ErrorException' with message 'Use of undefined constant AEROSPIKE_CONFIG_NAME2 - assumed 'AEROSPIKE_CONFIG_NAME2'' in /opt/local/src/git/aerospike/aerospike-client-hhvm/src/aerospike/tests/Reconnect.inc:77
Stack trace:
#0 /opt/local/src/git/aerospike/aerospike-client-hhvm/src/aerospike/tests/Reconnect.inc(77): astestframework_error_handler()
#1 (): Reconnect->testReconnectNonPersistentConnectionPositive()
#2 /opt/local/src/git/aerospike/aerospike-client-hhvm/src/aerospike/tests/astestframework/ASTestFramework.inc(197): ReflectionMethod->invoke()
#3 /opt/local/src/git/aerospike/aerospike-client-hhvm/src/aerospike/tests/astestframework/astest-phpt-loader.inc(28): ASTestFramework->runSingleTest()
#4 /opt/local/src/git/aerospike/aerospike-client-hhvm/src/aerospike/tests/phpt/Reconnect/TestReconnectNonPersistentConnectionPositive.php(3): aerospike_phpt_runtest()
#5 {main}