apache / couchdb-erlfdb

Erlang API for FoundationDB

An Erlang Binding to FoundationDB

This project is a NIF wrapper for the FoundationDB C API. Documentation on the main API can be found here.

This project also provides a conforming implementation of the Tuple and Directory layers.

Building

Assuming you have installed the FoundationDB C API library, building erlfdb is as simple as:

$ make

Alternatively, adding erlfdb as a rebar dependency should Just Work ®.

Documentation for installing FoundationDB can be found here for macOS or here for Linux.

Quick Example

A simple example showing how to open a database and read and write keys:

Eshell V9.3.3.6  (abort with ^G)
1> Db = erlfdb:open(<<"/usr/local/etc/foundationdb/fdb.cluster">>).
{erlfdb_database,#Ref<0.2859661758.3941466120.85406>}
2> ok = erlfdb:set(Db, <<"foo">>, <<"bar">>).
ok
3> erlfdb:get(Db, <<"foo">>).
<<"bar">>
4> erlfdb:get(Db, <<"bar">>).
not_found

Binding Tester

FoundationDB has a custom binding tester that can be used to test whether changes have broken compatibility. See the BINDING_TESTER documentation for instructions on building and running that system.

couchdb-erlfdb's Issues

Run the bindingtester as part of CI

This would be a good complement to the unit tests. Need to check on some best practices with FDB community regarding which tests to run: should they be deterministic? Should we use a matrix for supported API versions?

How to store Types like Atom using couchdb-erlfdb.

Hi, I'm coming again.
The problem is that I would like to store tuples into FoundationDB.
But it seems it won't work fine if I want to store tuples with atom() in it to FoundaitonDB.
#{context => {invalid_tuple_term,message},exception => error,stacktrace => [{erlfdb_tuple,encode,2,[{file,"er│me has already been fetched lfdb_tuple.erl"},{line,258}]}

Furthermore, If I want to store types other than binary(), the only way seems to be call erlfdb_tuple:pack() ?
Any advice if I want store arbitrary type in foundationdb? That will help a lot.

Fully support FDB API version 630

It seems there are a handful of changes in API version 630 that we ought to be accounting for:

https://apple.github.io/foundationdb/api-version-upgrade-guide.html#api-version-630

Specifically:

• The get_addresses_for_key function now returns strings that include the port in the address. Prior to API version 630, this required using the INCLUDE_PORT_IN_ADDRESS option, which has now been deprecated.
• The ENABLE_SLOW_TASK_PROFILING network option has been replaced by ENABLE_RUN_LOOP_PROFILING and is now deprecated.
• The FDBKeyValue struct’s key and value members have changed type from void* to uint8_t*.

hex.pm release

👋

I wonder if it would be possible to publish erlfdb on hex.pm?

Or maybe it's published already but I'm missing it ...

Add statistics

Linking to the original idea from @nickva in apache/couchdb#3427:

It would be nice have erlfdb emit extra stats such as counts for each types of error codes, how many times transactions retry, number of started transactions and some others.

@davisp already has a handy-dandy branch with some stats kept in C++ using some of the newer atomic features: davisp/couchdb-erlfdb@187b585 so one idea is to expand it.

The other idea is to depend on couch_stats and add some of the couch_stats metrics in erlang. That might be easier but the dependency on couch_stats might be a bit gross.

Tests should guard against setting random keys which write to \xFF

We need to fix a few cases like these:

https://github.com/apache/couchdb-erlfdb/blob/main/test/erlfdb_02_anon_fdbserver_test.erl#L34-L37

As they lead to spurious failures when the key is prefixed with \xFF. I added a gen function which does just that https://github.com/apache/couchdb-erlfdb/blob/main/test/erlfdb_03_transaction_options_test.erl#L100-L102 so we'd just have to re-use the pattern in other tests.

Fail build gracefully if FDB cannot be found

Currently if we can't find fdbcli we just crash rebar with a nomatch error on a regex. We should do better.

Publish code coverage

EUnit can generate surefire reports, but I've also seen some Erlang projects publish metrics using Coveralls. There's an Erlang plugin with detailed instructions on how to use it with GitHub Actions.

random keys in tests can randomly select the special key range

The test suites use of random keys can cause a random failure which look like this because of the special keys range.

I think the exact range may vary between foundationdb versions, but I verified that:

Key = <<255, 255, 97, 98, 99, 97, 98, 99>>, has a 2004 error and Key = <<253, 255, 97, 98, 99, 97, 98, 99>>, works in the current couchdb-erlfdb ci test matrix.

Support newer FDB API versions by dynamically detecting C client library version

We currently hardcode API version 620. Users can select an older version through the api_version .app environment variable, but newer versions are not allowed. The latest binary release is at 630, and the upcoming 7.0 release will be either 700 or 710.

The tricky bit is that if we hardcode a higher api version, erlfdb will fail to build if a user has an older client library installed. We should have a ./configure type of interrogation to see what version of the client library is installed, and dynamically configure FDB_API_VERSION during the build to match.

The CI system should test for 630 compatibility, but I don't think we should bother with unreleased versions at this time.

More discussion on this topic in the FDB forums.

`Close` api is not supported?

Hi, there. I looked through the erlfdb source code.
I'm surprised that close connection api is not supported? If it is true, I'm wondering why do you design like that?

How can I actively disconnect the connection to FDB?

Possible issue with mutex not being destroyed

Firstly, thanks for the bindings. I've been using them in production for quite a while without any serious issues, until now.

As our systems have scaled we noticed a marked increase in 'system' memory growth, until some servers were running out of memory. https://erlangforums.com/t/possible-memory-leak-with-a-nif/978/3

Digging a bit deeper and it looks like it may be an issue in the erlfdb_nif with a mutex not begin destroyed.

Using instrument.allocations().

       erlfdb_nif =>
           #{binary => {72,85,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0},
             driver_mutex =>
                 {2240960,206493,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0},
             driver_tid => {1,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0},
             drv_binary => {107,1717,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0},
             nif_internal => {0,0,0,21,0,0,0,0,0,0,0,0,0,0,0,0,0,0}},

and 20 minutes later

           #{binary => {162,215,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0},
             driver_mutex =>
                 {4634393,423889,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0},
             driver_tid => {1,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0},
             drv_binary => {257,4976,11,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0},
             nif_internal => {0,0,0,89,0,0,0,0,0,0,0,0,0,0,0,0,0,0}},

I may be approaching this incorrectly, but I can replicate the increase in mutex counts just by performing transaction set operations and the mutex count does not ever decrease.

1> I = fun() -> {ok, {_,_,#{erlfdb_nif := N}}} = instrument:allocations(), N end.
#Fun<erl_eval.45.79398840>
2> Gen = fun(Size) when is_integer(Size), Size > 1 -> RandBin = crypto:strong_rand_bytes(Size - 1), <<"asdf", RandBin/binary>> end.
#Fun<erl_eval.44.79398840>
3> Db = erlfdb:open(<<"fdb.local.cluster">>).
{erlfdb_database,#Ref<0.698868889.2384330753.87336>}
4> I().
#{binary => {1,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0},
  driver_mutex => {1,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0},
  driver_tid => {1,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0}}
5> [begin V = Gen(X), K = integer_to_binary(X,32), ok = erlfdb:transactional(Db, fun(Tx) -> erlfdb:set(Tx, K, V) end) end || X <- lists:seq(10,10000)].
[ok,ok,ok,ok,ok,ok,ok,ok,ok,ok,ok,ok,ok,ok,ok,ok,ok,ok,ok,
 ok,ok,ok,ok,ok,ok,ok,ok,ok,ok|...]
6> I().
#{binary => {1,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0},
  driver_mutex => {9989,3,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0},
  driver_tid => {1,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0}}
7> [begin V = Gen(X), K = integer_to_binary(X,32), ok = erlfdb:transactional(Db, fun(Tx) -> erlfdb:set(Tx, K, V) end) end || X <- lists:seq(10,10000)].
[ok,ok,ok,ok,ok,ok,ok,ok,ok,ok,ok,ok,ok,ok,ok,ok,ok,ok,ok,
 ok,ok,ok,ok,ok,ok,ok,ok,ok,ok|...]
8> I().                                                                                                                                                
#{binary => {1,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0},
  driver_mutex => {19980,3,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0},
  driver_tid => {1,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0}}

Using `bytes` options for TLS rather than `path` does not appear to work

FDB allows for:

tls_ca_path tls_ca_bytes
tls_key_path tls_key_bytes
tls_cert_path tls_cert_bytes

When erlfdb inits using the path env values I can connect to the cluster and use the DB.
If I use the bytes options erlfdb 'seems' to connect and returns a database when erlfdb:open(Cluster)., but all further operations just hang. A simple transactional set ends up stuck in erlfdb:wait and never returns.

The only error I see in the fdb trace logs is a 1026 network error which is of course helpfully undocumented.

Rationale:
I currently have a branch which does a lazy init of erlfdb_nif and passes in the network options as they are pulled from a key server on init of the parent application. Currently I'm having to store keys to disk and pass in the path variants, but would definitely prefer that keys do no get saved to disk at all.