In my system I needed to tweak SConstruct in # Environment:

Environment.

env = Environment(CFLAGS=['-Wwrite-strings'],
CCFLAGS=['-Wall', '-Wextra', '-Wno-error=deprecated-declarations', '-Wformat=2',

Basically since there are deprecated functions I needed to add '-Wno-error=deprecated-declarations' to work around this error: readdir_r is deprecated.

When trying to build on Ubuntu 18.04.1 LTS with all dependencies installed I get the following message:

src/base/event_semaphore.cc:93:44: error: dynamic exception specifications are deprecated in C++11 [-Werror=deprecated]
 void TEventSemaphore::Push(uint64_t count) throw(std::system_error) {
                                            ^~~~~
cc1plus: all warnings being treated as errors
scons: *** [out/release/base/event_semaphore.o] Error 1

That does not seem to be dramatic, but as the routine is told to stop if there is any warning the build does not complete.
Do I have to change the configuration or is there any other way to build dory with a compiler using C++ >= V11 ?
I got the version 7.3.0 of the gcc compiler installed on my system.

Regards,
Jelko

Hi There,
I am attempting to compile dory on a CentOS-6.7 system following your directions located at doc/centos_6_5_env.md and build_install.md. I have compiled gcc482-4.8.2-1.el6.x86_64 and have installed our sites python-2.7 RPM's which are in production use. It looks like I'm having issues with scons though.

(dory)[pwright@host-10-10-20-75 dory]$ build --release --import_path dory
scons: Entering directory `/home/pwright/git/dory'
  File "/home/pwright/git/dory/SConstruct", line 135

    prog_libs = {'pthread', 'dl', 'rt'}

                          ^

SyntaxError: invalid syntax

I did verify that I am using python2.7 inside my virtualenv environment - but perhaps I'm missing something else?

(dory)[pwright@host-10-10-20-75 dory]$ python --version
Python 2.7.6
(dory)[pwright

Thanks in advance!

After reading this: https://blog.cloudflare.com/squeezing-the-firehose/
I wonder if zstd is or becomes an option in dory?

Hi,

Looking to implement TLS connection with Kafka using Dory.
Please explain how we can achieve this.

@dspeterson
We have a ipv6 kafka server, the dory_conf.xml file as felllows:

We made a very simple modification to the ConnectToHost function in connect_to_host.cc.
then works right.

void Dory::Util::ConnectToHost(const char *host_name, in_port_t port,
TFd &result_socket) {

int tmp_family;

assert(host_name);
result_socket.Reset();

tmp_family = AF_INET;
if ( strstr(host_name,":") != NULL )
{
tmp_family = AF_INET6;
}

/* Iterate over our potential hosts. /
for (Db::TCursor csr(host_name, nullptr, tmp_family, SOCK_STREAM, 0,
AI_PASSIVE);
csr;
++csr) {
/ Get the address of the host we're going to try and set the port. */
TAddress address = *csr;
address.SetPort(port);

Docker build fails with,

$ docker build -t swestcott/dory:latest .
<snip>
--2016-09-06 11:59:10--  http://googletest.googlecode.com/files/gtest-1.7.0.zip
Resolving googletest.googlecode.com (googletest.googlecode.com)... 64.233.166.82, 2a00:1450:400c:c09::52
Connecting to googletest.googlecode.com (googletest.googlecode.com)|64.233.166.82|:80... connected.
HTTP request sent, awaiting response... 404 Not Found
2016-09-06 11:59:11 ERROR 404: Not Found.

I tried switching to https://github.com/google/googletest/archive/release-1.7.0.zip, but it looks like the build process for gtest has changed also. I hope to look into this further, but if someone else wants to to raise a PR, that'd be grand.

Hi,
when i start Dory it complains like this:

Starting dory: Error during server initialization: [src/dory/dory_server.cc, 106]; You didn't specify allow_large_unix_datagrams, and max_input_msg_size is large enough that clients sending large datagrams will need to increase SO_SNDBUF above the default value.  Either decrease max_input_msg_size or specify allow_large_unix_datagrams.

I don't pass --max_input_msg_size so it should be at the default 64k. However i pass messages to dory using a unix stream socket (--receive_stream_socket_name) on which the --allow_large_unix_datagrams setting has no effect and i don't pass a unix datagram socket (--receive_socket_name). Yet it still asks to set --allow_large_unix_datagrams. And after setting that - it now asks to set --receive_socket_name :)

Starting dory: Error: Option --allow_large_unix_datagrams is only allowed when --receive_socket_name is specified.

So now i have to specify an additional datagram socket even though i'm not going to use it, and also set --allow_large_unix_datagrams. Perhaps during the config parameter check Dory should only ask to specify --allow_large_unix_datagrams if --receive_socket_name is used?

Dory version: 2.0.2 (built from source).

Hey,

I'm using dory to produce messages to kafka 0.8.2.1,

I have two brokers listed in dory_conf.xml, something like this:

    <initialBrokers>
            <broker host="some-broker-host1" port="9092" />
            <broker host="some-broker-host2" port="9092" />
     </initialBrokers>

broker1 (some-broker-host1) had died and dory started listing these in the log:

2017-07-30T11:29:32.371899000Z dory[6]: Metadata request failed for all known brokers, waiting 6762 milliseconds before retry (1)
2017-07-30T11:29:39.137721000Z dory[6]: Router thread getting metadata from broker some-broker-host2 port 9092
2017-07-30T11:29:39.144926000Z dory[6]: Failed to build metadata structure from response: [src/dory/metadata.cc, 106]; Metadata contains partition with unknown broker ID
2017-07-30T11:29:39.145557000Z dory[6]: Router thread did not get valid metadata response from broker
2017-07-30T11:29:39.145998000Z dory[6]: Router thread getting metadata from broker some-broker-host1 port 9092
2017-07-30T11:29:39.147938000Z dory[6]: Router thread failed to connect to broker for metadata

I get the error for some-broker-host1 (it died and therefor unreachable) but why can't dory fetch the metadata from some-broker-host2?

This is the interesting error:

2017-07-30T11:29:39.144926000Z dory[6]: Failed to build metadata structure from response: [src/dory/metadata.cc, 106]; Metadata contains partition with unknown broker ID

When debugging this, I tried to kill some-broker-host2 while host1 is still alive, dory behaved the same (host2 was unreachable and for host1 we got the above error), meaning it needed all brokers to be available in order to fetch metadata properly.

Thanks,
Aviad.

hi, i have installed building env following https://github.com/dspeterson/dory/blob/master/doc/centos_6_5_env.md, then i execute ./pkg rpm to build, but error happened like:

TypeError: 'tuple' object is not callable:
  File "/root/dory/out/pkg/release/rpm/build/BUILD/dory-1.1.2/SConstruct", line 192:
    env.SConscript(src.File('SConscript'), variant_dir=out, duplicate=False)
  File "/usr/local/lib/python2.7/site-packages/scons-2.5.0/SCons/Script/SConscript.py", line 541:
    return _SConscript(self.fs, *files, **subst_kw)
  File "/usr/local/lib/python2.7/site-packages/scons-2.5.0/SCons/Script/SConscript.py", line 250:
    exec _file_ in call_stack[-1].globals
  File "/root/dory/out/pkg/release/rpm/build/BUILD/dory-1.1.2/src/SConscript", line 289:
    }[ext]()
  File "/root/dory/out/pkg/release/rpm/build/BUILD/dory-1.1.2/src/SConscript", line 275:
    env['DEP_SUFFIXES'], True, dep_libs), dep_libs),
  File "/root/dory/out/pkg/release/rpm/build/BUILD/dory-1.1.2/src/SConscript", line 153:
    dep_source_set = impl({main}, set(), dep_libs)
  File "/root/dory/out/pkg/release/rpm/build/BUILD/dory-1.1.2/src/SConscript", line 140:
    for dep_header in source.get_implicit_deps(env, scanner, (src,)):
  File "/usr/local/lib/python2.7/site-packages/scons-2.5.0/SCons/Node/__init__.py", line 941:
    path = path_func(scanner)
Stopping on failure
error: Bad exit status from /var/tmp/rpm-tmp.zbAKlF (%build)

What's wrong with me? Did i missing some parts here?
Thank you.

No flame war between snappy and lz4, just another option available for those who want to use lz4 compression.

When trying to build on Ubuntu 18.04 with gcc 7.3.0, I'm getting the following error:

src/dory/compress/gzip/gzip_codec.cc: In function 'int CheckStatus(int, const z_stream&, const char*)':
src/dory/compress/gzip/gzip_codec.cc:82:21: error: this statement may fall through [-Werror=implicit-fallthrough=]
       ThrowZlibError(strm, zlib_function_name, "Z_STREAM_ERROR");
       ~~~~~~~~~~~~~~^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
src/dory/compress/gzip/gzip_codec.cc:85:5: note: here
     case Z_VERSION_ERROR: {
     ^~~~

It occors at several places in the gzip code.
To get dory building anyway, -Wno-implicit-fallthrough can be added to the CXX compiler flags.

After "Setting Up a Ubuntu Build Environment (16.04 LTS, 15.04 LTS, 14.04.1 LTS, and 13.10)" we expected a Debian based install. But following the link is a Red Hat based install.

To get dory compiling with c++17

#include <functional>

should be added to /dspeterson/dory/blob/master/src/io/chunk_and_pool.h

I think the following lines should be added to the list of packages in /dspeterson/dory/blob/master/doc/ubuntu_env.md :

sudo apt-get install libboost-dev
sudo apt-get install libsnappy-dev
sudo apt-get install libxerces-c-dev

Then Ubuntu 18.04 LTS could be added to the list of compatible versions.

(Except for the error mentioned in #13 , which can be bypassed by adding -Wno-noexcept-type to the CXX Flags)

Does dory support the security mechanisms of kafka?

Like mentioned here: https://kafka.apache.org/documentation/#security

Something like:
SASL/PLAIN - starting at version 0.10.0.0
SASL/SCRAM-SHA-256 and SASL/SCRAM-SHA-512 - starting at version 0.10.2.0
Encryption and Authentication using SSL