zeromq / zproto Goto Github PK

I would like to point out that the use of double underscores in specific identifiers does not fit to the expected naming convention of the C++ language standard.
Would you like to adjust your selection for unique names?

If the connection fails for some reason, then the actor is destroyed. If there isn't an actor, then the client is destroyed too. By destroying the client, the reason and status information is cleaned up. This results in the calling code not being able to handle a creation failure intelligently, nor report an intelligent error to a user.

An example from malamute is shown below, see lines 13-14.

01  MLM_EXPORT mlm_client_t *
02  mlm_client_new (const char *endpoint, uint32_t timeout, const char *address)
03  {
04      mlm_client_t *self = (mlm_client_t *) zmalloc (sizeof (mlm_client_t));
05      if (self) {
06          zsock_t *backend;
07          self->msgpipe = zsys_create_pipe (&backend);
08          self->actor = zactor_new (mlm_client, backend);
09          if (self->actor)
10              self->status = mlm_client_constructor (self, endpoint, timeout, address);
11          if (self->status == -1)
12              zactor_destroy (&self->actor);
13          if (!self->actor)
14              mlm_client_destroy (&self);
15      }
16      return self;
17  }

I like to use the c_codec over zyre but therefore I need access to the internal msg_t which can be handed to zyre. To solve this issue I introduced the encode/decode methods in c_codec v1 but those are really ugly.

Any suggestions?

Hi:
You may consider changing lines 22-23 and 30 this way to make it work on mac as well as linux.
Lines 22-23

PROJECT_DIR="$(dirname $(dirname $(readlink -m $0)))"
MY_PROJ_NAME=$(basename "$1")

to

SCRIPTSDIR=$(dirname $(cd ${0%/*} 2>>/dev/null ; echo `pwd`/${0##*/}))
PROJECT_DIR="${SCRIPTSDIR}/.."
MY_PROJ_NAME=$(basename "$1")
MY_PROJ_NAME_CAPS=`echo $MY_PROJ_NAME | tr '[a-z]' '[A-Z]'`

Line 30:

find ./ -type f -exec sed -i -e 's/MYPROJ/'${MY_PROJ_NAME^^}'/g' {} \; || exit 6

to

find ./ -type f -exec sed -i -e 's/MYPROJ/'${MY_PROJ_NAME_CAPS}'/g' {} \; || exit 6

Regards
SRC

So this compiles fine:

<header>
  <field name = "req_id" type = "string" size = "2">An UUID to uniquely identify each message.</field>
</header>

But this doesn't:

<header>
  <field name = "req_id" type = "string" size = "2">An UUID to uniquely identify                                                     
  each message.</field>
</header> 

This problem is due to fields' comments being started by // and therefor expected to be one line.
I believe it would be better to comment with /* ... */ to avoid this.

I'll make a PR changing this, unless there is reason not to.

This issue is to discuss to structure of this project and how it may be installed.

Project tree structure

I think the zcodec project tree structure could look like this.

/model
/model/generate
/model/sample.xml

/scripts
/scripts/codec_c.gsl
/scripts/library.gsl

Within the model folder is a generate script and a sample.xml which shows how to write a protocol in xml.
Within the scripts folder are all codec scripts and subscripts.

Possible implementation of /model/generate

export PATH = ../zcodec/scripts:$PATH
gsl -q -script:codec_c sample.xml

Setup and installation

Setup up might be done by adding zcodec as git submodule and coping it to the parent project. The user gets a sample.xml to edit and can easily get results because the generate script already points to the scripts folder.

- Go to your project folder

git submodule add <link to zcoded repository>
cp -R zcodec/model .

- Edit the sample.xml with your protocol specifications.
- Run generate

It might be useful if server's engine generated with zproto_server_c.gsl could call a user defined function when one or more configuration parameter changes. Now it calls only functions internal to engine.

$(class.name)_t * $(class.name) = $(class.name)_decode (&msg);
if (zsocket_type (zsock_resolve (input)) == ZMQ_ROUTER)
$(class.name)->routing_id = routing_id;

decode may return null, therefore part of the condition should be $(class.name)_t doesn't equal null

The Java code generator produces an invalid switch statement when using local variables. The scope of variables in each case clause correspond to the whole switch statement. However zproto generates local variables within each case clause, which are not wrapped within a locally scoped block. This results in a multiple declaration compilation error.

Example of generated code.

        case INSERT_OK:
            ByteBuffer bbUuid = ByteBuffer.wrap(self.getBlock(16));
            self.uuid = new UUID(bbUuid.getLong(), bbUuid.getLong());
            break;

        case INSERT_FAILED:
            self.error_code = self.getNumber2 ();
            self.error_reason = self.getString ();
            break;

        case UPDATE:
            ByteBuffer bbUuid = ByteBuffer.wrap(self.getBlock(16));
            self.uuid = new UUID(bbUuid.getLong(), bbUuid.getLong());
            self.content = self.getLongString ();
            break;

Each case block should use a local scope as follows:

        case INSERT_OK:
            {
                ByteBuffer bbUuid = ByteBuffer.wrap(self.getBlock(16));
                self.uuid = new UUID(bbUuid.getLong(), bbUuid.getLong());
            }
            break;

        case INSERT_FAILED:
            self.error_code = self.getNumber2 ();
            self.error_reason = self.getString ();
            break;

        case UPDATE:
            {
                ByteBuffer bbUuid = ByteBuffer.wrap(self.getBlock(16));
                self.uuid = new UUID(bbUuid.getLong(), bbUuid.getLong());
                self.content = self.getLongString ();
            }
            break;

https://github.com/zeromq/zproto/blob/master/src/zproto_server_c.gsl#L990

Should be able to handle this gracefully

Source: http://www.gnu.org/software/automake/manual/html_node/Extending.html

[...]
With the -local targets, there is no particular guarantee of execution order; typically, they are run early, but with parallel make, there is no way to be sure of that. 
[...]

There are hooks which are guaranteed to be executed after, but we need quite the opposite :-|

Digging through the documentation the only way to ensure order are SUBDIRS and DIST_SUBDIRS in the toplevel Makefile.

Solution: probably revert #17

Solution: do not install libzproto nor zproto.h, nor man pages.

These files are not needed at the system level; they serve only as test / templates.

Hi,

I've been trying to follow the example shown in the README.md but I'm still getting the same error:

> gsl hello_server.xml
2015/09/22 12:21:09: GSL/4.1a/github Copyright (c) 1996-2011 iMatix Corporation
2015/09/22 12:21:09: gsl/4 I: Processing hello_server.xml...
2015/09/22 12:21:09: (zproto_server_c.gsl 12) Undefined expression: xml.load_file (...)

The content of hello_server.xml is the following:

<class
    name = "hello_server"
    title = "Hello server"
    script = "zproto_server_c"
    protocol_class = "hello_msg"
    package_dir = "../include"
    source_dir = "."
    project_header = "hello.h"
    >
    A Hello, World server

    <state name = "start">
        <event name = "HELLO">
            <action name = "send" message = "WORLD" />
        </event>
    </state>
</class>

After reading the line 12 of zproto_server_c.gsl, I understood that I'm also supposed to create a file named hello_msg.xml but I doesn't seems to solve the problem since the XML object itself is not known in the hello_server.xml.

Did I installed the wrong GSL version ? (I took it from the master branch of its repository at the same time as I took zproto from this repository / master branch).

Am I missing something obvious ?

Thanks in advance.

If it generated this, we'd get bindings automatically for the zproto client layer.

Solution: generate an API model for the client API?

Solution: make a script that does the work.

• copy the skeleton structure to another location
• rename files that refer to myproj to xxx (where xxx is the new project prefix)
• do same inside all files

When sending to a ROUTER socket via send_(...) instead of send() the routing id assert in the send method will stop the program.

This works, because we can set a routing id

zproto_example_t *self = zproto_example_new (ZPROTO_EXAMPLE_LOG);

zsock_t *socket = zsock_new (ZMQ_ROUTER);
zsock_connect (socket, "inproc://selftest");

zproto_example_set_routing_id (self, <some_id>);
zproto_example_set_sequence (self, 123);
zproto_example_set_level (self, 123);
zproto_example_set_event (self, 123);
zproto_example_set_node (self, 123);
zproto_example_set_peer (self, 123);
zproto_example_set_time (self, 123);
zproto_example_set_host (self, "Life is short but Now lasts for ever");
zproto_example_set_data (self, "Life is short but Now lasts for ever");
zproto_example_send (socket, output);

This fails, because we can't set a routing id

zsock_t *socket = zsock_new (ZMQ_ROUTER);
zsock_connect (socket, "inproc://selftest");
zproto_example_send_log (socket, 123, 123, 123, 123, 123, 123, "Life is ...", "Life is ...");

When thinking of the simplest possible solution all that comes to my mind is making zsock more intelligent. In that scenario zsock would hold the routing id from the last read message.

A BNF like the following cannot be realized with the current mechanisms:

directory = path *file
path = string
file = path size timestamp
size = number-8
timestamp = number-8

I would suggest to add 'rules' like e.g.

<rule name = "myfile">
  <field name = "path" type = "string" />
  <field name = "size" type = "number" size = "8" />
  <field name = "timestamp" type = "number" size = "8" />
</rule>

<message name = "directory">
  <field name = "path" type = "string" />
  <field name = "files type = "myfile" multiple= "true" />
</message>

And add a struct for each rule in the c code in the same file. With methods

• _new()
• _destroy()
• _first()/_next()
• getter/setter

What do you think? Especially about the c part I have doubts.

Solution: add "repeat = "n"" to fields.

Difficulty: how to store these in class. Serialization is easy (2-octet count + fields). For API, add "index" argument when field has repeat attribute.

I'm trying to get my head around this zproto. I'm playing with just the skeleton project. When I run gsl with myproj_msg.xml I get an error that 'dictionary' is an unknown type. Can anyone help me with what I am doing wrong?

Solution: generate a grammar text from the protocol specification.

This is pretty simple and very helpful for RFC writing.

We have a few different works that would be good together as a new package, the skeleton project in zproto, and the project makefile generator from CZMQ.

Solution: start a new zproject project and move these pieces there.

Please correct me if I'm wrong.

Let's assume I have the following event with two actions. If the client sends the HELLO event the prepare_greeting method is invoked and afterwards the server sends a WORLD reply.

<state name = "start">
    <event name = "HELLO">
        <action name = "prepare greeting" />
        <action name = "send" message = "WORLD" />
    </event>
</state>

So far, so good. If something went wrong in the prepare greeting I'm able to interrupt the action flow by calling engine_set_exception and tell it to execute my exception event which sends an ERROR reply instead.

<event name = "exception">
    <action name = "send" message = "ERROR" />
</event>

Now, regarding my issue. I would like to choose an alternative action flow based on the outcome of a previous action. Currently, I'll have to create a new event with an action and call it by engine_set_exception which seems to me tedious and somewhat wrong as this might not be an exception. It would be nice to have something like:

<state name = "start">
    <event name = "HELLO">
        <action name = "prepare greeting">
            <action onReturn = "0" name "send" message = "WORLD" />
            <action onReturn = "1" name "send" message = "EUROPE" />
            <action onReturn = "2" name "send" message = "ASIA" />
        </action>
        <action name = "do something else" />
    </event>
</state>

Based on the return code of prepare_greeting a different action can be taken.

Currently it's only possible to store c strings in a list, but got a need to store a list of iovec like data, e.g. where you know the starting pointer and a length.

Problem: server code needs logging hook

Solution: engine_log () function in generated engine

Currently, the codec and server code seems to be written with the assumption of only a single frame of routing in mind.
https://github.com/zeromq/zproto/blob/master/src/zproto_codec_c.gsl#L874
https://github.com/zeromq/zproto/blob/master/src/zproto_codec_c.gsl#L317
https://github.com/zeromq/zproto/blob/master/src/zproto_server_c.gsl#L257

This means that clients must be connected directly to the server, and can't be behind any multiplexing proxy layers. Is this an acceptable limitation, or do we need to revise our strategy to account for this usage?

If we do not accept this limitation, I can see two broad categories of approach for solutions:

1. rewrite the server and codec structures to store the routing info as a zlist of zframes instead of a single zframe.
2. rewrite the recv end of the codec to serialize multiple zframes of routing into a single zframe for storing, rewrite the send end of the codec to deserialize the single zframe into multiple zframes to use as the routing for the outgoing message.

If there is another approach that is more appropriate, let's discuss it!

I'm attempting to use zproto to build a protocol on top of zyre. Very similar to zbroker's setup. I've noticed that the code generated by zproto for zbroker's zpipes_msg class seems to include encode/decode methods that translates their "zpipes_msg" to and from "zmsg" which can be easily pushed into zyre's interface for zyre_shout, zyre_whisper.

It seems that these methods are generated in zproto_codec_c_v1 but that codec creates messages that are not compatible with servers generated with zproto_codec_c, which my server was generated with.

I'm reaching out with the hope that there is some support for generating these methods or approximating this functionality already in zproto that I'm just missing. Any thoughts or suggestions in this regard would be greatly appreciated.

Failing that, and if the functionality seems globally useful, I'm willing to offer to contribute to zproto by adding this functionality back in to zproto_codec_c.

Thanks

https://github.com/zeromq/zproto/blob/master/src/zproto_codec_c.gsl line and 1067 and 1002.

The UUID is fix size so even if the field is null the needle should advance and maybe null bytes should be written instead (the frame_size include the UUID size even the field is null).

At the moment, make check runs zero tests, i belive it has been like that since the beginning, is that intended?

Solution: add encoding/decoding for longer strings

We may also encode string lists and dictionaries using longer strings; current encoding is limited to 255 entries of short strings.

I am getting compile errors from the hello_client.xml example:

gsl -trace:1 hello_client.xml
GSL/4.1c Copyright (c) 1996-2016 iMatix Corporation
gsl/4 I: Processing hello_client.xml...
gsl/4 M: Generating stub for connect_to_server...
cc -c -o hello_client.o hello_client.c
In file included from hello_client.c:39:0:
hello_client_engine.inc:626:1: error: conflicting types for ‘hello_client_set_verbose’
hello_client_set_verbose (hello_client_t *self, bool verbose)
^
In file included from hello_client.c:18:0:
./hello_client.h:62:5: note: previous declaration of ‘hello_client_set_verbose’ was here
hello_client_set_verbose (bool verbose);
^
hello_client.c: In function ‘hello_client_test’:
hello_client.c:71:5: error: ‘self’ undeclared (first use in this function)
self->verbose = verbose;
^

Looking into zproto_client_c I see the signatures for *_set_verbose do mismatch:
$ grep _set_verbose /usr/local/bin/zproto_client_c.gsl
$(class.name)_set_verbose (bool verbose);
$(class.name)_set_verbose ($(class.name)_t *self, bool verbose)

and "self" is undeclared within $(class.name)_test (bool verbose)

I have the most recent code from github. Did I use zproto_client_c incorrectly?

thanks,
Jeff

After running:

./autogen.sh
./configure
make check

The following output is received, indicating a failure in the self test:

Making check in src
/Applications/Xcode.app/Contents/Developer/usr/bin/make  check-TESTS
../config/test-driver: line 107: 72776 Segmentation fault: 11  "$@" > $log_file 2>&1
FAIL: zproto_selftest
/Applications/Xcode.app/Contents/Developer/usr/bin/make  all-am
make[5]: Nothing to be done for `all-am'.
============================================================================
Testsuite summary for zproto 0.0.1
============================================================================
# TOTAL: 1
# PASS:  0
# SKIP:  0
# XFAIL: 0
# FAIL:  1
# XPASS: 0
# ERROR: 0
============================================================================
See src/test-suite.log
Please report to [email protected]
============================================================================
make[3]: *** [test-suite.log] Error 1
make[2]: *** [check-TESTS] Error 2
make[1]: *** [check-am] Error 2
make: *** [check-recursive] Error 1

src/test-suite.log contains:

======================================
   zproto 0.0.1: src/test-suite.log
======================================

# TOTAL: 1
# PASS:  0
# SKIP:  0
# XFAIL: 0
# FAIL:  1
# XPASS: 0
# ERROR: 0

.. contents:: :depth: 2

FAIL: zproto_selftest
=====================

Solution: allow in the XML, then resolve unknown types and denormalizing at the start.

OS X wants this to be:

tcp://*:5670

If you use the word localhost in the bind() call it will fail with zmq_strerror(errno) returning:

Operation not supported by device

As the self test is not regenerated, its an easy fix for the code, but perhaps this should be modified in the GSL?

Solution: install zproto_c.gsl and library.gsl into /usr/local/bin during 'make install'

The README.md says:

If a protocol message is not valid in the current state (i.e. there is no matching event), then it is queued for later processing.

I would expect this to be in in s_server_handle_protocol() in zproto_server_c.gsl but in this function the message is received from the socket, pushed to the state machine and then thrown away.

Solution: Add methods to encode/decode from/to zmsg_t

zmsg_t *
    $(class.name)_encode ($(class.name)_t * self);
$(class.name)_t *
    $(class.name)_decode (zmsg_t *msg);

An example where these are useful is zyre. Here the user is not in control of sending an receiving.

I got this implemented just fine, but do ignore the routing id. In case of zyre this is no problem as zyre takes care of that.
In my opinion that's okay, let the one in control take care of it. Otherwise we could add a socket_type attribute to zmsg.

What are your thoughts on that?

I currently have a need to encode/decode messages outside of the scope of codec_recv/send, in my case over zyre_whisper/shout messages.

Would be cool if that would be possible.

I have problem with zproto. Suppose we have the sample definition of the protocol :

<state name = "connected" inherit = "defaults">
  <event name = "send request">
    <action name = "send" message = "REQUEST" />
  </event>
</state>
<!-- API methods -->
<method name = "send request" return = "status">
  <field name = "params" type = "string" />
  <accept reply = "SUCCESS" />
  <accept reply = "FAILURE" />
</method>

Basically I have event "send request" connected with API function.
This event/api method is valid only in "connected" state.
But what happens if I call this API function in other state?
For example I want to call “send_request” method but before this server is down and now I'm in disconnected state.

In this case "send request" never come back!
Why? Because it sends event invalid in current state and wait for reply in
s_accept_reply function. Reply never come back ...

In s_accept_reply we have:

       char *reply = zstr_recv (self->actor);
        if (!reply)
            break;              //  Interrupted or timed-out

and default timeout for actor is set to infinity.

Of course I can:

1. set timeout for actor socket. This is solution but not ideal (why do I have to wait 15 seconds for function return in situation that this is impossible to return with success?)

2. I can change protocol definition and handle "send request" event in all states. But is very hard to maintenance and not elegant solution...
   I think that this situation should be handled better in code generator.

On decode and encode methods there is comment saying that if the socket type is router than the first frame will be decoded/encoded as routeing id, there is no way for those methods to know, and actually it never checks and never decode/encode the routering id.

Suggest solutions:

1. fix the comment and maybe even add that on router socket the user should send or receive the routing id by himself or use the send/receive methods instead.
2. pass a boolean to indicate if the first frame is routing id.

The repeated fields are particularly nasty, costing a lot of repeated code. @sappo, you're the main user for this. I was thinking that the codecs should be two layer, one for a thin binary framing, that delivers contents which can be parsed by a second layer -- JSON for instance. That would remove any need for structures and arrays in the protocols.

Would you be willing to use this approach so we can simplify the codecs?

The ZeroMQ team "zproto maintainers" actually maintains ztools :)

Solution: add 'make code' step and remove dependency on 'generate' script

Can we have script, probably gsl script to generate another zxxx_msg in case user need 2 or more codecs?

If we will have such script. User can use zproto to generate simple generic zXXX project without zXXX_msg and later on one can append codec when needed.

The server generator places some of the generated files in unusual places. Let's assume I want to generate the "myserver" engine: The myserver_engine.h and the myserver.c go to the same folder as the model xml while the myserver.h goes to ../include/myserver.h. I want to place the model in a separate folder, as the CLASS rfc suggests, which causes trouble. Of course I can create a custom shellscript, that moves the files afterwards, but why not follow CLASS in the first place.

I have a state machine that has many different events that can cause it to enter or leave the active state, but I have an action or actions that I need to run each time the state transitions to or from the active state, before processing any events in the new state.

Currently, there is no clean, non-redundant way to do this. If I specify my actions in all of the events that lead to or from the active state, I end up with duplicated lines all over my model. There are a few other things I've tried, including adding "transition" events, but they all make the model more complicated than it needs to be and harder to maintain because I have to make sure that every time I add an entry or exit path I also add my transition actions.

Proposed solution: implement handling of the following kind of "transition" models in the engine:

<state name = "active" inherit = "defaults">
    <transition type = "enter">
        <action name = "activate foo engine" />
        <action name = "activate bar engine" />
    </transition>
    <event name = "FOOMESSAGE">
        <action name = "process foo message" />
        <action name = "prepare foo response" />
        <action name = "send" message = "FOOBACK" />
    </event>
    <event name = "BARMESSAGE">
        <action name = "process bar message" />
        <action name = "prepare bar response" />
        <action name = "send" message = "BARBACK" />
    </event>
    <transition type = "exit">
        <action name = "shutdown foo engine" />
        <action name = "shutdown bar engine" />
    </transition>
</state>

I think this is the clearest way to show intent when you want actions to run for any entry or exit point of the active state, and would be simple to add to the generated engines.

Does anyone have any objections, preferences, or better ideas?

I want to use ZProto codec with PUB and only use one frame. Problem is that the topic must be at the beginning of the message.

So we can create a new field in the codec, called topic, it will be reserved like the routing_id and will be populate when used with SUB or XSUB and will be sent when using with XPUB or PUB.

One more issue is that we have to make official delimiter that cannot be used in the topic (or escape it), we cannot start a topic with a number of bytes because it cannot be prefixed matched.

Thoughts?

https://github.com/zeromq/zproto/blob/master/src/zproto_codec_c.gsl#L774-L787 (correct) vs https://github.com/zeromq/zproto/blob/master/src/zproto_codec_c.gsl#L1064-L1076 (incorrect)

Missing a call to zlist_destroy before zlist_new. Might be better to treat receiving from a socket as a special case of deserializing instead of having two nearly identical functions?