fraunhofer-iis / libjapi Goto Github PK

In GitLab by @Michael-M-Baron on Apr 18, 2023, 09:07

The intended use of the push services is to specify some predefined services and activate them with japi_pushsrv_start() before the libjapi-server is started and deactive them afterwards with japi_pushsrv_stop(). The push services are fixed for the complete runtime and are embedded in the server-software. For every new push-service, the server-software has to be updated with the new services. We found out during the development of the
Interstellar-DAC-GUI, that it could be very usefull to decouple the push-service from the server-development and move it to the client-development, so that the GUI can for example create new push services on the fly during runtime with the desired JAPI-Requests.

We therefore implemented JAPI-Requests which can add and remove push-services during runtime. The code is written in a generic way and can be found in the Interstellar control software dbcd and server_common in the file jr_general.c starting at line 363. The Request for adding a push service works as following:

{ 
  "japi_request": "add_push_service",
  "args": 
  {          
    "SERVICE": "push_temperature",    
    "INTERVAL": 5,           
    "WRAPPER": 
    {                 
      "japi_request": "get_temperature",         
      "args": 
      {                       
        "DEVICE": "device1",    
        "UNIT": "KELVIN"  
      }         
    }    
  }
}

The client specifes the push-Service with the desired JAPI-Request and sets a time-interval in which the request should be repeated. The Push-Service is then started and calls the given JAPI-Request with parameters in an endless loop with a waiting time specified in INTERVAL. More detailled documentation can be found under add_push_service and remove_push_service.

Discussion:

• The accuracy of the Time-Intervall is fixed to full seconds. Maybe this can be enhanced to support milli-seconds for shorter time-intervalls
• Concurrency problems can occur if the JAPI-Requests are working on the same data

In GitLab by @jannismain on Feb 6, 2020, 17:10

Summary

Responses cannot be assigned to requests unambiguously in asynchronous contexts. I suggest to extend the API with an optional japi_request_no key to allow for unambigiuous request-response assignment.

Issue

--> {"japi_request": "get_temperature", "args": {"DEVICE": "A"}}
--> {"japi_request": "get_temperature", "args": {"DEVICE": "B"}}
<-- {"japi_response": "get_temperature", "data": {"temperature": 23.4}}
<-- {"japi_response": "get_temperature", "data": {"temperature": 24.5}}

Question: Which temperature belongs to Device A, which to Device B?

Answer: Currently, it depends on the order of requests. But what if the order of requests is not determined?

What happens if one response is lost and only one response is received:

--> {"japi_request": "get_temperature", "args": {"DEVICE": "A"}}
--> {"japi_request": "get_temperature", "args": {"DEVICE": "B"}}
<-- {"japi_response": "get_temperature", "data": {"temperature": 24.5}}

Question: Which request failed and which succeeded?

Answer: 🤷

Proposal: Add Request Number

To solve this issue, a japi_request_no might be provided in the request and, if given, will be included with the response:

--> {"japi_request": "get_temperature", "args": {"DEVICE": "A"}, "japi_request_no": "2020-02-06T16-59-54-532465Z"}
--> {"japi_request": "get_temperature", "args": {"DEVICE": "B"}, "japi_request_no": "2020-02-06T16-59-54-644578Z"}
<-- {"japi_response": "get_temperature", "data": {"temperature": 23.4}, "japi_request_no": "2020-02-06T16-59-54-644578Z"}
<-- {"japi_response": "get_temperature", "data": {"temperature": 24.5}, "japi_request_no": "2020-02-06T16-59-54-532465Z"}

Pro:

• compatible with existing API (only extends allowed top-level keys)
• every requests can be matched unambiguously
• overhead only added when required, as japi_request_no key is optional

In GitLab by @fraunhofer-iis-anon on Feb 4, 2019, 12:53

https://git01.iis.fhg.de/abt-hfs/libjapi/blob/master/src/japi.c#L264

Der verlinkte Code führt dazu, dass eine neue Anfrage abgelehnt wird, wenn die alte Verbindung noch nicht vollständig geschlossen ist. Siehe Aufzeichnung: rfsc_fail2.pcap

In GitLab by @cstender on Jul 8, 2019, 15:57

It shall be discussed if we better pass a japi_client context instead of raw socket ids to the following functions:

• japi_remove_client
• japi_pushsrv_remove_client
• japi_process_message
• japi_pushsrv_subscribe
• japi_pushsrv_unsubscribe

In GitLab by @bmi on Mar 29, 2023, 11:25

The error occurs when trying to perform a subscription/unsubscription to a service that does not exist

japi_pushsrv_subscribe line 149, this behaviour is allowed and will capture later in the code with an error message but in debug mode the Assertion will brake the program

the same problem exist with unsubscribe

In GitLab by @jannismain on Feb 28, 2020, 12:46

Rationale

A common use-case for multi user applications might be to restrict the ability of certain users.

For example, the first user may get full hardware access, subsequent concurrent users might only be allowed to use safe request, like subscribe to push services and use get-request. Some applications might also define own roles and privileges on a per-request-handler basis. Alternatively, only certain users (authorized via password) or hosts (via hostname/ip) receive full privileges, while others are restricted to safe requests.

For this concept, there are some additions to libjapi required:

• client roles (clients may have different privileges)
• client authentication (new command japi_authenticate)
• client session management (first client might disconnect, now second client should get access to set-requests)

Use Case

abt-hfs/interstellar/gui_fe_dac>

• Multiple web clients are connected to interstellar board controller
  • Web Client A connects to interstellar controller C (libjapi)
  • Then, Web Client B connects to same interstellar controller C
  • A sends request to change hardware configuration -> successful
  • B must be informed about the changes made by A (--> push service)
  • B must be prohibited from making changes to the hardware (set-request) while A is connected
  • B might issue get-requests or subscribe to any push service, as these are safe request

In GitLab by @fraunhofer-iis-anon on Aug 5, 2019, 14:00

In GitLab by @jannismain on Nov 3, 2021, 09:15

This is required to implement the advanced solution outlined in #27

In GitLab by @cstender on Nov 15, 2019, 09:51

"JSON Schema is a draft standard for describing the format of JSON data. The schema itself is also JSON data. By validating a JSON structure with JSON Schema, your code can safely access the DOM without manually checking types, or whether a key exists, etc. It can also ensure that the serialized JSON conform to a specified schema."

Resources:

• http://json-schema.org/
• https://www.w3resource.com/JSON/JSON-Schema.php

Possible implementations to look at:

• https://github.com/netmail-open/wjelement
• https://github.com/helmut-jacob/jsonschema-c
• https://github.com/tristanpenman/valijson
• http://avro.apache.org/docs/1.4.1/api/cpp/html/index.html

In GitLab by @bmi on Mar 29, 2023, 11:32

Die Doxydir folder included examples are out of date and not working.

the libjapi demo is the only example that currently works.

"test.py" in libjapi/doxydir is not the same as in the demo.

In GitLab by @fraunhofer-iis-anon on Mar 29, 2023, 11:35

As a user of the library it would be nicer if the test framework was optional and the generation of cmake and build would work without it.

Solution at an cmake Option to deactivate the google framework.

In GitLab by @Michael-M-Baron on May 5, 2023, 16:06

We are currently facing a problem using the libjapi in Interstellar dbcd. In short, our libjapi-server is fixed to allow only one client at a time. If a client was connected and has network issues, the connection is lost but the client is still open inside the server. This means that no new client can connect to the server anymore and the aplication has to be restarted.

Under normal circumstances, the client will send an EOF while disconnecting. The server recognizes this, closes the socket and removes the client properly. Problems start to occur, when the client disconnects without terminating the connection properly (Due to network problems etc.). The client socket will then still be seen as active inside the server. The server needs to remove the client socket in such cases.

To fix this problem, a mechanism has to be implemented to remove ungracefull disconnected zombie clients inside the server. The server has to somehow verify if the client sockets are still connected properly. There are two ideas how to implement this:

1. The server could frequently try to write dummy packages to the connected client sockets. If the write to the client doesnt work, the server could remove the client. It has to be ensured, that the client can handle these dummy package.
2. A timeout mechanism could be implemented inside the server. If a client doesnt write after certain time interval, the server automatically closes the socket. For this approach, the client has to frequently ping the server to not get disconnected.

Both approaches should be optional and backwards compatible.

This seems to be also a kind of general problem working with servers/clients and TCP/IP. May be there are better solutions?

In GitLab by @bmi on Mar 29, 2023, 11:29

Trying to delete a japi_pushsrv_destroy at runtime is not cleaning the linked list in the ctx->push_service element

This result is a sigseg if you try to read a japi_pushsrv_list after the destroy command becouse the pointer of the push_service is deletet but the linked list entry is still existing.

Solution: Find and delete linked list element in the ctx->push_service pointer.

In GitLab by @Michael-M-Baron on Jan 13, 2021, 10:30

After a server shutdown it can happen that the server socket gets into the TIME_WAIT state and it is no longer possible to use it again. This mechanism is part of the Kernel/TCP-Protocol and prevents possible package losts.

Since this is more of a theoretical problem, the server socket option could be adjusted adjusted with the SO_REUSEADDR flag to enable and ensure an instant reboot of the server after a shutdown.

In GitLab by @cstender on Dec 9, 2020, 18:01

To make use of libjapi in a python program bindings shall be added.

The following page gives a good overview about the possibilites:
https://realpython.com/python-bindings-overview/

In GitLab by @kolb on Jun 17, 2019, 11:02

Boolean type usage is currently dependent on the compiler’s default settings and may generate errors when different compilers are used (for example Clang).

In order to clean this up, multiple steps are necessary:

1. Explicitely set the C standard the compiler should use, at least -std=c99
2. Add #include <stdbool.h> where necessary
3. Replace all variants of boolean types with the standard bool, true and false keywords

For more info about C99-booleans see https://en.cppreference.com/w/c/types/boolean

For a problematic example, see https://git01.iis.fhg.de/ks-ip-lib/software/libjapi/blob/47e7871b/src/japi_pushsrv.c#L175

In GitLab by @cstender on Jul 26, 2019, 12:07

Websocket support shall be implemented directly into libjapi.

List of existing libraries:

https://en.wikipedia.org/wiki/Comparison_of_WebSocket_implementations

In GitLab by @fraunhofer-iis-anon on Aug 27, 2019, 12:02

• returns int 0/-1
• move stdbool.h header from utils.c to utils.h
• call by reference
• adjust unit tests

In GitLab by @fraunhofer-iis-anon on Mar 13, 2020, 12:33

if request with an JSON-keyword or typo in keyword is send, the Client doesn't end.

Example:

cmd_request = {
    'japi_requesTYPO': 'get_temperature',
    'unit': 'kelvin',
    }

In GitLab by @cstender on Aug 21, 2019, 17:22

Add a dedicated gitlab page including packages and documentation for each branch as soon as the following feature is implemented:

https://gitlab.com/gitlab-org/gitlab-ce/issues/35141

In GitLab by @fraunhofer-iis-anon on May 28, 2019, 14:18

In each push service response the source should be verifiable.

In GitLab by @fraunhofer-iis-anon on Jul 8, 2019, 15:42

In GitLab by @Michael-M-Baron on Aug 11, 2020, 15:34

While developing, we discoverd a strange behavior of the libjapi Server. It seems, that the current version of the lipjapi sever has problems processing multiple requests. We send send multiple requests at once, separated by a '\n', and only the first request gets processed.

The detailed Problem is described in Issue.

I found the problem in the libjapi server code and have a solution/bugfix.
The Solution is also described in the ticket above.

In GitLab by @Michael-M-Baron on Sep 13, 2021, 11:35

The problem occurs in the main loop of the japi server while processing the received lines.
After a client is disconnected, the client object gets cleaned and freed with japi_remove_client(ctx,client->socket);
At the end of the loop, the client object is accessed again in the loop condition, but the memory has already been freed, what produces an error in valgrind.

Japi.c: Line 538

In GitLab by @fraunhofer-iis-anon on Mar 2, 2020, 16:51

While reading the code of japi.c, I noticed a small issue.

Code snippet in question:

	free(ctx);

	psc = ctx->push_services;
	while (psc != NULL) {
		psc_next = psc->next;
		pthread_mutex_destroy(&(psc->lock));
		japi_pushsrv_destroy(psc);
		psc = psc_next;
	}

	return 0;

found here.

Apparently, ctx (directly, and indirectly with psc, as it is also a reference to the content of ctx) is referenced after it has been freed. This shouldn't be the case.

It seems like this hasn't caused any issues yet, but fixing it might prevent an annoying search for this bug in the future.

Please forgive me if I am wrong about this, my C knowledge is a bit rusty ;)

In GitLab by @vornkat-iis on Jul 4, 2023, 17:20

In GitLab by @cstender on May 5, 2021, 09:12

The documentation should be generated using a single source.

In GitLab by @cstender on Dec 30, 2021, 11:18

The current implementation requires the user to call the blocking japi_start_server function in order to start the JAPI server. A thread-based non-blocking implementation shall be provided as well.

e.g. japi_start_server_thread

Caution: proper methods to stop the JAPI server thread need to be implemented as well!

In GitLab by @cstender on Mar 23, 2020, 16:36

The list of japi_get_value_as_XXX helper functions in japi_utils shall be extended to int, int64 and double.

In GitLab by @HTE on Sep 24, 2019, 17:11

It would be good if you can configure the libjapi so that just one connection is allowed (like it was in v1). Then we could detect in our GUI if already someone is connected to the board and it cannot happen that two peaple are working on the same board (could lead to errors).

In GitLab by @fraunhofer-iis-anon on Jun 17, 2019, 11:26

In GitLab by @cstender on Jul 12, 2019, 14:26

In GitLab by @fraunhofer-iis-anon on Nov 20, 2018, 10:33

beim updaten des servers wird der base 64 decodede tarball über einen json-string gesendet. Er ist allerdings zu lang für creadline und das update schägt fehl. ich habe manuell jetzt die maximale zeilenlänge erhöht und es geht. ich würde also vorschlagen wir erhöhen die maximale zeichenlänge, bin mir aber noch nicht sicher auf welchen wert. das boot_adc hat nach dem dekoden 9MB. wenn wir also alles auf einmal senden(boot_adc + sw_adc + fw_adc) wird eine relativ lange "nachricht" gesendet.

In GitLab by @jannismain on Jul 8, 2019, 13:26

❯ make run-static
./demo-static 1234
ERROR: JAPI_COMMAND not found!
ERROR: JAPI_COMMAND not found!
ERROR: JAPI_COMMAND not found!

When sending unknown commands to japi-server, the error message should include, which JAPI_COMMAND it didn't understand.

Bonus: It could offer a list of available commands, so one can see, what the problem was (typo, wrong libjapi version, etc)

In GitLab by @jannismain on Feb 26, 2020, 10:54

An Example

--> {"japi_request": "get_temperature", "args": {"DEVICE": "A"}}
<-- {"japi_response": "get_temperature", "data": {"temperature": 24.5}}

Question: Without remembering the request and it's content, how does the client know what temperature value he just received?

Issue

JAPI responses are not (what REST calls self-descriptive messages:

Each message includes enough information to describe how to process the message.

Currently, the duty of providing self-descriptive messages falls to the application. If it doesn't provide self-describing messages, the client has to remember both request and response, bring them together and then reason about the data received with the response.

In Summary: The burden falls either to the client or the server, when the framework could solve this issue for both of them.

Proposal: Include args in response

If libjapi would allow an option, where request arguments are included with the corresponding response, the above problem is solved for both server and client:

--> {"japi_request": "get_temperature", "args": {"DEVICE": "A"}}
<-- {"japi_response": "get_temperature", "args": {"DEVICE": "A"}, "data": {"temperature": 23.4}}

From this response it is clear, that Device A reported a temperature of 23.4 (likely °C), without remembering the actual request sent before.

Pro:

• all information is provided with the response (--> self-describing responses, see below)
• compatible with existing API (only extends allowed top-level keys)

In short, support for self-describing messages (where arguments are included with response) makes server and client development easier while adding minimal complexity to libjapi.

In GitLab by @cstender on Dec 30, 2021, 10:58

Support for RESTful Web APIs shall be evaluated and -if possible- implemented based on the JSON-API specification.

Ressources:

• https://www.restapitutorial.com/

• https://entwickler.de/webentwicklung/restful-api-design-eine-einfuhrung

• https://restfulapi.net/rest-api-design-tutorial-with-example/

• https://www.heise.de/hintergrund/Standardisiert-Die-JSON-API-Spezifikation-4427368.html?seite=all

• https://jsonapi.org/

In GitLab by @fraunhofer-iis-anon on Aug 27, 2019, 16:54

In GitLab by @kolb on Jul 30, 2019, 17:41

libjapi cannot currently be linked in a C++ program because C and C++ compilers generate different symbol names.

To fix this, we have to tell the C++ compiler to use C-style symbol names for all libjapi references and therefore put the following construct into all externally used headers:

#ifdef __cplusplus
extern "C" {
#endif

// Function definitions here

#ifdef __cplusplus
}
#endif

In GitLab by @jannismain on Feb 27, 2020, 17:23

Summary

Provide JAPI command to discover commands registered with the controller.

Explanation

Similar to how push services can be discovered via the japi_pushsrv_list command, discovering registered japi commands via japi_cmd_list would be helpful in client development:

-> {"japi_request": "japi_cmd_list"}
<- {"japi_response": "japi_cmd_list", "data": {"commands": ["get_temperature", ...]}}

Alternatively, a slightly different API would provide even more utility:

-> {"japi_request": "japi_cmd_list"}
<- {"japi_response": "japi_cmd_list", "data": {"get_temperature": {"unit": "celsius"}, ...}}

Here, in addition to the command name (as key of data object), the accepted arguments and their default values are returned.

In GitLab by @jannismain on Nov 3, 2021, 11:15

For a generic GUI client, that works with different libjapi-based applications, some identifier is needed (an application name? id? version?) to distinguish those applications as well as inform the user to which application they are currently connected.

This could be combined with #36, so that the command initially sent after connection was established to verify the connection status would return the application id (or name? or both?).

Flow of events would look like something like this

1. client connects to ws://<host>:<port>
        2. libjapi accepts incoming connection
3. client is connected to libjapi-based application (but which one?)
4. client sends request for identification (e.g. {"japi_request": "japi_id"})
        5. libjapi responds with application id (e.g. {"japi_response": "japi_id", "data": {"id": "libjapi-demo"}})
6. client is connected to libjapi-demo and has verified that everything (application, ws connection) is working properly

The data value of the japi_id request could also be extended to include an application id (id), an application name (name) and a version (version) to allow for more granular identification.

In GitLab by @jannismain on Jul 8, 2019, 16:50

Current Implementation

Right now, the format of JAPI responses is partly specified and partly up to japi_request_handler, which is implemented for each server application:

JAPI command handled by libjapi:

-> { "japi_request": "japi_pushsrv_list"}
<- { "japi_response": "japi_pushsrv_list", "services": [ "push_temperature", "push_counter" ] }

JAPI command handled by application-defined response handler (but still including japi_response key):

-> { "japi_request": "get_temperature" }
<- { "japi_response": "get_temperature", "temperature": 27.0, "unit": "celsius" }

Another command handled by application-defined response handler (without japi_response key):

-> { "japi_request": "japi_pushsrv_subscribe", "service": "push_temperature" }
<- { "temperature": 39.91664810452468, "japi_pushsrv": "push_temperature" }
<- { "temperature": 39.73847630878195, "japi_pushsrv": "push_temperature" }
<- { "temperature": 39.46300087687415, "japi_pushsrv": "push_temperature" }
<- { "temperature": 39.09297426825681, "japi_pushsrv": "push_temperature" }
...

Problems

A badly (or maliciously) designed request handler could use JAPI package identifiers listed above to trick the parser into doing things he shouldn't be doing.

Example of a forged response:

-> { "japi_request": "japi_pushsrv_list", "services": [ "push_temperature", "push_counter" ] }
-> { "japi_request": "japi_pushsrv_subscribe", "service": "push_temperature" }
<- { "japi_response": "japi_pushsrv_subscribe", "service": "push_temperature", "success": true }
<- { "temperature": 39.91664810452468, "japi_pushsrv": "push_temperature" }
<- { "temperature": 39.73847630878195, "japi_pushsrv": "push_temperature" }
// forged response to advertise service that doesn't exist:
<- { "japi_response": "japi_pushsrv_list", "services": [ "push_temperature", "push_counter", "push_bitcoin_miner" ] }
<- { "temperature": 39.09297426825681, "japi_pushsrv": "push_temperature" }
...

If the client-side parser has no logic to match responses to initiated requests, it will now think, there is a registered push_bitcoin_miner service, which there isn't really.

But even if the client-parser only handles responses with matching requests, a push service package (with a forged response) might always arrive in between request and authentic response.

Possible Solutions

1. Blacklist internal JAPI keys (japi_response, japi_response_msg, japi_pushsrv, service, services, success) from being used in request handler functions.
2. Strictly specify first level of JAPI response messages (request handler values would be attached on the second level )

Proposed Solution

Specify the first level of all JAPI responses to guarantee, that the same parser can parse all JAPI response packages.

A specification could look like this:

Regular JAPI packages:

{
    "japi_response": str,
    "value": any,
    "success": bool
}

JAPI Push Service Update Packages:

{
    "japi_pushsrv": str,
    "value": any
}

Here, the application defined fields (any) are separated from any internal JAPI fields ("value", "japi_pushsrv") through hierarchy, rendering the forged response above meaningless:

Example of a forged response:

...
<- { "value": 39.91664810452468, "japi_pushsrv": "push_temperature" }
<- { "value": 39.73847630878195, "japi_pushsrv": "push_temperature" }
// forged response to advertise service that doesn't exist:
<- { "value": { "japi_response": "japi_pushsrv_list", "services": [ "push_temperature", "push_counter", "push_bitcoin_miner" ] }, "japi_pushsrv": "push_temperature" }
// client is indifferent to this attack:
-> 凸(￣ヘ￣)
<- { "temperature": 39.09297426825681, "japi_pushsrv": "push_temperature" }
...

In GitLab by @kolb on Nov 25, 2019, 10:52

I run a JAPI server as part of a larger program where it is not started from the main() function, but from another thread.

When the program shuts down, I'd like to cleanly shut down the JAPI server as well. That is currently not possible because japi_start_server() is blocking and cannot be interacted with from the outside.

Example code for explanation of the issue:

// called by larger application's framework
void start(void)
{
   d_japi_thread = start_thread(japi_thread); // starts japi_thread() as separate thread
}

// called by larger application's framework
void stop(void)
{
  // Shut down the thread -> how? japi_shutdown() would be nice
  d_japi_thread->join();         // join() blocks forever :-(
}

void japi_thread(void)
{
  japi_context *ctx = japi_init(this);

  // japi setup code left out for brevity

  int ret = japi_start_server(ctx, this->port); // blocking call

  // this part is never reached except if an error occurs

  japi_pushsrv_stop(pushctx_stats);
  japi_destroy(ctx);
}

As pointed out above, a japi_shutdown(ctx) function would be nice that causes japi_start_server() to return. japi_shutdown() would have to be thread-safe.

In GitLab by @cstender on May 24, 2019, 15:42

In GitLab by @cstender on May 24, 2019, 15:42

In GitLab by @jannismain on Jul 8, 2019, 13:28

❯ make clean
rm -rf obj lib testobj doc testsuite
cd googletest/googletest/make && make clean
/bin/sh: line 0: cd: googletest/googletest/make: No such file or directory
make: *** [clean] Error 1

if googletest dir is not available, it should be skipped and make clean should continue with remaining dirs.

In GitLab by @jannismain on May 19, 2021, 15:33

Rationale

A ping-like command is required to test, whether

• the connection has been established successfully (right after it has been established)
• the server is ready to handle requests (also after longer periods of inactivity)
• the connection is compromised in any other way not reflected by the TCP connection status (e.g. server busy)

This could also be used to check, whether the server is still alive, in cases where a server status is reported.

Story

• As a libjapi client
• I want to check whether the connection to the server is working
• In order to inform the user, when the connection might not be working properly

Example

The messages could look like this:

Request:

{
    "japi_request": "japi_ping"
}

Response:

{
    "japi_response": "japi_ping",
    "data": {
        "success": true
    }
}

Inspiration

This addition was inspired by Interstellar's test_connection command

In GitLab by @cstender on Jul 12, 2019, 14:26

In GitLab by @bmi on Mar 29, 2023, 11:39

Include files with "" or <> are not trivial

for local files use always ""
for not local files(like librarys) use <>

Problem occurs when if the include path not set as expected by user of the library
using the correct include pattern solves the problem

In GitLab by @cstender on May 24, 2019, 15:13

Each pushsrv message shall include the pushsrv name.

"japi_pushsrv" : "xyz"

In GitLab by @vornkat-iis on Apr 27, 2023, 10:33