ndsev / zswag Goto Github PK

Currently, request MIME types other than application/x-zserio-object lead to a parsing error. The behavior of the parser should be changed, such that other request body MIME types are simply ignored.

Additionally, the zswag client should set the HTTP Accept header, so that remote services know exactly what format to send.

We should build pyzswagcl also for Python 3.10 and 3.11, so that applications can use these more modern Python versions in conjunction with zswag.

The zswag-client should support additional OpenApi features to enable more REST-like interfaces:

• #16 Support PUT/DELETE/PATCH [Released with 0.6.0]
• #18 Support x-zserio-request-part=* [Released with 0.6.0]
• #20 Support parameter schema formats (string/hex/byte/binary)
• #17 Support value array passing schemes
• #19 Support x-zserio-request-part with request field name
• #15 Support path-based argument passing (to support GET)

Comprehensively formulated, this would enable method specs like this:

paths:
  <Path>
    [get|put|post|patch|delete]:
       operationId: <Service-Function>
       parameters:
         - in: [query|path]
           name: <Name>
           x-zserio-request-part: <[*|field-name]>
           style: [matrix|form|simple]
           explode: [true|false]
           schema:
             format: [string|byte|hex|binary]
       requestBody:
         content:
           application/x-zserio-object

Implement extended logging on all steps executed by zswag, including timings.
In case zswag is waiting on some (service) response, zswag should do periodic logging of what it is doing as well.
The behavior should be parameterized (disabled by default).

Reason: currently zswag api acts as a black box. If a service connected to is responding slow or not at all it is unclear why this is happening.
By adding a more verbose mode such issues can be better investigated.

The generic C++ client talks to any zserio service that is running via HTTP/REST, and provides an OpenAPI specification of its interface. When using the C++ OAClient with your zserio schema, make sure that the flag -withTypeInfoCode is passed to the zserio C++ emitter.

It should be ensured that the flag -withReflectionCode is passed? Enable reflection code.

The current Python bindings do not offer docstrings.

Let us deploy the zswagcl and httpcl libraries to conan center. This will simplify using zswag as a dependency, as there is then no need anymore to copy the sub-dependencies into the main project's conanfile.

It would be helpful if zswag would provide a Docker base image with all requirements necessary to run a zserio-swagger-based service.

Current State: Currently, the zswag server doesn't check if a present OpenAPI YAML is usable because of version compatibility.

Desired Behavior: zswag check if a present OpenAPI YAML is usable (for example zswag could put the version that has been used to generate the OpenAPI spec directly in the YAML file) - zswag should output a message if the OpenAPI spec is not compatible.

In order to support integration with other services I would like to host extra endpoints which are not part of the zserio service itself like /health or /metrics.

Options explored:

• Start another web server which only host the extra endpoints: works but not ideal to not hosting the same api
• Write a custom zserio ServiceInterface: this requires the openApi to be written with zserio encoding, even if zserio is not used
• Invoke add_api directly to host another service: this causes a naming conflict in connexion.

Some experimenting revealed that the issue could be solved by making a small change to https://github.com/ndsev/zswag/blob/v1.4.0/libs/zswag/app.py#L178 where an existing routing controller should be kept if it was explicitly defined in the openapi spec.

Open questions:

• Would it be desirable for OAServer to support this usecase?
• If not: What would the recommended approach be?

Currently, the zswag client cannot handle redirects. To handle them, we need to set set_follow_location(true) for the internal httplib client.

1. Start docs with a visualization of the artifacts in a typical project using zswag (zserio sources, generated zserio, zswag-gen and the OpenAPI spec, implementations of client/service). Ideally with a super-short overview of how the workflow steps utilize different parts of zswag.

2. Move the component UML further down - it is informative, but the details are not important for all users and the image looks scary for the uninvolved.

3. Rename "Server Component" section to "Python Server" and add a stub section "C++ Server" noting that it does not exist.

Building target zswagcl-test fails on MacOS as it redefines a typedef already defined by the MacOS SDK

[build] .../zswag/libs/zswagcl/test/src/openapi-parameter-helper.cpp:9:7: error: typedef redefinition with different types ('Parameter::Style' vs 'unsigned char')
[build] using Style = Parameter::Style;
[build]       ^
[build] /Applications/XCode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX13.0.sdk/usr/include/MacTypes.h:577:41: note: previous definition is here
[build] typedef unsigned char                   Style;
[build]                                         ^

The following changes are requested:

• There should be a way to manipulate or replace a HttpLibHttpClient's HttpSettings instance.
• There should be a way to manipulate or initialise HttpSettings programmatically with custom values.

Currently, zswag extensively uses git submodules to fetch its dependencies. This is not a good approach, because it leads to source duplication a source tree where a dependency appears multiple times. Instead, we will use Conan 2.0, which allows flexible declaration of dependencies via a conanfile.txt.

Support decompression on the client in case the server (and OpenAPI spec) offers a supported Content-Encoding
See https://www.iana.org/assignments/http-parameters/http-parameters.xhtml#content-coding for a list of compressions. I would suggest to start with supporting deflate and zstd

The following changes must be documented:

• Featureset of the pyzswagcl Python binding-based client module (OAClient)
• Expanded OpenApi compliance (#12), x-zserio-request-part, application/x-zserio-object Mime Type
• Authentication mechanisms available through httpcl/http-settings.yaml
• How to use the C++ client
• zswag project structure

Currently, only generic HTTP error codes are reported. These should be annotated with some known reasons:

• 302 (redirect to authelia) Forgot to login to NDS?
• 404 Not found - wrong URL of OpenAPI JSON, wrong reference in JSON file?
• Check other error codes and add contextual information on-demand, e.g. from OpenAPI error specifications.

At the moment, https://github.com/ndsev/zswag#url-compound-parameter is not supported.
It would be handy to be able to use compound parts of the request object in a path for example.
For this we would need the possibility to get a URL encoded string of the compound object which we then can map to a certain piece of the path. So if it is only about a 100% match of a compound object (e.g. a header structure) that would work great, no need to flatten things further.
The actual problem that I would like to target is to be able to map a request object which consists out of a scalar ID and a more complex header structure to a path of /{id}/{url_encoded_blob}

My understanding of zswag as now is that I can only url-encode the complete request object so that the ID cannot be mapped individually which kind of affects the backend more than necessary in terms of deployment.

The pyzswagcl dependency is currently only deployed for 3.8. It should also be built for 3.9

For developers working with languages other than Python, it would be helpful if they could use the OpenAPI YAML generation/validation functionality without a Python function call. This could by achieved by exposign these features in a CLI tool

Currently, the zswag client lib is reinstantiating its httplib-client for each request. This may be slow, because it breaks HTTP keepalive optimizations. A new socket and a new TCP/SSL handshake are run for every tile.

In the case of multiple services sharing a similar method name, it is possible that zswag.gen associates the docstring of methodA in one service with the doc string of methodA in another service.

The following formats should be supported

• string: Default string representation of the x-zserio-request-part field (only works for native types).
• hex: Hex representation of the x-zserio-request-part field.
• byte | base64: Base64 representation of the x-zserio-request-part field.
• base64url: Base64url representation of the x-zserio-request-part field.
• binary: Binary representation of the x-zserio-request-part field.

Tasks:

• Client Side
• Server Side

• Client Side
• Server Side

Implement parameters for OAServer to indicate preferences regarding HTTP method and parameter passing style when the spec is auto-generated.

Currently, we are only compatible with 2.3rc1. We will need to make adjustments for the final 2.3.

The pyzswagcl wheel depends on OpenSSL. Therefore, the binaries must be shipped inside of it.

• Client Side
• Server Side

As the OpenAPI generator now supports adding a base yaml with an OpenAPI info section, it would be awesome to get this also in the client code.

// Fetch the OpenAPI configuration using the HTTP client
    auto openApiConfig = fetchOpenAPIConfig(openApiUrl, *httpClient);

I would assume that the above call would give me access to it.

Currently, we have zswagcl::HttpService which only works for zsr-based requests. In order to work for Python too, the request builder part must be extracted into a utility OpenApiClient class, like this:

class OpenApiClient {
    void callMethod(
        string methodName,
        function<vector<buffer>(string)> valueFun, // path->value(s)
        function<vector<string>(string)> enumFun); // path->members
}

From Python side:

• Bind Python class which inherits zserio.ServiceClient and overloads ServiceClient.callMethod()
• Has instance of PyOpenApiClient as member to which calls are forwarded

Tasks:

• OpenApiClient
• ZsrOpenApiClient
• PyOpenApiClient

Every error message generated by the parser should mention the offending value, the allowed values, and the location in the document where the error occurred.

Currently, the zswag client does not interpret the Server's OpenAPI spec. This could be improved, such that the zswag client verifies that the spec conforms to one of the ways the client can pass parameters. Additionally, the client could be moved to a separate package, such that it can be used without pulling the server dependencies.

I tried to build zswag on msys2 but an error occurred.
The details of the error are not output and I am not sure how to deal with it.

Do you have any advice on how to resolve this error?
Also, what development environment (OS, etc.) does the zswag team use?

Development Environment

$ python3 --version
Python 3.9.13

$ pip --version
pip 22.0.4 from C:/msys64/mingw64/lib/python3.9/site-packages/pip (python 3.9)

$ cmake --version
cmake version 3.23.2

$ gcc --version
gcc.exe (Rev2, Built by MSYS2 project) 12.1.0

$ java --version
openjdk 18.0.1.1 2022-04-22
OpenJDK Runtime Environment (build 18.0.1.1+2-6)
OpenJDK 64-Bit Server VM (build 18.0.1.1+2-6, mixed mode, sharing)

$ ant -version
Apache Ant(TM) version 1.10.12 compiled on October 13 2021

Procedure

Install dependent packages related to keychain.

$ pacman -S mingw-w64-x86_64-libsecret
$ pacman -S glib2

$ git clone --recursive https://github.com/ndsev/zswag.git

$ mkdir build
$ cd build
$ cmake ..

Error message

-- Building for: Ninja
-- The C compiler identification is GNU 12.1.0
-- The CXX compiler identification is GNU 12.1.0
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working C compiler: C:/msys64/mingw64/bin/cc.exe - skipped
-- Detecting C compile features
-- Detecting C compile features - done
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Check for working CXX compiler: C:/msys64/mingw64/bin/c++.exe - skipped
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Found OpenSSL: C:/msys64/mingw64/lib/libcrypto.dll.a (found version "1.1.1o")
-- Found Python3: C:/msys64/mingw64/bin/python3.9.exe (found version "3.9.13") found components: Interpreter Development Development.Module Development.Embed
-- Build spdlog: 1.10.0
-- Looking for pthread.h
-- Looking for pthread.h - found
-- Performing Test CMAKE_HAVE_LIBC_PTHREAD
-- Performing Test CMAKE_HAVE_LIBC_PTHREAD - Success
-- Found Threads: TRUE
-- Build type: Release
-- pybind11 v2.6.3 dev1
CMake Warning (dev) at C:/msys64/mingw64/share/cmake/Modules/CMakeDependentOption.cmake:89 (message):
  Policy CMP0127 is not set: cmake_dependent_option() supports full Condition
  Syntax.  Run "cmake --help-policy CMP0127" for policy details.  Use the
  cmake_policy command to set the policy and suppress this warning.
Call Stack (most recent call first):
  deps/pybind11/CMakeLists.txt:98 (cmake_dependent_option)
This warning is for project developers.  Use -Wno-dev to suppress it.

-- Performing Test HAS_FLTO
-- Performing Test HAS_FLTO - Success
=> Running main zserio ant install ...
=> Running zserio cpp extension ant install ...
=> Running zserio doc extension ant install ...
=> Running zserio bundle install ...
CMake Error at deps/zserio-cmake-helper/CMakeLists.txt:62 (message):
  Failed to build zserio.jar in
  C:/msys64/home/70206703/work/zswag/deps/zserio.

It would be nice if a new Zswag project could be instantiated from the command-line, akin to cargo new.

As per the documentation, to have base64 encoding, the format must be specified as "byte," "base64" or "base64url".
By default, zswag.gen does not support setting this format on the command line.

It would be nice to have the format element supported on the command line. Since Zserio is a binary format, it will always need base64 encoding.

Hi, @josephbirkner , I'd like to report a vulnerability issue in pyzswagcl_1.2.1.

Dependency Graph between Python and Shared Libraries

Issue Description

As shown in the above dependency graph (Here shows part of the dependency graph, which depends on vulnerable shared libraries), pyzswagcl_1.2.1 directly or transitively depends on 14 C libraries (.so). However, I noticed that some C libraries are vulnerable, containing the following CVEs:
libcrypto-990425a4.so.1.1 and libssl-337c2b28.so.1.1 from C project openssl(version:1.1.1f) exposed 5 vulnerabilities:
CVE-2021-3711, CVE-2021-3712, CVE-2020-7043, CVE-2020-7042, CVE-2020-7041
libuuid-f64cda11.so.1.3.0from C project util-linux(version:2.27.1) exposed 3 vulnerabilities:
CVE-2018-7738, CVE-2021-37600, CVE-2016-5011

Suggested Vulnerability Patch Versions

openssl has fixed the vulnerabilities in versions >=1.1.1l
util-linux has fixed the vulnerabilities in versions >=2.37.2

Python build tools cannot report vulnerable C libraries, which may induce potential security issues to many downstream Python projects.
As a popular python package (pyzswagcl has 6,985 downloads per month), could you please upgrade the above shared libraries to their patch versions?

Thanks for your help~
Best regards,
Joe Gardner

I tried to build "C++ Client" in REAME but an error occurred.
Please tell me how to deal with this error.

Development Environment

$ cat /etc/lsb-release
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=20.04
DISTRIB_CODENAME=focal
DISTRIB_DESCRIPTION="Ubuntu 20.04.4 LTS"

$ python3 --version
Python 3.8.10

$ pip --version
pip 20.0.2 from /usr/lib/python3/dist-packages/pip (python 3.8)

$ cmake --version
cmake version 3.16.3

$ gcc --version
gcc (Ubuntu 9.4.0-1ubuntu1~20.04.1) 9.4.0

$ java --version​
openjdk 17.0.3 2022-04-19​

​$ ant -version​
Apache Ant(TM) version 1.10.7 compiled on October 24 2019

Procedure

$ cd myapp
$ ls
CMakeLists.txt  api.yaml  client.cpp  services.zs  zswag/

$ mkdir build
$ cd build
$ cmake ..
$ cmake --build .

Error message

...
...
[ 75%] Building CXX object CMakeFiles/myapp.dir/client.o
/home/m-kitahara/work/zswag/example/10_cpp/myapp/client.cpp:7:45: error: ‘MyService’ in namespace ‘myapp_services::services’ does not name a type
    7 | using MyService = myapp_services::services::MyService;
      |                                             ^~~~~~~~~
In file included from /home/m-kitahara/work/zswag/example/10_cpp/myapp/client.cpp:3:
/home/m-kitahara/work/zswag/example/10_cpp/myapp/build/myapp-zserio-cpp.zserio-gen/myapp_services/services/MyService.h:24:11: note: ‘myapp_services::services::MyService’ declared here
   24 | namespace MyService
      |           ^~~~~~~~~
/home/m-kitahara/work/zswag/example/10_cpp/myapp/client.cpp: In function ‘int main(int, char**)’:
/home/m-kitahara/work/zswag/example/10_cpp/myapp/client.cpp:18:45: error: ‘specUrl’ was not declared in this scope
   18 |     auto openApiConfig = fetchOpenAPIConfig(specUrl, *httpClient);
      |                                             ^~~~~~~
/home/m-kitahara/work/zswag/example/10_cpp/myapp/client.cpp:26:28: error: ‘MyService’ has not been declared
   26 |     auto myServiceClient = MyService::Client(openApiClient);
      |                            ^~~~~~~~~
make[2]: *** [CMakeFiles/myapp.dir/build.make:63: CMakeFiles/myapp.dir/client.o] Error 1
make[1]: *** [CMakeFiles/Makefile2:497: CMakeFiles/myapp.dir/all] Error 2
make: *** [Makefile:130: all] Error 2

As per zserio 2.5.0, there is a native zserio C++ reflection interface, so we can transition away from ZSR.

See ndsev/zserio#103

Currently, only the path and query locations are supported. We shall support header as well.

Currently, the MIME type application/x-zserio-object is only visible in OpenAPI.yaml for requests. This leaves some gaps:

• application/x-zserio-object should be generated in OpenAPI.yaml for responses.
• application/x-zserio-object should be the server's response MIME type.

Additionally, a content type should only be set when the body is non-empty. Currently, if the body is empty, and empty-string content type will be put into the request.

As cpp-httplib does now support OpenSSL 3.0, we can move away from 1.1.

The following types of OpenAPI Authentication shall be supported:

• API Key (see https://swagger.io/docs/specification/authentication/api-keys/)
• Basic Auth (see https://swagger.io/docs/specification/authentication/basic-authentication/)
• Cookie Auth (see https://swagger.io/docs/specification/authentication/cookie-authentication/)
• Bearer Auth (see https://swagger.io/docs/specification/authentication/bearer-authentication/)

These mechanisms are already supported by the zswag client, by either using the HTTP_SETTINGS yaml file or passing custom headers to the constructor.

The idea is to check the provided OpenAPI spec (via URL in the constructor) and cross-check with the entries in the HTTP_SETTINGS file. If the settings file contains the right cookie, api-key, etc. then directly use this. If not, provide an error message.

This part has been moved to #60:
Also add the possibility to programmatically add entries to the HTTP config, in cases where the token needs to be retrieved in a separate step before actually calling the service.

I have observed that when the user provides junk input data (wrong payload type, or no payload) that an internal server error code (500) is returned to the user (before userspace code is called).
I think it would be useful to instead report an http 400 error to show that the error is a user error rather than a server error.

How to reproduce:

• Perform a call to an endpoint in the /ui for which a payload request is expected, but provide none / junk text.
• Observe trace in server, and the status code 500 in the ui.

Desired result:

• Perform a call to an endpoint in the /ui for which a payload request is expected.
• HTTP 400 is returned (stack trace on server might still be useful for debugging, but not desirable in production)

• Client Side
• Server Side

We need CI for the following tasks:

• Run tests on zswagcl and httpcl
• Create binary wheel artifacts for pyzswagcl by building the wheel target

Currently, it is not possible to work with the HTTPConfig class directly in code. We need a C++ API to set values on the class instances, and expose this API to Python.