educloudalliance / xroad-rest-gateway Goto Github PK

Hi,
I am trying to understand how to the rest adapter works. I have edited consumer-gateway.properties to match my security server and sample subsystem "SV.GOB.1001.testquery". This works as expected and I am also able to pass different services to which my subsystem has been granted access. Now, I would like to use a different subsystem, I don't see a way to add multiple client ids inside consumer-gateway.properties. I've tried commenting out the id.client value inside consumer.properties and passing de X-XRd-Client header, but it gets ignored:

11.04.2017 16:40:28 WARN ConfigurationHelper : Client can not be null.
11.04.2017 16:40:28 WARN ConsumerGatewayUtil : ConsumerMember not found.

Am I missing something or this functionality is not implemented yet?

Thanks in advance for your help.

In the discussions online and in emails there has been mentions about some Developer Day going to happen in Feb 2015.

The X-Road and EduCloud Developer Day is still under planning, but something is already known.
It will take place in Tallinn. The date is one of the following: 10.2., 11.2. or 12.2.2015. Main responsible party for the event is Estonian EduCloud builders.

The planning for the event happens online in Google Drive. There will be 5 tracks/groups that work for a day:

Track 1: Identification and role management (Lead:XX)
Track 2: Metadata application profiles (Lead: Mart Laanpere)
Track 3: X-Road Integrations and API management (Lead: Andres Kütt)
Track 4: Bi-lateral Open Source Development Process (Lead:Timo Väliharju, COSS)
Track 5: Matriculation

At the end of the day all show and tell what they have achieved. The day is likely to be long and end around 6pm. After that we are planning to go for dinner together; have F2F discussions and get to know each other.

We really like the rest-gateway because it allows developers to build their services without having to add xroad's mandatory headers for each service.
I know the xroad-adapter developed in Java is fairly simple to tweak but it requires to do it once for each SOAP service behind my security server. Are there plans to build a multi-service SOAP adapter/gateway? I think this will speed up development and make communication transparent for internal services.

Thanks in advance for you help.

The below REST Gateway request throws an exception.

http://localhost:8080/rest-gateway-0.0.7/Consumer/api.finna.fi/v1/record?id[]=fennica.123&id[]=fennica.124

The exception is caused by [] characters that are special characters in XML. The numeric character reference should be used instead of the character for parameter values. In this case [] characters are in the parameter name which is why they cannot be used even with numeric character reference, because REST Gateway uses parameter names as XML element names.

01.09.2016 09:27:31 DEBUG ConsumerGateway : Add parameter : "id[]" -> "fennica.123".
01.09.2016 09:27:31 ERROR AbstractServiceRequestSerializer : INVALID_CHARACTER_ERR: An invalid or illegal XML character is specified.
org.w3c.dom.DOMException: INVALID_CHARACTER_ERR: An invalid or illegal XML character is specified.
        at com.sun.org.apache.xerces.internal.dom.CoreDocumentImpl.checkQName(CoreDocumentImpl.java:2603)
        at com.sun.org.apache.xerces.internal.dom.ElementNSImpl.setName(ElementNSImpl.java:121)
        at com.sun.org.apache.xerces.internal.dom.ElementNSImpl.<init>(ElementNSImpl.java:84)
        at com.sun.xml.internal.messaging.saaj.soap.impl.ElementImpl.<init>(ElementImpl.java:98)
        at com.sun.xml.internal.messaging.saaj.soap.impl.ElementFactory.createElement(ElementFactory.java:82)
        at com.sun.xml.internal.messaging.saaj.soap.SOAPDocumentImpl.createElement(SOAPDocumentImpl.java:98)
        at com.sun.xml.internal.messaging.saaj.soap.impl.ElementImpl.createElement(ElementImpl.java:393)
        at com.sun.xml.internal.messaging.saaj.soap.impl.ElementImpl.addElement(ElementImpl.java:373)
        at com.sun.xml.internal.messaging.saaj.soap.impl.ElementImpl.addChildElement(ElementImpl.java:122)
        at com.sun.xml.internal.messaging.saaj.soap.impl.ElementImpl.addChildElement(ElementImpl.java:130)
        at com.pkrete.xrd4j.tools.rest_gateway.ConsumerGateway$RequestSerializer.serializeRequest(ConsumerGateway.java:449)
        at com.pkrete.xrd4j.client.serializer.AbstractServiceRequestSerializer.serializeBody(AbstractServiceRequestSerializer.java:95)
        at com.pkrete.xrd4j.client.serializer.AbstractServiceRequestSerializer.serialize(AbstractServiceRequestSerializer.java:63)
        at com.pkrete.xrd4j.client.SOAPClientImpl.send(SOAPClientImpl.java:94)
        at com.pkrete.xrd4j.tools.rest_gateway.ConsumerGateway.processRequest(ConsumerGateway.java:176)
        at com.pkrete.xrd4j.tools.rest_gateway.ConsumerGateway.doGet(ConsumerGateway.java:275)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:620)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:727)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:303)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
        at org.apache.catalina.core.ApplicationDispatcher.invoke(ApplicationDispatcher.java:748)
        at org.apache.catalina.core.ApplicationDispatcher.processRequest(ApplicationDispatcher.java:486)
        at org.apache.catalina.core.ApplicationDispatcher.doForward(ApplicationDispatcher.java:411)
        at org.apache.catalina.core.ApplicationDispatcher.forward(ApplicationDispatcher.java:338)
        at com.pkrete.xrd4j.tools.rest_gateway.filter.ConsumerURIFilter.doFilter(ConsumerURIFilter.java:53)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
        at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:220)
        at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:122)
        at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:501)
        at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:170)
        at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:98)
        at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:950)
        at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:116)
        at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:408)
        at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1041)
        at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:607)
        at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:313)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)
01.09.2016 09:27:31 WARN  AbstractServiceRequestSerializer : Failed to serialize ServiceRequest message to SOAP.

Hi,

we have moved the xroad-rest-gateway repository under VRK's GitHub account to be more closely a part of Finnish X-Road. For more information about the moving, see #26.

New repository locates now here: https://github.com/vrk-kpa/REST-adapter-service
There can be found the latest version of component. This version includes some further development work & updated instructions for using it etc.

Could you close down the xroad-rest-gateway repository (or change it to be an Private Repository)?
Currently it confuses Finnish X-Road users and developers as there are two existing repositories for pretty much same purpose.

Thanks,
Hannu
Suomi.fi-palveluväylä (X-Road) Product Owner

X-Road SOAP headers (e.g. client, service, id, userId) need to be transmitted from REST Gateway to adapter server as the adapter server may need information stored into them. We need to define in which way this is done. Instead of adding the information stored into headers to the message body, it could be a better idea to transmit them as custom HTTP request headers. In this case we need to define the headers to be used.

Currently it's possible to use the below HTTP headers when sending a request to Consumer Gateway:

• X-XRd-UserId
• X-XRd-MessageId
• X-XRd-NamespaceSerialize
• X-XRd-NamespacePrefixSerialize
• Accept

In addition to HTTP headers, it should be possible to use URL parameters too for sending the same information. For example:

http://localhost:8080/RESTGateway/Consumer/FI_PILOT.GOV.1019125-0.Demo2Service.getOrganizationList.v1/49?X-XRd-UserId=test&X-XRd-MessageId=22&X-XRd-NamespaceSerialize=http://vrk-test.x-road.fi/producer&X-XRd-NamespacePrefixSerialize=ks&Accept=application/json

We would like to move the xroad-rest-gateway repository to be as a part of official X-Road and due to this, create a new Github repository for Rest Gateway component. Development work would be happening in the future using this repository. With this change the Finnish X-Road development team (later also Estonian development team) would start intensively develop the new features for REST Gateway and responsibility of REST Gateway component would be in the hands of same dev team as the official X-Road development currently is. Ultimate goal is to include REST Gateway as a default add-on component to X-Road installation package.

We would like to proceed as follows:

1. First step is to start using a new repository. We have already created the new repository for REST Gateway component and copied the source codes from original repository to below (new) repository:
   https://github.com/vrk-kpa/REST-adapter-service

This new repository would still be open source repository and open for all the open source contributors to make enhancements for components using pull request procedure.

2. Second step later in this year (2017) is to integrate REST Gateway component as a part of official X-Road:
   https://github.com/ria-ee/X-Road

Also, licence of REST Gateway component would be changed from EUPL => MIT (MIT is default licence for X-Road). We are currently checking and validating the licence type change requirements.

With your permission, we start using the new repository and continue development work using that repository? Preliminary we have agreed with Petteri that it is ok.

Thanks,
Hannu
Suomi.fi-palveluväylä (X-Road) Product Owner

In restful world you identify service using URL (example http://api.service.fi/users) but in security server/proxy world that wouldnt be the case I presume?

So the vision is to use something like http://localproxy (indicating the url of security server rest proxy) and then using X-XRd-Service http header to indicate what service you are calling?

In the readme.md the following links are broken:

• X-Road overview
• X-Road regulations

They point to SpeakerDeck:
https://speakerdeck.com/pilvivayla/...

Whereas there is no such account.

Currently Provider accepts only text/xml and application/json content-type:

https://github.com/educloudalliance/xroad-rest-gateway/blob/master/src/src/main/java/com/pkrete/xrd4j/tools/rest_gateway/ProviderGateway.java#L127

As application/xml is fairly common, perhaps it should be supported as an alternative form to text/xml.

As a developer I want to use such a self service portal, which allows me to register, get access and API key to all public APIs. I do not want to use separate service to apply for the permission or API key, it has to be automated eg work outside office hours. From the same service I want to see API docs and sandbox. Tools to communicate with rest of the developers would be nice. Also, ability to comment APIs would benefit all.

Perhaps this issue is not in the correct context but I do not know yet where the "palveluväylä" github issue list is.

@andreskytt, @margusf What sdsbInstance codes are you planning to use in Estonia? In Finland we'll probably use FI_DEV, FI_TEST and FI. I'm asking this, because in the Java library that I've developed and that is used in REST Gateway component, sdsbInstance codes are defined as enumeration types. I could add the Estonian codes to the library, if you just let me know what they are.

Consumer gateway throws java.lang.IllegalStateException: No modifications are allowed to a locked ParameterMap, because filterRequestParameters method tries to modify request parameter map which is unmodifiable.

We agreed that we need to collect more cases while doing tests/PoC with one organization. What about the rest? Some kind of open call to participate? Invite people to tell use cases from their organization's perspective?

It must be possible to repeat the same parameter multiple times on service calls, e.g. http://api.test.com/123?key=value1&key=value2&key=value3. Currently this is not possible, because parameters are stored in a key-value map Map<String, String> which means that each key can have only single value.

RIA has prepared an amendment which cancels the requirement to use an EUPL licence from X-road legislation (section 4). @andreskytt , what is the reasoning behind this?

When @kyyberi announced the X-Road REST API Design Guide Developer Day on Twitter it made me think about my chances to contribute to the project. I'm wondering if it is possible for anyone to set up a test environment? Is the process documented somewhere? For me it would be a prerequisite to test writing an X-Road service before I could contribute to a guide.

So is the current plan to wrap rest/json request inside SOAP request?( as per this image https://github.com/educloudalliance/xroad-rest-gateway/blob/master/images/rest-soap.png)?

This is a topic that came up in Dev Day meeting in Estonia.
This might be a good starting point https://github.com/18f/api-standards

We need to agree on license to use. EUPL might be one option since that is according to rumors thought for X-Road. Of course we are not obligated to use the same,but I sitll some compatible license would be wise choice. Currently the repository contains reference to MIT license. That has to be changed if our choice is different.

Are there any tools for RAML -> WSDL conversion available? Horizon scanning is needed to identify possible tools and/or libraries.

Should we have just for this purpose (X-Road development) a joint chatting environment such as Slack? So far the discussions under Educloud Slack has been minimal. This would be one task for the community manager to fix or solve; selecting own co-operation exchange environment.

I think Markus Lanthaler's presentation about JSON-LD and Hydra is worth seeing. If you are in a hurry jump straight to [13:45] where functional Hydra-annotations are discussed.

JSON-LD/Hydra tooling is limited compared to RAML and friends. Markus says in SO that he is working on a document generator but I haven't see anything yet. On the video he presents HydraConsole for automated interaction but it would also be nice to have a traditional document generator for onboarding traditional developers who don't want to understand JSON-LD.

JSON-LD and Hydra could be useful in the X-Road context for the horizontal scalability. Maybe it would even help in bridging WDSL-wisdom to the REST-world (and vice versa).

Should this repository be replaced under Joint X-Road organisation in Github https://github.com/jointxroad ?