mfatihercik / dsm

Declarative Stream Mapping (DSM) is a stream de/serializer library for XML and JSON. DSM allows you to make custom parsing, filtering, transforming, aggregating, grouping on any JSON or XML document at stream time(read only once).

Introduction

Declarative Stream Mapping(DSM) is a stream deserializer library that makes parsing of XML and JSON easy. DSM allows you to make custom parsing, filtering, transforming, aggregating, grouping on any JSON or XML document at stream time(read only once). DSM uses yaml or json for configuration definitions

If you parsing a complex, huge file and want to have high performance and low memory usage then DSM is for you.

Features

• Work for both XML and JSON
• Custom stream parsing
• Filtering by value on any field with very low cognitive complexity
• Flexible value transformation.
• Default value assignment
• Custom function calling during parsing
• Powerful Scripting (Apache JEXL, Groovy, Javascript and other jsr223 implementations are supported)
• Multiple inheritance between DSM config file (DSM file can extends to another config file)
• Reusable fragments support
• Very short learning curve
• Memory and CPU efficient
• Partial data extraction from JSON or XML
• String manipulation with expression

Installation

Jackson

<dependency>
  <groupId>com.github.mfatihercik</groupId>
  <artifactId>dsm</artifactId>
  <version>1.0.5</version>
</dependency>

Gradle

compile ('com.github.mfatihercik:dsm:1.0.5')

Documentation.

dsm's Issues

The conversion function is executed correctly only if path is set for the field. Is this necessary

I want to be able to execute the function correctly even if the path mapping is not set. One way I can do this is to set default, but if default is set to null, the function will not execute correctly

One more duplicate data exists when converting json array

JSON:

 [

  {
    "id": 8531,
    "category": {
      "id": 0,
      "name": "string"
    },
    "name": "doggie",
    "photoUrls": [
      "string"
    ],
    "tags": [
      {
        "id": 3,
        "name": "string"
      }
    ],
    "status": "available"
  },
  {
    "id": 9009,
    "category": {
      "id": 0,
      "name": "string"
    },
    "name": "doggie",
    "photoUrls": [
      "string"
    ],
    "tags": [
      {
        "id": 2,
        "name": "string"
      }
    ],
    "status": "available"
  }
]

yaml:

sourceSystem: "ASD"
destinationSystem: "ASDF"
params:
  dateFormat: yyyy-MM-dd'T'HH:mm:ss'Z'
result:
  type: array
  path: ..../
  #  filter: $self.index%3==0
  xml:
    path: "/pets/Pet"
  fields:
    category:
      type: object
      fields:
        id:
          dataType: long
        name:
    #          tagId:
    #             type: object
    #             filter: $self.index==3
    #             path: ../tags
    ##             parentPath: ../tags
    #             xml:
    #               path: /pets/Pet/tags/tag
    #             fields:
    #               id: int
    #               name:
    #
    tagIds:
      dataType: long
      type: array
      path: tags/id
      xml:
        path: tags/tag/id

    id:
      dataType: long
    name:
    status:
    photoUrls:
      type: array
      path: photoUrls
      xml:
        path: photoUrls/photoUrls
    tags:
      type: array
      path: tags
      xml:
        path: tags/tag
      fields:
        id:
          dataType: long
        name:
         

Unit Test:

@Test
    public void parsePet() throws IOException {

        Path rootPath = Paths.get(TestUtils.getTestResourcePath(), pathPlace);

        DSMBuilder builder = new DSMBuilder(rootPath.resolve("pet.yml").toFile(), rootPath.toString());
        DSM dsm = builder.setType(DSMBuilder.TYPE.JSON).create(Pet.class);
        File file = rootPath.resolve("pet.json").toFile();

        Object object = dsm.toObject(file);

        List<Pet> list = (List<Pet>) object;
        String s = JSONUtil.toJsonStr(list);
        System.out.println(s);
    }

Result:

[
	{
		"photoUrls": [
			"string"
		],
		"tagIds": [
			3
		],
		"tags": [
			{
				"name": "string",
				"id": 3
			}
		],
		"name": "doggie",
		"id": 8531,
		"category": {
			"name": "string",
			"id": 0
		},
		"status": "available"
	},
	{
		"photoUrls": [
			"string"
		],
		"tagIds": [
			2
		],
		"tags": [
			{
				"name": "string",
				"id": 2
			}
		],
		"name": "doggie",
		"id": 9009,
		"category": {
			"name": "string",
			"id": 0
		},
		"status": "available"
	},
	{
		"photoUrls": [
			"string"
		],
		"tagIds": [
			2
		],
		"tags": [
			{
				"name": "string",
				"id": 2
			}
		],
		"name": "doggie",
		"id": 9009,
		"category": {
			"name": "string",
			"id": 0
		},
		"status": "available"
	}
]

There are two items with the same id "9009". It seems one more data exists.
I researched there is a new version 1.0.5 to fix similar bug. But when I debuged in the code at "case END_ARRAY", found it go to another condition at line 27 as below.

Is there any missing configuration for me?
@mfatihercik

help to check XML node

great stuff!.

how can I map the following when the XML node define the value:

XML Source:
<GigabitEthernet> <name>1/0/5</name> <speed> <auto/> </speed> </GigabitEthernet>

or

<GigabitEthernet> <name>1/0/6</name> <speed> <1g/> </speed> </GigabitEthernet>

Expected result:
{ "name":"1/0/5", "speed":"AUTO" }
or
{ "name":"1/0/6", "speed":"1000" }

java.lang.IllegalArgumentException: URI is not hierarchical at java.io.File.<init>(File.java:418)

Hi.....

I got the same error even i added new File("") in DSMBuilder

this is my sample code

DSMBuilder builder = new DSMBuilder(new File("dsm-config-file.yaml"));
DSM dsm = builder.setType(DSMBuilder.XML).create();
Pet pet = dsm.toObject(new File("path/to/xmlFile.xml"),Pet.class);

i try to solve my problem by using below question.....

java.lang.IllegalArgumentException: URI is not hierarchical..

but i codn't figure out the problem please help me..

if you want i will share my code ...

Can field maps support wildcards

I only want to map the object, and the fields in the object and the child objects are mapped automatically

java.lang.IllegalArgumentException: URI is not hierarchical

Hi, I'm trying to use this library according to documentation but no matter what path I use i get the same error as in title. For example :

DSMBuilder builder = new DSMBuilder("/tmp/targetFile.yaml");
DSM dsm = builder.setType(DSMBuilder.XML).create();

Caused by: java.lang.IllegalArgumentException: URI is not hierarchical
at java.base/java.io.File.(File.java:418)
at com.github.mfatihercik.dsb.resource.FileSystemResourceAccessor.addRootPath(FileSystemResourceAccessor.java:54)
at com.github.mfatihercik.dsb.resource.AbstractResourceAccessor.init(AbstractResourceAccessor.java:39)
at com.github.mfatihercik.dsb.resource.FileSystemResourceAccessor.init(FileSystemResourceAccessor.java:47)
at com.github.mfatihercik.dsb.resource.FileSystemResourceAccessor.(FileSystemResourceAccessor.java:27)
at com.github.mfatihercik.dsb.configloader.AbstractConfigLoader.setResourceAccessor(AbstractConfigLoader.java:89)
at com.github.mfatihercik.dsb.configloader.AbstractConfigLoader.(AbstractConfigLoader.java:32)
at com.github.mfatihercik.dsb.configloader.YamlConfigLoaderStrategy.(YamlConfigLoaderStrategy.java:11)
at com.github.mfatihercik.dsb.DSMBuilder.create(DSMBuilder.java:99)
at com.github.mfatihercik.dsb.DSMBuilder.create(DSMBuilder.java:93)

I don't understand what the problem is, the path is correct and the file is present.

Handling arrays of arrays

Trying to write config to handle array of arrays.
Input to be parsed looks like:

[
  {
    "columns": [
      "name1",
      "name2",
      "name3"
    ],
    "events": [
      [
        "value1_1",
        "value1_2",
        "value1_3"
      ],
      [
        "value2_1",
        "value2_2",
        "value2_3"
      ],
      [
        "value3_1",
        "value3_2",
        "value3_3"
      ]
    ]
  }
]

Config (in JSON):

{
  "version": 1.0,
  "result":{
    "type":"object",
    "path":"/",
    "function": "processStuff",
    "fields":{
      "columns": {
        "type": "array"
      },
      "events": {
        "type": "array"
      }
    }
  }
}

The result looks like:
{columns=[name1, name2, name3], events=[value1_1, value1_2, value1_3, null, value2_1, value2_2, value2_3, null, value3_1, value3_2, value3_3, null]}

Looking to get a result like:
columns=[name1, name2, name3], events [[value1_1, value1_2, value1_3], [value2_1, value2_2, value2_3], [value3_1, value3_2, value3_3]]}

Tried a few variations of config but wasn't able to get the result I was looking for. Can DSM handle this kind of input?

Working with DSM in my project

I Need to save a complex XML file in DB then get it in Json format, I need to work with DSM, Please Help

Can we support object to string and string to object

I need to map an object in json to a string in the new json, can I do that?

bigDecimal dataTYpe not working

bigDecimal dataTYpe not working

It's a great project，

Is it still maintained?

parsing element is not merged when path is null

First:

         METHOD:
         -  parentPath: header
            default: $all.result.data.TYPOLOGY
            transformationCode: METHOD

Second

         METHOD:
         -  parentPath: header
            default: $self.parent.data.TYPOLOGY

second extends to first

dynamic fieldName

Currently, the field name is static. it is not changed. Defining field name by expression will be very efficient

The conversion seems a little problematic when the root node is an array

version:1.0.4

source json:
[
{
"address": {
"se": {
"db": "f"
},
"addressName": "2dsf3",
"addressPwd": "323"
}
}
]

mapping json:
{
"result" : {
"path" : "/",
"type" : "array",
"fields" : {
"address": {
"type": "object",
"path": "/address",
"fields": {
"addressName" :{
"path": "/address/addressName",
"dataType": "string"
},
"addressPwd": {
"path": "/address/addressPwd",
"dataType": "string"
}
}
}
}
}
}
result:
[{address={addressName=2dsf3, addressPwd=323}}, {address={addressName=2dsf3, addressPwd=323}}]

new count type

new type to get the number of tags exists in XML or JSON.

dynamic filter - clarification - Urgent

with this library, is it possible to have filter expression to match inputs?
for eg: self.data.salary>10000, is it possible to accept the value to be compared (instead of hard coded value 10000) from input? like from java.

I have a requirement as follows
I have an array of objects as follows
{
"operations" : [
{
"name" : "operationname1",
"version" : "version",
"operationtype" : "single",
"transformationclass" :"fullpackage class"
},
{
"name" : "operationname2",
"version" : "version",
"operationtype" : "single",
"transformationclass" :"fullpackage class"
}
]
}

a) I get operation name and version from input, so i need to load only operation type and transformationClass only for the provided inputs. Is that even possible with DSM library?

b) If thats not possible, I can try loading all the elements always and do normal java Collection Streams operation to filter - In this case what benefit will i get , as it is same as loading using ObjectMapper ( jackson Library).

c) is DSM supported for concurrent operations?

setParams function not work as expected

DSMBuilder.setParams function is not affecting to params object.

How to parse XML if format in below manner?

@mfatihercik

Consider I have my xml file in below format (its a partial file)

<package name="com/example/configure">
		<class name="com/example/configure/AutoConfigure"
			sourcefilename="AutoConfigure.java">
			<method name="tryItOut;" desc="()V" line="8">
				<counter type="INSTRUCTION" missed="3" covered="0" />
		        </method>
			<method name="sampleController"
				desc="()Lcom/example/controller/SampleController;" line="13">
				<counter type="INSTRUCTION" missed="4" covered="23" />
			</method>
                 </class>
</package>
<package name = "something>
                <class name = 
                       ...

I have to loop through multiple packages which can have multiple class names and which in-turn can have multiple methods inside the class

using DSM how should I capture the package name , class name and method name only if covered !=0
and filter the XML document and convert it to JSON output format

PS:I am new to java and the examples I saw on your website have data inserted between <class>"ClassName"</class> but not as <class name = "ClassName">...</class>

Filtering

Firstly, congratulations on the project, it's quite creative. For the filter, let's suppose I have a source.json
//source.json
[
{
"provider": "VODAFONE"
},
{
"provider": "VERIZON"
},
{
"provider": "TEXAS"
}
]

//config
version: 0.1.0
result:
type: array
path: /
filter: $self.data.provider == 'TEXAS' || $self.data.provider == 'VODAFONE'
#filter: $self.data.provider == 'TEXAS' or $self.data.provider == 'VODAFONE' i tried it too
fields:
provider: string

I would like to filter by texas and vodafone, but it is returning only records for the first branch, maybe i lost something.