mtconnect / cppagent_dev Goto Github PK

WG Proposal #2288

• Create test for XML Parser of relationships
• Create unit test for originator data item relationship
  • XML
  • JSON
• Create unit test for specification relationships
  • XML
  • JSON
• Create relationship model
• Verify ids for relationships
• Parse data item relationship
• Generate XML
• Generate JSON

Refactor storage for checkpoints. Currently this uses a raw array allocated on the heap. This can be replaced by std::vector to make things easier.

Parse and generate xml/json

WG Proposal #2342

• Create parse test for attribute from XML
• Create unit test for attribute in XML
• Create unit test for attribute in JSON
• Parse attribute
• Generate XML
• Generate JSON

I am not sure if I got the entire picture, but it seems RefCountedPtr behaves like a std::shared_ptr and could be replaced. In addition there is a flag that no reference is taken. This could be modeled with std::weak_ptr. (Where is it needed anyway?)

If that's true I could to the dirty work :)

WG Proposal #1790

• Create an asset test set for assets
• Add support for cutting tool model generation in JSON

A Matrix will support offset tables required by machine tools. They will have a header and rows using the data set functionality to support a sparse matrix of values.

Remove threading of adapter connector and use ASIO to remove threading complexity.

This is part of Epic #3

Remove dlib::unit64 and replace with uint64_t

This is part of the Epic #3.

Refactor ChangeObserver to remove dlib

• Replace dlib::rmutex with std::recursive_mutex
• Replace dlib::rsignaler with std::condition_variable_any

This is part of the Epic #3

Hey @wsobel,

While running some tests on the json output I immediately noticed that @ characters were being added to the attributes. I'm not sure why adding the @ is necessary. Could you explain that?

I am suggesting to (and willing to do the work) remove them from the jsonPrinter because the @ symbols actually make the data difficult to work with in JavaScript and potentially other languages.

For example we would typically access json objects as such:

const bufferSize = data.MTConnectStreams.Header.bufferSize;

With the @ symbol we would have to write additional code such as:

const bufferSize = data.MTConnectStreams.Header['@bufferSize'];

{
  "MTConnectStreams": {
    "Header": {
      "@bufferSize": 131072,
      "@creationTime": "2019-08-07T22:16:38Z",
      "@firstSequence": 1,
      "@instanceId": 1565029019,
      "@lastSequence": 80,
      "@nextSequence": 81,
      "@sender": "fw-mdcs-01",
      "@testIndicator": false,
      "@version": "1.2"
    },
    "Streams": [
      {
        "DeviceStream": {
          "@name": "C282FM",
          "@uuid": "arduino1",
          "ComponentStreams": [
            {
              "ComponentStream": {
                "@component": "Device",
                "@componentId": "ardiuno1",
                "@name": "C282FM",
                "Events": [
                  {
                    "AssetChanged": {
                      "@assetType": "",
                      "@dataItemId": "ardiuno1_asset_chg",
                      "@sequence": 1,
                      "@timestamp": "2019-08-05T18:16:59.003225Z",
                      "Value": "UNAVAILABLE"
                    }
                  },
                  {
                    "Availability": {
                      "@dataItemId": "avail",
                      "@name": "avail",
                      "@sequence": 79,
                      "@timestamp": "2019-08-07T02:16:18.712846Z",
                      "Value": "UNAVAILABLE"
                    }
                  },
                  {
                    "Message": {
                      "@dataItemId": "sys_msg",
                      "@name": "system",
                      "@sequence": 4,
                      "@timestamp": "2019-08-05T18:16:59.003225Z",
                      "Value": "UNAVAILABLE"
                    }
                  }
                ],
                "Samples": [
                  {
                    "Flow": {
                      "@dataItemId": "flow1",
                      "@name": "flow",
                      "@sequence": 80,
                      "@timestamp": "2019-08-07T02:16:18.712846Z",
                      "Value": 0
                    }
                  }
                ]
              }
            }
          ]
        }
      }
    ]
  }
}

To support additions to the model for version 1.5 specifications, will need to parse and generate content in XML and JSON.

I compiled a v1.5 agent. When I output json, the agent version shows v1.3. (schema ver)

Header": {"@assetBufferSize": 1024,"@assetCount": 0,"@buffersize": 4096,"@CreationTime": "2021-05-16T05:57:16Z","@instanceid": 1621144500,"@sender": "apm-86xxx-ie4000","@testIndicator": false,"@Version": "1.3"}

When I switch back to xml…

<MTConnectDevices xmlns:m="urn:mtconnect.org:MTConnectDevices:1.3" xmlns="urn:mtconnect.org:MTConnectDevices:1.3"xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="urn:mtconnect.org:MTConnectDevices:1.3 /schemas/MTConnectDevices_1.3.xsd">

Add support for Relationship, and Coordinate Systems.

Related to #25.

There is a need of supporting device xml file per adapter.

Reasons: Customers wants to be able to obtain data from different adapters using a single device ID. As a result, vendors providing adapter need to modify OEM devices.xml to add their own tags.
It becomes an issue when devices.xml is updated by OEM.

It would be best if vendors can provide their own device xml file and agent can support device xml file per adapter for each device ID.

Thanks

Per 1.6, need to make sure the discrete flag is set for the data item.

WG Proposal #2192

• Create unit test for capitalization in XML
• Add to exceptions

Refactor memory management in Agent class

• Replace several instances of objects being managed on the heap with new/delete and convert to use std:unique_ptr

I got the following errors when validating the schema 1.4 for sample and current in the latest version

Attribute 'assetType' is required in element .
Annotations of element 'AssetChanged' (see below)
An asset was just modified
Annotations of type 'AssetChangedType' (see below)
An asset was just modified
Error location: MTConnectStreams / Streams / DeviceStream / ComponentStream / Events / AssetChanged
Details
cvc-complex-type.4: Required attribute 'assetType' is missing from element .
cvc-type.3.2: Element is not valid with respect to type definition 'AssetChangedType'.
cvc-elt.5: The element is not valid with respect to the actual type definition 'AssetChangedType'.
cvc-assess-elt.1.1: Strict assessment of element with governing element declaration 'AssetChanged' failed.

Add Mosquito client to MTConnect Agent to support the publishing of streaming data and meta-model. Issues are as follows:

• Need to create a TR regarding topic format and structure.
• Is payload JSON (my vote is yes)?
• Is there any special treatment of the Probe metadata?

Add XCode test framework support for OS X and GoogleTest

This issue represents an Epic.

Dependencies on Dlib should be migrated to std or Boost equivalents. Boost serves as a precursor to feature adoption in the C++ standard, so using types from this library should make the Agent more future proof and easier to maintain in the future.

As this is such a large task this should be done in multiple phases.

• Remove any Dlib primitive types which have subsequently been added to the C++ standard, such as dlib::uint64
• Remove more complex types such as dlib's file systems abstractions, threads, arrays, queue.
• Replace functional items such as the config_reader and tokenizer
• Last is to replace Dlib's web-server components with a Boost-ASIO or Boost-Beast implementation

As GitHub does not natively handle Epic stories I suggest that associated items are linked to this one as an when required.

With Git Submodules one has very limited option for downloading (no way to specify specific hash or similar) and building. With ExternalProject from CMake we gain better control on what to download, what to patch and how to build/install.

This would also keep our own project clean from others (e.g. with add_subdirectory we get all definitions of the subprojects if they do not use the proper set_target_XXX() commands which they don't...)

ExternalProject should also ease the planned integration of Boost, I guess.

Any objections?

Begin migration to boost by changing the HTTP handler from the dlib http server to boost beast.

Memory allocation/de-allocation should be managed using std::unique_ptr in the Agent::CachedFile class.

Moving away from new/delete, malloc/free makes memory management consistent and trackable by static code analyzers.

Abstract XML parser to make it extensible and handle different representations like json and new asset types in a cleaner fashion.

Test case 'ChangeObserverTest::testSignalObserver' occasionally fails when testing that an observer has timed-out before being signalled.

This is due to spurious thread wake-up on Windows and POSIX systems.

I suggest the tests are re-factored to solve this issue and remove the dlib threading portions (to contribute towards the larger Epic #3).

For the record....

I need help with an explanation of how the disambiguation of tag names is done in MTConnect, especially since many clients use xpath for XML searching to find an element.

Below is two vendor MTConnect structuring of XML assuming we have a two spindle CNC:

Vendor 1 Devices.xml:
<Components> <Axes id="a1" name="base"> <Components> <Rotary id="d1" name="SPINDLE1"> <DataItems> <DataItem type="SPINDLE_SPEED" id="cs1" category="SAMPLE" name="Srpm" units="REVOLUTION/MINUTE" nativeUnits="REVOLUTION/MINUTE"/> </DataItems> </Rotary> <Rotary id="d2" name="SPINDLE2"> <DataItems> <DataItem type="SPINDLE_SPEED" id="cs2" category="SAMPLE" name="Srpm2" units="REVOLUTION/MINUTE" nativeUnits="REVOLUTION/MINUTE"/> </DataItems> </Rotary>

Vendor 2: (very simple and parseable XML but probably wrong)
<Components> <Controller name="Controller" id="c41"> <DataItems> <DataItem category="SAMPLE" id="spindlespeed" name="p1_Srpm" type=" SPINDLE_SPEED " /> <DataItem category="SAMPLE" id="spindlespeed2" name="p2_Srpm" type=" SPINDLE_SPEED " />

So one cannot just use //DataItem[@srpm] to search for the srpm tag in the second vendor. So lets say we rename p1_Srpm to Srpm we can now search for the correct tag name. However, the adapter will still be emitting p1_Srpm as the spindle rpm so that the agent won’t find a counterpart in the devices.xml. This is why having adapter tag renames in the agent.cfg might be useful.

( I realize these may be imperfect device models and not adherent to the standard. They parse.)

Anyway, what is the correct XML layout to avoid name disambiguation, since at the latest MTConnect user group meeting the correct XML structure was ?

And finally if you have multiple spindles and you really only care about the spindle activity, this too would have to post processed upstream? Or is there a way to merge spindle readings.

Currently, we are not testing all the unit conversions. We should add a test for each conversion pair. I would suggest adding a template to reduce code repetition.

The header for each mime chunk has mime type of text/xml instead of json when accept is json.

This will require the following tasks:

1. Merge the upstream probe results to create a new configuration.
2. Rewrite the Ids to make sure they are unique (maintaining the old ids for incoming data).
3. Stream data from the upstream agent and add to outgoing buffer.

For incoming, we could use libcurl in the beginning and possibly boost beast when we migrate to boost. There is some C streaming code that could be leveraged to process chunked data.

Currently we can add and update assets. But we cannot remove / Delete an asset via HTTP

WG Proposal #2163

• Add Solid Model to Agent Lib
• Create test XML Configuation File
• Create unit test for Solid Model parsing
• Parse Solid Model element
• Create unit tests for generation
  • XML
  • JSON
• Refactor common transformation for Coordinate System, Kinematics (todo), and Solid Model

WG Proposal #2227

1 psi = 6,894.75 Pa

Switch the current cppunit/CUTI code to google test and google mock libraries.

Use boost logging module instead of the current dlib rolling logger.

Part of Epic #3

I got the following error in my extended schema
The variety of simple type definition '{anonymous}' is absent. This is only permitted for xs:anySimpleType.
Error location: xs:schema / xs:complexType / xs:simpleContent
Details
st-props-correct.1: The variety of simple type definition '{anonymous}' is absent. This is only permitted for xs:anySimpleType.

It is working on 1.3 and 1.4 but not in 1.5 schema.

<xs:complexType name="SwitchValueType">
	<xs:annotation>
		<xs:documentation>
    The possible values for switch.
  </xs:documentation>
	</xs:annotation>
	<xs:simpleContent>
		<xs:restriction base="mt:EventType">
			<xs:enumeration value="ON"/>
			<xs:enumeration value="OFF"/>
			<xs:enumeration value="UNAVAILABLE"/>
		</xs:restriction>
	</xs:simpleContent>
</xs:complexType>

the schema is using <xs:attribute name='cuttingToolId' type='CuttingToolIdType' use='required'/> but
document is using toolId instead of cuttingToolId.

Which one should I use? I checked 1.4 and 1.5 and they are the same.

Thanks

Abstract the XML generation so alternatives, like json, are possible.

We have a lot of cases where it would be beneficial to reference other agents that effect overall systems but aren't part of the original adapter.

Example: A water temperature sensor that directly broadcasts data to its own water temp type agent. If you had two machines that shared this water source and needed to report the temperature with them, the agent's existing device file could reference this child agent, allowing both of the machine agents to report the data when polled. This also allows for easier integration of sensors with built in adapters reporting to purpose built agents to feed data that isn't currently supported. The child agent could also be hit on its own if you needed only its values.

Example: An older CNC machine might be able to report minimal data on spindle and x, y coordinates. I might also want data involving the condition of lubrication and coolant using a small microcontroller adapter into a second agent. A parent agent when polled would then poll both children agents as needed or the children agents would update the parent on changes. This parent agent would represent the overall device.

This could be used further to integrate vibration sensors in older hardware to collect values for machine learning, as well as create edge device that could alert when failures were detected.

Enhance sample query to support the concept of "To=" with a specified # of items.

Instead of "From=X" and "Count=Y" to retrieve Y data items starting from X, we want to find out the history of a particular path up to the sequence num X.

https://<authority>/sample?count=100&to=12345&path=//DataItem[@id='XYZ']

Last 100 data items of 'XYZ' before sequence num 12345

Refactor memory management in tests

Replace instances of objects being managed on the heap with new/delete and convert to std:unique_ptr

Use boost spirit recursive decent parser module to parse dlib configuraiton format into property lists.

Part of Epic #3

Remove dlib::create_thread and replace with std::thread

This refactors all thread related functionality/tests to use standard C++thread functions/concepts.

Lambdas can be used to allow thread functions to be located closer to their context.

This is part of the larger Epic #3

WG Proposal #2192

Add parser/generator support for relationships.

When POSTing a data item, if the data item is not defined a success response and a body of is returned to the sender.

Ideally the http status code would have an error. But at least the body should have the (oh so painful to parse) MTConnectError.

wrt to the status code, posting to invalid device name would also ideally return a status code error, not success

WG Proposal #2289

• Parse originator, limit types, and Process Specification
  • Test originator
  • Test Specification
  • Test Process Specification
• Create unit test for originator attribute
  • XML
  • JSON
• Create unit test for additional limit types
  • XML
  • JSON
• Create unit test for Process Specification
  • XML
  • JSON