mtconnect / cppagent_dev Goto Github PK
View Code? Open in Web Editor NEWPrivate C++ Agent for Pending Release Development
License: Apache License 2.0
Private C++ Agent for Pending Release Development
License: Apache License 2.0
Add XCode test framework support for OS X and GoogleTest
Switch the current cppunit/CUTI code to google test and google mock libraries.
Remove dlib::create_thread and replace with std::thread
This refactors all thread related functionality/tests to use standard C++thread functions/concepts.
Lambdas can be used to allow thread functions to be located closer to their context.
This is part of the larger Epic #3
Per 1.6, need to make sure the discrete flag is set for the data item.
Parse and generate xml/json
WG Proposal #2342
This issue represents an Epic.
Dependencies on Dlib should be migrated to std or Boost equivalents. Boost serves as a precursor to feature adoption in the C++ standard, so using types from this library should make the Agent more future proof and easier to maintain in the future.
As this is such a large task this should be done in multiple phases.
config_reader
and tokenizer
As GitHub does not natively handle Epic stories I suggest that associated items are linked to this one as an when required.
WG Proposal #2192
Use boost spirit recursive decent parser module to parse dlib configuraiton format into property lists.
Part of Epic #3
Test case 'ChangeObserverTest::testSignalObserver' occasionally fails when testing that an observer has timed-out before being signalled.
This is due to spurious thread wake-up on Windows and POSIX systems.
I suggest the tests are re-factored to solve this issue and remove the dlib threading portions (to contribute towards the larger Epic #3).
Remove dlib::unit64 and replace with uint64_t
This is part of the Epic #3.
Begin migration to boost by changing the HTTP handler from the dlib http server to boost beast.
Remove threading of adapter connector and use ASIO to remove threading complexity.
This is part of Epic #3
WG Proposal #2288
Hey @wsobel,
While running some tests on the json output I immediately noticed that @
characters were being added to the attributes. I'm not sure why adding the @
is necessary. Could you explain that?
I am suggesting to (and willing to do the work) remove them from the jsonPrinter because the @
symbols actually make the data difficult to work with in JavaScript and potentially other languages.
For example we would typically access json objects as such:
const bufferSize = data.MTConnectStreams.Header.bufferSize;
With the @
symbol we would have to write additional code such as:
const bufferSize = data.MTConnectStreams.Header['@bufferSize'];
{
"MTConnectStreams": {
"Header": {
"@bufferSize": 131072,
"@creationTime": "2019-08-07T22:16:38Z",
"@firstSequence": 1,
"@instanceId": 1565029019,
"@lastSequence": 80,
"@nextSequence": 81,
"@sender": "fw-mdcs-01",
"@testIndicator": false,
"@version": "1.2"
},
"Streams": [
{
"DeviceStream": {
"@name": "C282FM",
"@uuid": "arduino1",
"ComponentStreams": [
{
"ComponentStream": {
"@component": "Device",
"@componentId": "ardiuno1",
"@name": "C282FM",
"Events": [
{
"AssetChanged": {
"@assetType": "",
"@dataItemId": "ardiuno1_asset_chg",
"@sequence": 1,
"@timestamp": "2019-08-05T18:16:59.003225Z",
"Value": "UNAVAILABLE"
}
},
{
"Availability": {
"@dataItemId": "avail",
"@name": "avail",
"@sequence": 79,
"@timestamp": "2019-08-07T02:16:18.712846Z",
"Value": "UNAVAILABLE"
}
},
{
"Message": {
"@dataItemId": "sys_msg",
"@name": "system",
"@sequence": 4,
"@timestamp": "2019-08-05T18:16:59.003225Z",
"Value": "UNAVAILABLE"
}
}
],
"Samples": [
{
"Flow": {
"@dataItemId": "flow1",
"@name": "flow",
"@sequence": 80,
"@timestamp": "2019-08-07T02:16:18.712846Z",
"Value": 0
}
}
]
}
}
]
}
}
]
}
}
This will require the following tasks:
For incoming, we could use libcurl in the beginning and possibly boost beast when we migrate to boost. There is some C streaming code that could be leveraged to process chunked data.
A Matrix will support offset tables required by machine tools. They will have a header and rows using the data set functionality to support a sparse matrix of values.
I compiled a v1.5 agent. When I output json, the agent version shows v1.3. (schema ver)
Header": {"@assetBufferSize": 1024,"@assetCount": 0,"@buffersize": 4096,"@CreationTime": "2021-05-16T05:57:16Z","@instanceid": 1621144500,"@sender": "apm-86xxx-ie4000","@testIndicator": false,"@Version": "1.3"}
When I switch back to xml…
<MTConnectDevices xmlns:m="urn:mtconnect.org:MTConnectDevices:1.3" xmlns="urn:mtconnect.org:MTConnectDevices:1.3"xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="urn:mtconnect.org:MTConnectDevices:1.3 /schemas/MTConnectDevices_1.3.xsd">
Abstract XML parser to make it extensible and handle different representations like json and new asset types in a cleaner fashion.
Add parser/generator support for relationships.
Abstract the XML generation so alternatives, like json, are possible.
Add support for Relationship, and Coordinate Systems.
Related to #25.
For the record....
I need help with an explanation of how the disambiguation of tag names is done in MTConnect, especially since many clients use xpath for XML searching to find an element.
Below is two vendor MTConnect structuring of XML assuming we have a two spindle CNC:
Vendor 1 Devices.xml:
<Components> <Axes id="a1" name="base"> <Components> <Rotary id="d1" name="SPINDLE1"> <DataItems> <DataItem type="SPINDLE_SPEED" id="cs1" category="SAMPLE" name="Srpm" units="REVOLUTION/MINUTE" nativeUnits="REVOLUTION/MINUTE"/> </DataItems> </Rotary> <Rotary id="d2" name="SPINDLE2"> <DataItems> <DataItem type="SPINDLE_SPEED" id="cs2" category="SAMPLE" name="Srpm2" units="REVOLUTION/MINUTE" nativeUnits="REVOLUTION/MINUTE"/> </DataItems> </Rotary>
Vendor 2: (very simple and parseable XML but probably wrong)
<Components> <Controller name="Controller" id="c41"> <DataItems> <DataItem category="SAMPLE" id="spindlespeed" name="p1_Srpm" type=" SPINDLE_SPEED " /> <DataItem category="SAMPLE" id="spindlespeed2" name="p2_Srpm" type=" SPINDLE_SPEED " />
So one cannot just use //DataItem[@srpm] to search for the srpm tag in the second vendor. So lets say we rename p1_Srpm to Srpm we can now search for the correct tag name. However, the adapter will still be emitting p1_Srpm as the spindle rpm so that the agent won’t find a counterpart in the devices.xml. This is why having adapter tag renames in the agent.cfg might be useful.
( I realize these may be imperfect device models and not adherent to the standard. They parse.)
Anyway, what is the correct XML layout to avoid name disambiguation, since at the latest MTConnect user group meeting the correct XML structure was ?
And finally if you have multiple spindles and you really only care about the spindle activity, this too would have to post processed upstream? Or is there a way to merge spindle readings.
There is a need of supporting device xml file per adapter.
Reasons: Customers wants to be able to obtain data from different adapters using a single device ID. As a result, vendors providing adapter need to modify OEM devices.xml to add their own tags.
It becomes an issue when devices.xml is updated by OEM.
It would be best if vendors can provide their own device xml file and agent can support device xml file per adapter for each device ID.
Thanks
The header for each mime chunk has mime type of text/xml instead of json when accept is json.
I am not sure if I got the entire picture, but it seems RefCountedPtr behaves like a std::shared_ptr and could be replaced. In addition there is a flag that no reference is taken. This could be modeled with std::weak_ptr. (Where is it needed anyway?)
If that's true I could to the dirty work :)
When POSTing a data item, if the data item is not defined a success response and a body of is returned to the sender.
Ideally the http status code would have an error. But at least the body should have the (oh so painful to parse) MTConnectError.
wrt to the status code, posting to invalid device name would also ideally return a status code error, not success
I got the following error in my extended schema
The variety of simple type definition '{anonymous}' is absent. This is only permitted for xs:anySimpleType.
Error location: xs:schema / xs:complexType / xs:simpleContent
Details
st-props-correct.1: The variety of simple type definition '{anonymous}' is absent. This is only permitted for xs:anySimpleType.
It is working on 1.3 and 1.4 but not in 1.5 schema.
<xs:complexType name="SwitchValueType">
<xs:annotation>
<xs:documentation>
The possible values for switch.
</xs:documentation>
</xs:annotation>
<xs:simpleContent>
<xs:restriction base="mt:EventType">
<xs:enumeration value="ON"/>
<xs:enumeration value="OFF"/>
<xs:enumeration value="UNAVAILABLE"/>
</xs:restriction>
</xs:simpleContent>
</xs:complexType>
WG Proposal #2227
Refactor memory management in Agent class
new
/delete
and convert to use std:unique_ptr
Add Mosquito client to MTConnect Agent to support the publishing of streaming data and meta-model. Issues are as follows:
the schema is using <xs:attribute name='cuttingToolId' type='CuttingToolIdType' use='required'/> but
document is using toolId instead of cuttingToolId.
Which one should I use? I checked 1.4 and 1.5 and they are the same.
Thanks
Memory allocation/de-allocation should be managed using std::unique_ptr
in the Agent::CachedFile
class.
Moving away from new/delete, malloc/free makes memory management consistent and trackable by static code analyzers.
1 psi = 6,894.75 Pa
Currently, we are not testing all the unit conversions. We should add a test for each conversion pair. I would suggest adding a template to reduce code repetition.
I got the following errors when validating the schema 1.4 for sample and current in the latest version
Attribute 'assetType' is required in element .
Annotations of element 'AssetChanged' (see below)
An asset was just modified
Annotations of type 'AssetChangedType' (see below)
An asset was just modified
Error location: MTConnectStreams / Streams / DeviceStream / ComponentStream / Events / AssetChanged
Details
cvc-complex-type.4: Required attribute 'assetType' is missing from element .
cvc-type.3.2: Element is not valid with respect to type definition 'AssetChangedType'.
cvc-elt.5: The element is not valid with respect to the actual type definition 'AssetChangedType'.
cvc-assess-elt.1.1: Strict assessment of element with governing element declaration 'AssetChanged' failed.
To support additions to the model for version 1.5 specifications, will need to parse and generate content in XML and JSON.
We have a lot of cases where it would be beneficial to reference other agents that effect overall systems but aren't part of the original adapter.
Example: A water temperature sensor that directly broadcasts data to its own water temp type agent. If you had two machines that shared this water source and needed to report the temperature with them, the agent's existing device file could reference this child agent, allowing both of the machine agents to report the data when polled. This also allows for easier integration of sensors with built in adapters reporting to purpose built agents to feed data that isn't currently supported. The child agent could also be hit on its own if you needed only its values.
Example: An older CNC machine might be able to report minimal data on spindle and x, y coordinates. I might also want data involving the condition of lubrication and coolant using a small microcontroller adapter into a second agent. A parent agent when polled would then poll both children agents as needed or the children agents would update the parent on changes. This parent agent would represent the overall device.
This could be used further to integrate vibration sensors in older hardware to collect values for machine learning, as well as create edge device that could alert when failures were detected.
Refactor storage for checkpoints. Currently this uses a raw array allocated on the heap. This can be replaced by std::vector
to make things easier.
Currently we can add and update assets. But we cannot remove / Delete an asset via HTTP
Use boost logging module instead of the current dlib rolling logger.
Part of Epic #3
Refactor ChangeObserver to remove dlib
dlib::rmutex
with std::recursive_mutex
dlib::rsignaler
with std::condition_variable_any
This is part of the Epic #3
Refactor memory management in tests
Replace instances of objects being managed on the heap with new
/delete
and convert to std:unique_ptr
Enhance sample query to support the concept of "To=" with a specified # of items.
Instead of "From=X" and "Count=Y" to retrieve Y data items starting from X, we want to find out the history of a particular path up to the sequence num X.
https://<authority>/sample?count=100&to=12345&path=//DataItem[@id='XYZ']
Last 100 data items of 'XYZ' before sequence num 12345
With Git Submodules one has very limited option for downloading (no way to specify specific hash or similar) and building. With ExternalProject from CMake we gain better control on what to download, what to patch and how to build/install.
This would also keep our own project clean from others (e.g. with add_subdirectory we get all definitions of the subprojects if they do not use the proper set_target_XXX() commands which they don't...)
ExternalProject should also ease the planned integration of Boost, I guess.
Any objections?
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.