Can PlotJuggler help with plotting data indexed from multiple clocks? about plotjuggler

[facontidavide]

@ahundt and @Chunting, no need to apolgize !
I am glade that you are both helping suggesting improvements and reporting bugs! If people don't report this kind of issue, I can not make PlotJuggler better on my own.

But I just can't resist the temptation to use silly memes... like this one:

[Chunting]

It works as well as expected. You can close this issue now. Thanks.

[ahundt]

cc @Chunting

[facontidavide]

Let's see if I can help you...

1. dump the data into a rosbag
   if you can do it easily, that might be a shortcut.

2. dump to CSV (must every csv entry be occupied or can some be blank?)
   Yes, every entry must be occupied :(

3. write a custom plugin
   it is not terribly hard, but time consuming for sure. I can help you if you want.

4. another alternative you might advise (is there a way to load json?)
   Nope. Options 1, 2, 3 are your best shot

Let me think about the other problem (comparing clocks) to see if I can figure out a solution :)

[ahundt]

Cool, thanks! That gives me some direction. I can start with just CSV with a "base" clock + differences. There will be some discontinuities and the file might be very large because I'll be looking at ~3 minutes data recorded with 1-4ms timesteps, but it would get me started.

[facontidavide]

If I correctly understood, you have to load a single log file, which contains all the information.

Looking at the flatbuffer description, few things come to my mind:

Keep in mind that a time series in PlotJuggler is identified by a name (a single string) and a sequence of [time,value] pairs of doubles.
There is no way to plot a string. I am sayind that because some of the fields of timeEvent are string, such as device_clock_id, event_name and local_clock_id.

Therefore, I expect the name of the time serie to be a combination of device_clock_id, event_name and local_clock_id AND the field name you want to plot (for instance device_time).

Do not forget that you also need to specify the X value (time axis) of the series; it is not clear to me which one you want to use.

In an ideal world what do you think that the plot name, the X axis and Y axis of a plot should be?

[ahundt]

The strings are basically a "rostopic" string so the different devices can be separated from each other in the message. In a CSV that will simply be the title.

What I might be able to do something like the following pseudocode at each time step to write out the csv:

# on the first message save local_request_time as the initial_local_request_time
# on the first message save device_time as the initial_device_time
X = local_request_time
local_request_offset = (local_request_time - initial_local_request_time)
current_device_offset = (device_time - initial_device_time)
Y = current_device_offset - local_request_offset

Then Y should drift around based on how the clocks change relative to each other without any counting to infinity, unless the clocks are actually drifting that way.

[ahundt]

Ok we've been able to plot the data via CSV by plotting all numbers as offsets relative to a single clock as described in my post above. This approach looks like it has great potential but we have run into a problem.

We started by simply merging the data from the three separate devices into one CSV, repeating previous values when there is no data from one time source but there is from the other. This led to the appearance that all the changes or no changes happened over very short periods of time, and thus there are massive discontinuities in the data.

For that reason, another approach is necessary. Is it possible to load two separate csv files and plot the graphs together in a single time frame? That has potential to do the trick.

[facontidavide]

The problem is that loading two files with the same field names is not possible, in theory.
To fix that, I just added a new feature to the Dialog that is showed when a rosbag is loaded.

I can add it also to the CSV plugin, that will take some more work

[Chunting]

This is the time offset we ploted via CSV.

[facontidavide]

is that... good news or bad news? I don't know what to think...
Is the Y axis scale nanoseconds?

[ahundt]

Unfortunately, that's bad news, the giant vertical jumps in the green bar are due to those discontinuities I was mentioning, rather than the actual data. Just trying to demonstrate the issue we're trying to resolve by loading two separate CSVs with different time steps from a single clock. We can give you sample CSVs if you'd like.

[facontidavide]

This commit might help a little. 0f2d09b

now it is possible to parse CSV files with empty cells

[ahundt]

cool that might do the trick! we will give it a try

[facontidavide]

Is there anything else I can do for you? Can we close this issue?

[Chunting]

Thank you. If some cell in the first line is empty, PlotJuggler can’t recognize it and ignore it. I need to fill it out by zero manually, can you fix it? It can’t recognize FT_local_request_time, FT_device_time_offset and Y_FT. Also could you please give more information about this error? “DataLoadCSV: problem parsing the first two lines”, sometimes I cannot plot data because of this error. Thanks Chunting

…

Sent from my iPhone

On Feb 14, 2018, at 8:27 AM, Davide Faconti ***@***.***> wrote: Is there anything else I can do for you? Can we close this issue? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.

[ahundt]

Thank you for the update! It is mostly working but as @Chunting mentioned, there seems to be minor bugs remaining for this use case.

@Chunting Would you mind actually uploading a CSV file to this github issue which demonstrates the problematic behaviors so he can actually reproduce and fix it? You can do it directly from this issue's page on the github website, since it appears you're using the email interface.

@facontidavide I wanted to thank you for this project it is turning out to work terrifically for us, especially with how easy it is to load up and see data at both a high level and at the finest levels of detail, plus it has a very responsive interface!

[facontidavide]

@Chunting
Also could you please give more information about this error? “DataLoadCSV: problem parsing the first two lines”, sometimes I cannot plot data because of this error.

"Luke

A sample file to reproduce the error would help

[Chunting]

Thanks, there was a bug in my code and I fixed it.
But how about this problem?
If some cell in the first line is empty, PlotJuggler can’t recognize it or ignore it. I need to fill it out by zero manually, can you fix it? It can’t recognize kuka_local_request_time, kuka_device_time_offset and Y_kuka.

Thanks,
Chunting

[facontidavide]

I just made a commit that might solve your issue.

please, if the problem persist, SEND ME the file. If it is solved, let me know

[Chunting]

Here is the CSV file.
2018_02_15_15_47_34_FTKUKA.csv.zip

Thanks.

[facontidavide]

With the last modifications I made yesterday, the file you sent loads just fine. close the issue if you can confirm on your side that it works

[ahundt]

@facontidavide My apologies for all the trouble here, this is (to my knowledge) @Chunting's first time trying to provide all the information necessary to reproduce a bug to someone on a public project whom he hasn't met. I recall it was tricky my first time too so I appreciate your patience!

I'll talk to him and see if there was still an issue he is able to reproduce or if that's it, I think there might have been when the CSV starts with blanks on the first line.

Thanks a million for your help!