Exception when recovering form Expansion Hub coms reset about revextensions2

[Windwoes]

@gearsincorg

Hmmmm.... I have a hunch as to what's causing this. When an Expansion Hub DCs, the SDK hot-swaps in a "fake" object so that your user code doesn't crash. When sending a bulk command to the expansion hub, RE2 does a check to make sure you actually sent it to the Hub you meant to:

    /***
     * Checks to see whether a Lynx controller is attached to the Lynx
     * module that this packet came from
     *
     * @param controller the controller to validate
     *
     * @return whether or not the controller is attached to the Lynx
     *         module that this packet came from
     */
    private boolean validateLynxController(LynxController controller)
    {
        return validateLynxModule(new OpenLynxController(controller).getLynxModule());
    }

Notice that the constructor of OpenLynxController is invoked:

    OpenLynxController(LynxController controller)
    {
        // We can get the underlying LynxModule object through a
        // LynxController object, but only through reflection.
        Method getModule_method;

        try
        {
            // The "getModule" method is located within the LynxController class
            getModule_method = LynxController.class.getDeclaredMethod("getModule");

            // Ensures the method is accessible for the next line. We still catch
            // the (impossible) IllegalAccessException just to be safe.
            getModule_method.setAccessible(true);

            // Actually get the value from the controller that was passed in.
            lynxModule = (LynxModule) getModule_method.invoke(controller);
            enhancedLynxModule = new ExpansionHubEx(lynxModule);
        }
        catch (Exception e)
        {
            throw new RE2Exception("Failed to reflect on LynxController!");
        }
}

So I think what's happening is that the reflection is failing on the fake object, or when the fake object is swapped for the real one again after the Expansion Hub re-connects.

Thoughts?

[Windwoes]

@gearsincorg does this issue occur if you just manually unplug the USB wire to the Hub and then replug it? I want to be able to reproduce this myself.

[gearsincorg]

Sorry for the late reply, I was prepping for MD State.

I have not seen this before, and I would assume I would have if it was a simple USB disconnect issue.

FWIW I have found that if the Parent Rev Hub times out, when talking to the Child Hub, the Bulk Data read returns a null pointer. I have protected against this, by just reusing the last reply, but I wonder if this worse exception is also related to communicating with the Child hub...

I will try specifically testing the "momentary disconnected USB" scenario. To see if I can trigger the exception.

Phil.

[Windwoes]

@gearsincorg

FWIW I have found that if the Parent Rev Hub times out, when talking to the Child Hub, the Bulk Data read returns a null pointer.

Well, I know what causes that:

    /***
     * Grab a bunch of useful data from the Lynx module in one go
     *
     * @return an object that contains a bunch of useful data
     */
    public synchronized RevBulkData getBulkInputData()
    {
        LynxGetBulkInputDataCommand command = new LynxGetBulkInputDataCommand(expansionHub);

        try
        {
            LynxGetBulkInputDataResponse response = command.sendReceive();
            return new RevBulkData(response, expansionHub);
        }

        catch (Exception e)
        {
            handleException(e);
        }

        return null;
}

What would you suggest instead of returning null? Making an empty object?

[Windwoes]

Hmm, regarding my question of what to return instead of null, it seems the SDK's convention is just to return empty/zero. For instance:

    @Override public synchronized int getMotorCurrentPosition(int motor)
        {
        this.validateMotor(motor); motor -= apiMotorFirst;
        LynxGetMotorEncoderPositionCommand command = new LynxGetMotorEncoderPositionCommand(this.getModule(), motor);
        try {
            LynxGetMotorEncoderPositionResponse response = command.sendReceive();
            return response.getPosition();
            }
        catch (InterruptedException|RuntimeException|LynxNackException e)
            {
            handleException(e);
            }
        return LynxUsbUtil.makePlaceholderValue(0);
        }

[gearsincorg]

Once I knew this could happen, the logical action for the app to take was to ignore it, and use the last set of vales that was returned.

Should the call do this for us? It would be safer most of the time, but it might mask an underlying hardware problem.

If you aren't going to return the last known values, then returning any other sensor value (eg zero) would be a bad idea. In this case, returning a null is a reasonable way to alert the user there is a problem, and they need to handle it.

Of course I discovered the problem when I tried to access the actual encoder value using the null bulk data object.

[Windwoes]

@gearsincorg

If you aren't going to return the last known values, then returning any other sensor value (eg zero) would be a bad idea. In this case, returning a null is a reasonable way to alert the user there is a problem, and they need to handle it.

I think I agree with that - but the SDK currently does not follow that convention - and I'd like to be consistent with the SDK..... perhaps you could suggest to Bob / Tom to change the SDK's convention?

[gearsincorg]

I'm not sure what I'd be suggesting actually... (I'm not that savvy with the lower levels of the SDK)

My original post was just letting you know about an anomaly I encountered. At this point, I'm just looking to recover from REVExtensions2 throwing an exception if I get a momentary (rareish) comms loss.

[Windwoes]

OK. I'll mess around with my robot today and see if I can reproduce the issue you're seeing.

[Windwoes]

@gearsincorg until I fix this, perhaps just wrapping the call in try/catch would be a temporary workaround for you?

[gearsincorg]

Yeah... that's kinda where I came into this issue. I was looking for a recommendation as to how to recover from the exception.

ie: What is a reasonable recovery action to be taken in the catch? Is there any value in just ignoring the error that one time, and hoping for it to recover, or do some objects need to be recreated?

[Windwoes]

Hmm. That is an interesting question regarding object re-instantiation, because if the SDK recreates the LynxModule object that the ExpansionHubEx object holds a reference to, then even if the SDK recovers, any RE2 calls would still fail. Hopefully I will be able to do some testing next week and find some answers...

[Windwoes]

@gearsincorg I did some testing today and I was not able to reproduce the issue you're seeing :( it seems to survive any type of disconnect whether it be USB, power, or 485. Speaking of 485, I did find a bug in the SDK actually.... if the 485 wire is disconnected for more than the 2.5s timeout and then reconnected while the opmode is running, the Hub communicates properly with the SDK again, but the SDK does not re-send the LED sequence command, so although it is working properly it shows the "disconnected" LED pattern, lol.

[gearsincorg]

I guess the good news is that it appears to be a low probability error, although the bad news is that it's still out there, waiting to throw an exception.

If it is just a transient problem, then I suppose catching it may keep the App running.

I appreciate your help.

[Windwoes]

Do you have any info on what the circumstances were up to the point where the exception was thrown? (e.g. brown-out from 12v, ESD)?

I'd really like to get to the bottom of this because I'm going to be running RE2 on our bot in Detroit....

[gearsincorg]

History:

The error occurred while deploying from the lander during driver practice in our facility.
Our hardware and electrical is pretty robust, with good cable restraint and shielding for the wiring.

Recently however, (days prior) we had been getting occasional issues from RE2, but they were less disruptive as they were easily detected (readBulkData returning a null object). These errors had been coinciding with latencies caused by coms timeouts between the two expansion hubs. We had decided that we were getting intermittent issues on the RS485 bus, possibly due to loose connector terminals. We had essentially eliminated the problem by running a pair of redundant cables between the two hubs using both of the RS485 terminals on each hub (the RS485 terminals are simply wired in parallel on the PCB).

This wiring change had eliminated the timeouts and so we felt we had fully addresses the issue, until this new exception during Auto.

It has never recurred, but from the log on my first post you can see that it definitely coincides with a recovering coms disruption. Possibly ESD, or just simple mechanical.

[Windwoes]

Hmm, interesting. Now, just to throw it out there, in one of my tests where I pulled the 485 wire during a live OpMode, I got a StuckInLoop() error, which seems to me to be the fault of the SDK rather than RE2. I was very careful when implementing all of RE2's commands to follow the exact error-handling structure that the SDK uses.

[Windwoes]

Hmm I think I see what caused the issue. If the isHooked boolean in LynxController is false, then getModule() returns pretendModule, causing the cast to LynxModule in OpenLynxController to fail....

[Windwoes]

Which then begs the question.... why wouldn't isHooked be false when a DC occurs?

[Windwoes]

@gearsincorg Ok I'm 99% sure I've fixed the possibility for this bug to ever happen in this commit.

[gearsincorg]

OK, I'll Load this in the first chance I get. It may be a while...

[Windwoes]

@gearsincorg Well, it will be difficult for you to test it out until I push a new release to Bintray, but if you feel so inclined you could test it by manually copying in the source code files.

[Windwoes]

I'm going to try to push a new release to Bintray shortly, though.

[Windwoes]

v1.2 has been released, which includes the fix for this issue