tbohne / msc

Master's Thesis at Osnabrueck University / DFKI

Execution Monitoring for Long-Term Autonomous Plant Observation with a Mobile Robot

Long-term autonomous robotic systems in highly dynamic real-world applications are becoming a reality. Nonetheless, ensuring a certain level of robustness for these systems is a major challenge, as it is impossible to anticipate all potential situations that a robot might encounter. A joint consistent integration of both monitoring and acting is vital, as plans do not always work out as expected when faced with the chaotic nature of reality. Consequently, execution monitoring techniques that address unexpected problematic situations by means of recovery mechanisms must be an essential component of robot architectures. This thesis attempts to reduce the barriers towards long-term autonomy of mobile outdoor robots by identifying and classifying key difficulties in such a context and developing a fully integrated monitoring and resolution framework capable of overcoming some of the typical limitations for those systems. Experimental evaluation of the proposed framework in a simulation environment indicates a drastically improved resilience with respect to the identified challenges.

• Thesis (PDF)
• OSI Slides (PDF)
• Execution Monitoring Framework
• Plan Generation Node

msc's Issues

Potentially integrated solutions

No hard requirements, but nice-to-have:

• battery watchdog module
• docking solution (autonomous energy supply)
• working planner
• If possible, it would be good to include real weather data form the region of the test field

Zusammenfassung / Skizze des Themas

Achten Sie drauf, dass Sie eine Zusammenfassung/Skizze des Themas und die am Ende des ersten Kapitels zu formulierende Fragestellung der Arbeit permanent weiter pflegen und im Kopf behalten (um am Ende der Arbeit in Zusammenfassung schreiben zu können, welche Ziele Sie womit und inwieweit erreicht haben und wenn welche nicht, dann warum nicht). Das klingt hier schon ein wenig an, aber halten Sie das bitte weiter durch.

Transfer relevant content of exposé to actual thesis (introduction etc.)

Plan executioner - do we have that?

if so, what is it able to do?

Plan Execution and Monitoring

• that would assume that there is a planning procedure - there isn't at the moment, right?
• of course I could just assume that there is one and just say "here I would replan"

Literature check: Pause and Resume Plan Execution

Is there a solution for something like that? is it hard? just execute partial (remaining) plan afterwards?

[...] the plan execution has to be interrupted and the robot needs to be able to somehow save the current state of the plan execution and continue precisely with that state after the reason for the interruption has been resolved.

Go explicitly through [What's in a Thesis Exposé?](https://kbs.informatik.uos.de/info/expose/)

Dynamic Environment

• mention that the considered scenario has a very dynamic environment (not static)

Describe plan generation + format in thesis

• I did not just use it as anticipated, but build it myself -> describe it briefly

Additional problem for LTA: Bad Localization

• that's a problem that I've even witnessed myself

Visualize Architecture of State Machines etc.

• how the processes interlock, the individual state machines, etc.
• also a specific example -> what happens when the plan is paused in a certain state?

Think about how to implement some of the LTA problems in Gazebo and implement them

• evaluate behavior
• check recoveries etc.
• good first attempt on one of the problems towards LTA

Ergänzung für die Liste

"Roboter fällt um"

Additional problem for LTA: incorrect rotation

• would be good to be able to detect that somehow
• e.g. because it's muddy and since the orientation is incorrect, also the sensor recordings are
• solution: reset map and drive ~2m back and forth or initially that the operator has to do something about it

'Mobile' + 'Autonomous'

Ich sollte nicht nur 'long-term' präzise definieren, sondern auch, was ich im Kontext der Arbeit unter diesen beiden Begriffen verstehe.

Plan Generation Node

• write plan generation node that is able to take a geojson file as input and creates a plan from that

Record videos that demonstrate the process in the simulation

• initially, the failure in problematic situations
• subsequently, the response to such situations

Check out concurrency SMACH

Architecture

• hierarchically structured state machines? parallel execution monitoring? -> read about SMACH concurrency state machine
• monitoring is essentially a parallel function -> you want to intervene during the execution of some other process
• of course it's possible to do that sequentially -> check between steps of the procedure, still everything ok?
• however, parallel seems to more natural here since something can go wrong any time
• the state machine can cause a state transition from another process (independent of the current state)
• Concurrency SMACH seems to be exactly what you need for high-level monitoring

Do I want to consider changing environment conditions / vegetation periods?

[...] those scans with changing environment conditions etc

extend the simulation? , e.g. with weather conditions

Implement option to pause and resume plans (manually for starters)

Implement option to manually say: 'Ok, now a problematic situation occurred, stop plan execution, drive back to base and save where you stopped.'

Assumptions of the work

Guru's plan (part of what I'm assuming / building upon in my work):

Deliverables:

• Gazebo world / spawn script with
  • Robot
  • Container
  • Virtual Charging Station
  • Regions / Objects of interest
• Documentation where to find / how to install
• Battery charger node which charges the battery whenever the robot is at a certain position or in vicinity of some object
• scheduler node which tells the robot to drive to n regions of interest and take a scan
• battery watchdog node which aborts the mission if battery voltage gets critically low
• integrated version of state machine "arox engine"

Scenario A: (Battery is enough)
1 Robot charges up to threshold
2 Robot drives to first roi
3 Robot scans
4 Repeat 2 & 3 according to user input
5 Robot completes last roi
6 Robot returns to charger, GOTO 1

Scenario B: (Battery is not enough)
1 Robot charges up to threshold
2 Robot drives to first roi
3 Robot scans
4 Repeat 2 & 3 according to user input
5 At some point battery watchdog kicks in
6 Robot returns to charger
7 Robot charges up to threshold
8 Robot resumes mission
9 Robot completes last roi
10 Robot returns to charger, GOTO 1

Execution Monitoring

• not only detecting (monitoring) such situations, but also classifying them - what kind of problem is it?
• -> classifying problematic situations
• how far do you push it and what kind of things do you deal with?
  • classify objects that are in the way based on image recognition algos?
  • detect sensor problems based on anomalies in the data

Incorporate exposé feedback into final work

Introduction for Execution Monitoring Section

quote GNT

Increase Langsenkamp simulation

The Langsenkamp simulation should probably be increased, a larger excerpt has to be considered - just a question of the world.

Expected scientific and/or technological contribution

• check 'future works should investigate' sections in the literature
• read literature (in particular the LTA survey) - really no such systems in literature?

Docking as potential extension in the end

• the docking solution should be an optional extension
• of course, the integrated system is more impressive with more integrated solutions (working docking would be cool)
• but work should not depend on it
• will start with Guru's setup of loading areas (simply certain coordinates)

Exposé Feedback - LTA

Zum Begriff Langzeitautonomie:Gut erkannt und bereits gut andiskutiert, was das im allgemeinen und für Agrarroboter im Besonderen bedeuten kann oder soll. Dazu noch wenige Gedanken, die das vielleicht eingrenzen/schärfen können:

• Wie lang ist Langzeit? Wenn man von den Anwendungsprozessen her denkt, dann scheint mir eine Obergrenze langzeitautonomer Episoden darin zu liegen, wann ein Mensch eh mal wieder physisch zum Roboter gehen wollen würde. Auf dem Mars entfällt das komplett und in der Tiefsee wäre es nachdem der Roboter fünf Monate lang unterm Schelfeis durchgetaucht wäre. In der Landwirtschaft würde ich vermuten, dass der Betreiber in den meisten Fällen kürzere Serviceintervalle haben will, wo er den Roboter eh mal wieder anguckt, die Kamera- und Scannerlinsen putzt und die Reifen aufpumpt, und dann kommt eine neue Episode. Das wäre für mich die sinnvolle Obergrenze von "Langzeit" im vorliegenden Prozess. Das übergreift natürlich bei weitem einzelne Ladezyklen, völlig richtig. Es ist aber andererseits auch nicht beliebig lang. Und es ist auch nicht überraschend, sondern auch das sollte der Roboter für seine eigene Planung wissen. ("Kleine Ausfälle schicke ich dem Meister per Email als geplanten Arbeitspunkt für den nächsten Boxenstop.")
• Grenzen von Langzeitautonomie. Langzeitautonomie ist nicht Zauberei. Grenzen liegen ausdrücklich in unvertretbaren und unvorhersehbaren äußeren Einflüssen. Das kam mir in Ihren Überlegungen ein wenig zu schwach heraus. Wenn die Akkus in Flammen aufgehen oder das Wildschwein vom Wald kommt und den Roboter umpflügt, dann ist ganz einfach Ende. Soweit man solche Ereignisse zumindest im Sinne von Wahrscheinlichkeit vorhersehen kann (Batterie beginnt schon vorm Spontanbrand komisches Entladeverhalten zu zeigen; externe Information über Wildschweine in der Region), könnte der Roboter das in einer Statusmeldung an den Betreiber schicken.
• Autonomie-Level. Nicht nur Langzeitautonomie, auch schon Autonomie/Vollautomation ist ein unpräzise definierter Begriff. Es muss nicht so sein, dass da komplett keine Verbindung zu einem Operator besteht. Gerade aus der Raumfahrt kommt das Konzept der traded control, weil man dem Roboter/Satelliten gelegentlich schon immer mal wieder etwas auf hoher Abstraktionsebene sagen können will. Das können wir für langzeitautonome Agrarroboter gern auch nicht wollen (weiß nicht), aber es würde im Sinne von Marsrobotern dem Konzept von Langzeitautonomie überhaupt nicht widersprechen. Ich hänge Ihnen mal einen Text an, den ich mal für Nicht-Techniker (Juristen) geschrieben habe (Joachim Hertzberg. Technische Gestaltungsoptionen für autonom agierende Komponenten und Systeme. in: Das Recht vor den Herausforderungen der modernen Technik. Beiträge der 1. Würzburger Tagung zu Technikrecht im November 2013 (E. Hilgendorf, S. Hötitzsch, eds.). Baden-Baden (Nomos) 2015, p. 63-73). Da wird das ein bisschen aufgedröselt. Da durfte ich nur wenige Referenzen machen; wenn man nach shared/traded control in der Robotik sucht, gibts da noch mehr. Ich will Ihnen da nicht noch eine Zusatzbaustelle vermitteln, aber drauf hinweisen, dass beim Begriff "Langzeitautonomie" nicht nur "Langzeit", sondern auch "Autonomie" (oder bescheidener Vollautomation) sinnvoll Varianten von Bedeutung haben kann -- sinnvoll hinsichtlich der Frage, was sich Anwender für ihre Prozesse eigentlich wünschen.

Scientific Background - Justification for my Approaches

Are there papers mentioning such challenges that I want to tackle?

List of issues for LTA - three classes of potential failures

- perhaps possible to get the whole list of potential barriers into type 2 as goal of the work?
- such that robot is always able to call for help
- at least for all the barriers that I am aware of - that I consider in the work
- could be interesting -> many monitoring nodes:
- gps monitoring -> no signal -> call for help
- sensor (RIEGL) monitoring -> no scan / outdated scan -> call for help
- etc.

• and then take 1-3 that are even classified type 1 afterwards (robot can resolve issue by itself)

Perhaps to be considered later

In addition to a monitoring system capable of detecting a subset of the issues identified in section \ref{sec:challenges_for_lta},
an online monitoring capable of detecting deviations of planning parameters on a continuous or timed basis and triggering replanning if necessary would be very valuable for practical applications.

Is that going to be part of the work?

Potential challenge:

Based on the growth state of the plants, the planning has to be adapted, which has to be evaluated in the simulation, because a long-term test that reflects seasonal changes would clearly be beyond the scope of this work. So, besides the need for a simulation in terms of potential hardware-related or other hurdles that prevent practical testing, we have another motivation for the simulation, which are seasonal changes that are needed in order to evaluate the different planning strategies. From a certain date, the plants are too high to drive above them and the scans have to be taken from a different spot, the whole strategy changes. When the plants are small enough, the scans are taken from above, when they become too big, that's no longer possible. That is a typical constraint in agricultural projects - the vegetation periods. In March, when this work is expected to end, the plants are going to be relatively small such that the robot can drive above them when taking scans, in the end of spring and summer that would no longer be possible and will be reflected in the simulation.

Interrupt MBF execution

• kill all running goals instantly -> atm I have to wait for an MBF goal when a contingency / catastrophe is being triggered

Monitoring for sensor failures

A fairly trivial approach to recognize sensor failures would be to just check whether there are still messages arriving on the corresponding topic:

• trivial, but still not yet there and needed..
• could fake that -> publish on some fake RIEGL topic and stop at some point -> then recognize that, transition to CONTINGENCY and call for help
• I could just write a fake scanner node that does that and that is actually executed when executing a scan action..
  - since the RIEGL does not seem to be part of the simulation yet, I could just assume that the scans are taken using the Velodyne
• could just write a node that republishes the Velodyne input and is able to interrupt it such that I can simulate a sensor failure
  - I'd have to demonstrate that my prototypical plan is no longer being correctly executed when I have these failures and then I detect it, call the operator, he solves it and I go ahead..

Contribution statement

etwas in dieser Art irgendwo einbauen:

"To the best of my knowledge, no such demonstration of a fully integrated solution for the described scenario has been proposed in the literature."

Exposé approach - make a rough sketch of where it can go

• what has to be done in order to have a basic running scenario in simulation (minimal example) that I can work with
• think about which problems I would like to tackle very specifically
• after simple setup is running: I have to deal more with what the problems are when it works, so these edge cases, so to speak, and how to solve them and how to react to them
• work in and estimate what is how difficult, that is the work I am doing right now in the form of the exposé, figuring that out is my first step - what kind of work should it be, not only what is great, but also what is possible?
• what do I need - how much work is it? - then adjust plan -> iteratively - then exposé is ready
• result of the exposé process is a relatively good plan -> idea of what is possible and what is not
• sketch rough system -> what should run at the end -> state graph
• I shouldn't solve all the discussed problems, but pick 1-3 interesting and doable ones

Exposé title page in german

• although I write it in English

tex or bibtex can't deal with certain characters in the bibliography - to be fixed

Identify / look for more challenges for LTA

Text block that might be used somewhere

Obviously, it is out of the scope of this work to generally solve the problem of long-term autonomous plant monitoring with mobile robots. Therefore, the work is going to be based upon a list of assumptions under which the developed integrated solution is expected to work.

Links to source code etc. of thesis

something like this:

High-Level State Machine - Architecture

• normal operation: executing a given plan / idle (waiting for plan)
• problematic
  • contingency: problem detected, e.g. sensor failure, but ability to act remains - drive back to base in safety mode
  • catastrophe:
    - unsolvable problem occurs (not even solvable by the operator), if still possible, save current state, send emergency signal and shutdown
    • also covers full breakdowns where nothing is possible (e.g. battery dead)

Catastrophe and contingency differ in the robot's ability to act. In a catastrophe case it's no longer really able to act.

• e.g. robot falls over (we can even try that)
  • can not recover (move), but is still able to communicate, save the state etc.
• different case -> lightning strikes and robot has a full breakdown and is not able to do anything
• Boundaries not 100% sharp -> there are edge cases

Battery-Example:

• normal operation: battery discharges faster than expected, e.g. because of low temperatures -> replanning + execution of adapted plan
• problematic:
  • contingency: battery too low already, watchdog triggers -> have to drive back to base
  • catastrophe: battery so low that I can already tell that it's not possible to reach the base
    • can still communicate the problem and save the state, but that's it (cannot recover)
    • if the battery just breaks down suddenly, nothing can be done - full breakdown

Incorporate Exposé Feedback (Benjamin)

see lab notes...

Thesis opening idea

begin with Moravec quote:
- Moravec's paradox
- roughly: The hard things are the things that are easy for human children

Autonomie-Spektrum

• Bsp. für Punkte auf dem Spektrum:
  • Teleoperation (Mensch steuert Maschine vollständig aus der Entfernung), e.g. Bombenentschärfung
  • shared control: Mensch startet autonome Funktion und kann diese jederzeit unterbrechen, e.g. Einparkautomatik
  • traded control: Mensch startet autonome Funktion, die bis zum Ende durchläuft, danach geht die Kontrolle zurück an den Mensche, e.g. Marsrover Opportunity
  • Vollautomatisierung / Autonomie: langfristig vorgegebene Ziele, Maschine arbeitet dauerhaft in geschlossener Regelung in Abhängigkeit von Sensordaten aus
    ihrer Umgebung; Regelung wird nur unterbrochen und an den Operatur zurückgegeben, falls die Maschine einen Fehler feststellt

Autonomous field monitoring data acquisition cycle

Create something like this for my scenario:

Which sensors to use for scanning

• based on the growth state of the plants, different sensors are required to record the relevant data
• the plan, i.e. the decision of which parcel is to be processed next, is based on a list of criteria
  • the primary criterion is the growth stage of the plants
  • there is a list of particular features that are supposed to be detected based on the current growth state
  • at the moment, this information is communicated manually, but in the future it is planned to have that automated
  • It is well traceable in time when a certain feature is to be expected such that this process could be time-triggered

Example with SMACH visualization - Pause Plan + Resume

• specific example -> what happens when the plan is paused in a certain state?

Implement monitoring node

The high-level parent SMACH is essentially not the monitoring itself, but the handling - the reaction to the monitoring, right?
I could just implement a monitoring state (node later) that publishes on a certain topic when something has been detected.

• should probably run parallel to the parent SMACH
• the idea is to be able to interrupt the NORMAL_OPERATION

Do I want to assume perfect weather conditions for my work?

The robot's mission to process the field should not start at random times, but based on certain criteria, e.g. weather conditions.
Especially when considering hyperspectral data, it is important that it is neither too sunny, nor too cloudy, ideally it should be grey and foggy with uniform illumination. Of course, it's not trivial to predict the weather.