stanleybak / aerobenchvvpython

Python Version of F-16 Maneuver Verification Benchmark

Note: This is the v2 branch of the code, which is now a python3 project and includes more modularity and general simulation capabilities. For the original benchmark paper version see the v1 branch.

AeroBenchVVPython Overview

This project contains a python version of models and controllers that test automated aircraft maneuvers by performing simulations. The hope is to provide a benchmark to motivate better verification and analysis methods, working beyond models based on Dubins car dynamics, towards the sorts of models used in aerospace engineering. Roughly speaking, the dynamics are nonlinear, have about 10-20 dimensions (continuous state variables), and hybrid in the sense of discontinuous ODEs, but not with jumps in the state.

This is a python port of the original matlab version, which can can see for more information: https://github.com/pheidlauf/AeroBenchVV

Citation

For citation purposes, please use: "Verification Challenges in F-16 Ground Collision Avoidance and Other Automated Maneuvers", P. Heidlauf, A. Collins, M. Bolender, S. Bak, 5th International Workshop on Applied Verification for Continuous and Hybrid Systems (ARCH 2018)

Required Libraries

The following Python libraries are required (can be installed using sudo pip install <library>):

numpy - for matrix operations

scipy - for simulation / numerical integration (RK45) and trim condition optimization

matplotlib - for animation / plotting (requires ffmpeg for .mp4 output or imagemagick for .gif)

slycot - for control design (not needed for simulation)

control - for control design (not needed for simulation)

Release Documentation

Distribution A: Approved for Public Release (88ABW-2020-2188) (changes in this version)

Distribution A: Approved for Public Release (88ABW-2017-6379) (v1)

aerobenchvvpython's Issues

AttributeError: 'list' object has no attribute 'shape'

Thanks, Dr. Bak for publishing this great simulation and experimentation framework as open source so that the community can use it. I am attempting to generate some of the example videos/images and I am having some trouble. I have slimmed down the run_all.sh file and created a run_gcash.sh file whose contents are:

#!/bin/bash

time python3 run_GCAS_anim3d.py gcas.mp4
time python3 run_GCAS_anim3d.py gcas.gif

When I run these commands, I get this result:

$ ./run_gcas.sh 
saving result to 'gcas.mp4'
GCAS transition waiting -> roll at time 4.999999999999992
GCAS transition roll -> pull at time 6.233333333333321
GCAS transition pull -> standby at time 11.833333333333302
Simulation Completed in 0.35 seconds

Saving 'gcas.mp4' at 40.00 fps using ffmpeg with codec 'libx264'.
Frame: 0/451 - Index 0 frame 0/451
Traceback (most recent call last):
  File "/home/hawkinsw/dependable/AeroBenchVVPython/code/aerobench/visualize/anim3d.py", line 359, in make_anim
    anim_obj.save(filename, fps=fps, extra_args=extra_args)
  File "/home/hawkinsw/.local/lib/python3.8/site-packages/matplotlib/animation.py", line 1128, in save
    anim._init_draw()  # Clear the initial frame
  File "/home/hawkinsw/.local/lib/python3.8/site-packages/matplotlib/animation.py", line 1703, in _init_draw
    self._draw_frame(next(self.new_frame_seq()))
  File "/home/hawkinsw/.local/lib/python3.8/site-packages/matplotlib/animation.py", line 1726, in _draw_frame
    self._drawn_artists = self._func(framedata, *self._args)
  File "/home/hawkinsw/dependable/AeroBenchVVPython/code/aerobench/visualize/anim3d.py", line 325, in anim_func
    trail_line.set_3d_properties(pos_zs[start_index:frame])
  File "/home/hawkinsw/.local/lib/python3.8/site-packages/mpl_toolkits/mplot3d/art3d.py", line 143, in set_3d_properties
    zs = np.broadcast_to(zs, xs.shape)
AttributeError: 'list' object has no attribute 'shape'

I was just wondering if you ever saw anything like that and whether you could help me debug.

Thank you again for the implementation and its publication as OS.

Will

Cannot locate

In v1 there are checks in the pass/fail automaton that guarantee airspeed, alpha, beta, etc are within proper limits:

        @abc.abstractmethod
        def advance(self, t, x_f16, autopilot_state, xd, u, Nz, ps, Ny_r):
        '''advance the automaton state based on the current aircraft state'''

        airspeed = x_f16[0]
        self.check('airspeed', t, airspeed, self.flightLimits.vMin, self.flightLimits.vMax)

        alpha = rad2deg(x_f16[1])
        self.check('alpha', t, alpha, self.flightLimits.alphaMinDeg, self.flightLimits.alphaMaxDeg)

        beta = rad2deg(x_f16[2])
        self.check('beta', t, beta, -self.flightLimits.betaMaxDeg, self.flightLimits.betaMaxDeg)

        self.check('Nz', t, Nz, self.flightLimits.NzMin, self.flightLimits.NzMax)

        altitude = x_f16[11]
        self.check('altitude', t, altitude, self.flightLimits.altitudeMin, self.flightLimits.altitudeMax)

However, in v2, I can find only that Nz is checked

        def get_checked_u_ref(self, t, x_f16):
        '''
        for the current discrete state, get the reference inputs signals and check them against ctrl limits
        '''

        rv = np.array(self.get_u_ref(t, x_f16), dtype=float)

        assert rv.size % 4 == 0, "get_u_ref should return Nz, ps, Ny_r, throttle for each aircraft"

        for i in range(rv.size //4):
            Nz, _ps, _Ny_r, _throttle = rv[4*i:4*(i+1)]

            l, u = self.llc.ctrlLimits.NzMin, self.llc.ctrlLimits.NzMax
            assert l <= Nz <= u, f"autopilot commanded invalid Nz ({Nz}). Not in range [{l}, {u}]"

        return rv

I am obviously missing something, but would really appreciate you pointing out my misunderstanding. It seems like in v2 there is no explicit pass/fail automaton and it's all integrated. That would seem to render the different check_*.py files moot. Am I correct in saying that?

Thanks for your help!
Will

Source of the ACAS Xu ONNX Files

Dr. Bak,

I was wondering how you sourced the ONNX files for the ACAS Xu branch? Did you generate those files yourself? Did you get them from another source? Any information you could provide would be incredibly helpful!

Thank you!
Will