robotpy / roborio-packages

Build files for RobotPy ipk packages

RobotPy: Python for FRC cRIO Robot Controller

Author:	Peter Johnson, FRC Team 294
Author:	Dustin Spicuzza, FRC Team 2423/1418
Copyright:	Copyright © 2010, Peter Johnson, Ross Light, Dustin Spicuzza

About RobotPy

RobotPy is a distribution of Python intended to be used for the FIRST Robotics Competition. Teams can use this to write their robot code in Python, a powerful dynamic programming language.

Features

• Python is simple to learn and easy to maintain.
• RobotPy lets you reload code without restarting.
• RobotPy provides access to the WPILib class library.
• You don't need to use WindRiver (unless you're rebuilding RobotPy itself).

Installation for most people

Download the binary releases from our FIRSTForge site, unpack them, and run install.py. Download from: http://firstforge.wpi.edu/sf/projects/robotpy

Installation from Source using our build VM

Using the provided build virtual machine is the easiest way to build RobotPy from source. See build-vm/README.txt for instructions.

Installation from Source (Advanced Users Only)

Step 1: Install Python

Python 2.7.5 is known to work. Ensure that you add your python installation directory to your PATH variable.

Note: This step is required for SIP installation

Step 2: Install SIP v4.15.3:

The cmake build process currently checks for exact versions of SIP, as the generated code changes from release to release.

http://www.riverbankcomputing.com/static/Docs/sip4/installation.html

Note: Installation of SIP on Windows requires Visual Studio or MinGW installed, see the SIP build instructions for more details.

Step 3: Build the source tree

TODO: Update these instructions for Windows too?

The RobotPy build process uses cmake to build the RobotPy distribution. frcmake is a wrapper around cmake that sets the build environment up to build for the cRio.

RobotPy currently uses the GCC 4.8 compiler distributed at the following URL to build the binary distribution. Wind River may be used, but we haven't built it with Wind River, so you may run into trouble using it with frcmake.

http://firstforge.wpi.edu/sf/projects/c--11_toochain

Create a directory (the 'binary directory') outside of the RobotPy source tree, and run this:

$ frcmake /path/to/robotpy

Once you've done this, you can build the code:

$ make

To copy the build output to the 'dist' directory, run this command from your binary directory:

$ make install

To build a binary release, just run the following command and a zip file will be created in your build directory.

$ make package

Step 4: Robot Installation

From the 'dist' directory inside of your binary directory, run install.py

Alternatively, you can FTP the contents of the 'robot' directory to your cRio directly. However, install.py is easier.

Development tools

See the utilities directory for useful development tools that may make your RobotPy development experience easier.

The pyfrc python package is a recommended development package that provides unit testing and other capabilities for your robot code.

https://github.com/robotpy/pyfrc

Technical Overview

RobotPy is a packaging of a patched Python 3.2 interpreter (found in the RobotPy/Python subdirectory of the source code). All access to the WPILib is generated by a SIP interface, which is found in Packages/wpilib/sip/. When the robot is started, it initializes the Python interpreter and runs the file py/boot.py. From there, all responsibility is given to the boot.py script, which is referred to as the bootloader.

If boot.py ever exits (due to an exception, for example), the C++ code exits. The default boot.py simply exits on any user exception. If this happens, you can reboot easily via NetConsole by simply typing "reboot" followed by hitting the enter key. This is how code reloads are performed. As boot.py is written in Python, this behavior can be customized as desired.

Major Differences from standard Python

• Several Python modules with large and/or incompatible dependencies removed, namely: curses, dbm, gdbm, tkinter, nis, ossaudiodev, resource, spwd, syslog, termios, audioop, bz2, crypt, grp, ssl, pwd, and mmap.

Licensing

A brief overview of licensing terms:

• Python: PSF License
• RobotPy (all other support code): MIT License

If you redistribute RobotPy and add other libraries, please include their licensing information here.

RobotPy

Copyright © 2010 Peter Johnson

Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

roborio-packages's Issues

os.getgroups() produces segmentation fault

Trying to execute the os.getgroups function in python produces a segmentation fault.

$ python3
>>> import os
>>> os.getgroups()

This works OK on my ubuntu desktop running python 3.4.1.

Originally filed by @dangyogi at robotpy/robotpy-wpilib#125

python3-dbg package doesn't work

Reading symbols from /usr/local/bin/python3...
warning: the debug information found in "/usr/local/bin/.debug/python3.4" does not match "/usr/local/bin/python3" (CRC mismatch).

A different BLAS?

From @auscompgeek : https://github.com/flame/blis

Add openssl support to python

https://peps.python.org/pep-0644/ requires OpenSSL > 1.1.1... NI only ships 1.0.2.

Rewrite to not use make

The overall approach here is clunky. Make isn't really the right tool, as it's very difficult to do smart things like check for existence of data trees, remote package installs, etc. Maybe this would be better implemented as a (set of?) Python scripts?

pip3 fails on roboRIO

This is very strange to me... any thoughts?

admin@roboRIO-1418:~# pip3 install -v ipython
Downloading/unpacking ipython
  Could not fetch URL https://pypi.python.org/simple/ipython/: connection error: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:600)
  Will skip URL https://pypi.python.org/simple/ipython/ when looking for download links for ipython
  Could not fetch URL https://pypi.python.org/simple/: connection error: HTTPSConnectionPool(host='pypi.python.org', port=443): Max retries exceeded with url: /simple/ (Caused by <class 'http.client.CannotSendRequest'>: Request-sent)
  Will skip URL https://pypi.python.org/simple/ when looking for download links for ipython
  Cannot fetch index base URL https://pypi.python.org/simple/
  Could not fetch URL https://pypi.python.org/simple/ipython/: connection error: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:600)
  Will skip URL https://pypi.python.org/simple/ipython/ when looking for download links for ipython
  Could not find any downloads that satisfy the requirement ipython
Cleaning up...
  Removing temporary dir /var/volatile/tmp/pip_build_admin...
No distributions at all found for ipython
Exception information:
Traceback (most recent call last):
  File "/usr/local/lib/python3.4/site-packages/pip/basecommand.py", line 122, in main
    status = self.run(options, args)
  File "/usr/local/lib/python3.4/site-packages/pip/commands/install.py", line 278, in run
    requirement_set.prepare_files(finder, force_root_egg_info=self.bundle, bundle=self.bundle)
  File "/usr/local/lib/python3.4/site-packages/pip/req.py", line 1177, in prepare_files
    url = finder.find_requirement(req_to_install, upgrade=self.upgrade)
  File "/usr/local/lib/python3.4/site-packages/pip/index.py", line 277, in find_requirement
    raise DistributionNotFound('No distributions at all found for %s' % req)
pip.exceptions.DistributionNotFound: No distributions at all found for ipython

gfortran doesn't seem to be distributed with 2016 opkg repos on RoboRIO

Not sure if there's someone at NI who could be contacted to fix this... I suspect not. This prevents usage of ATLAS, which is a dependency of numpy.

Create an ARM VM to compile these in

It seems like this might not be that hard to do, and would allow us to build things like OpenCV and not have to mess around with the limited resources that the RoboRIO has on it.

Why is the python3 ipk so big?

Just curious, @PeterJohnson , the python3 ipk is something like 32MB? Given that the windows MSI files are ~15MB or so, and older RobotPy was ~10MB, that seems a bit big. Not that it matters too much (LabVIEW is a 4GB download, after all), but I am curious about it.

gfortran/ATLAS runtime error for 2016

Not totally unexpected, but I figured I'd try it with the existing libraries:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/lib/python3.5/site-packages/numpy/__init__.py", line 180, in <module>
    from . import add_newdocs
  File "/usr/local/lib/python3.5/site-packages/numpy/add_newdocs.py", line 13, in <module>
    from numpy.lib import add_newdoc
  File "/usr/local/lib/python3.5/site-packages/numpy/lib/__init__.py", line 8, in <module>
    from .type_check import *
  File "/usr/local/lib/python3.5/site-packages/numpy/lib/type_check.py", line 11, in <module>
    import numpy.core.numeric as _nx
  File "/usr/local/lib/python3.5/site-packages/numpy/core/__init__.py", line 14, in <module>
    from . import multiarray
ImportError: /usr/local/lib/libatlas.so: undefined symbol: __aeabi_uidiv

There aren't instructions for updating ATLAS/libgfortran, otherwise I'd give it a try. It's probably not that bad though.. right? :p

Tar fails on macOS

Mac's tar does not support --owner or --group

Affects lines 15 & 16 in ipkg/Mk/Makefile:

tar -czvf control.tar.gz --owner=0 --group=0 control ${EXTRA_CONTROL}
cd data && tar -czvf ../data.tar.gz --owner=0 --group=0 . && cd ..

Workaround:

1. Install homebrew
2. Install gnu-tar
3. replace tar with gtar

Adding Python 3.5

If I were to try and add python 3.5 here, what would that involve? Would it be better to add a seperate python35 package, or to upgrade the existing python3?

Update Python for 2018

python3-dev error

admin@roboRIO-1418:~# opkg install python3-dev   
Installing python3-dev (3.4.2) to root...
Downloading http://www.tortall.net/~robotpy/feeds/2014/python3-dev_3.4.2_armv7a-vfp-neon.ipk.
Collected errors:
 * check_data_file_clashes: Package python3-dev wants to install file /usr/local/include/python3.4m/pyconfig.h
    But that file is already provided by package  * python3
 * opkg_install_cmd: Cannot install package python3-dev.

Build new packages for 2019

• Replace python 3.6 with 3.7
• I think there's a GCC upgrade, so we'll probably need an ATLAS rebuild...

Build ptvsd and pydevd with Cython extensions

See-also: robotpy/robotpy-wpilib#99

Build a robotpy-gdb package for on-robot debugging?

The NI package depends on NI's python3, which might confuse users. Also, robotpy-installer doesn't seem to know how to install it.

Bitbake error

I'm trying to make the cross-compiling of the library libgfortran on a virtual machine in a windows pc, to later build the library on a linux device. Following the guide I have encountered two problems:

• When I execute the command "git cherry-pick de2aa7a56790581406f219339c9022638cd47494", I get the error "fatal: bad object".

• When I execute "bitbake libgfortran", the following error appears:

ERROR: Function failed: do_compile (log file is located at /home/burruzola/nilrt/build/tmp_nilrt_3_0_x64-glibc/work/x86_64-linux/automake-native/1.14.1-r0/temp/log.do_compile.4677)
ERROR: Logfile of failure stored in: /home/burruzola/nilrt/build/tmp_nilrt_3_0_x64-glibc/work/x86_64-linux/automake-native/1.14.1-r0/temp/log.do_compile.4677
Log data follows:
| DEBUG: Executing shell function do_compile
| NOTE: make -j 6
| : && /bin/mkdir -p doc && { PATH='/home/burruzola/nilrt/build/tmp_nilrt_3_0_x64-glibc/work/x86_64-linux/automake-native/1.14.1-r0/build/t/wrap':$PATH && export PATH; } && /usr/bin/perl /home/burruzola/nilrt/build/tmp_nilrt_3_0_x64-glibc/work/x86_64-linux/automake-native/1.14.1-r0/automake-1.14.1/doc/help2man --output=doc/automake-1.14.1 automake-1.14
| help2man: can't get `--help' info from automake-1.14
| Try `--no-discard-stderr' if option outputs to stderr
| Makefile:3707: recipe for target 'doc/automake-1.14.1' failed
| make: *** [doc/automake-1.14.1] Error 255
| ERROR: oe_runmake failed
| WARNING: exit code 1 from a shell command.
| ERROR: Function failed: do_compile

Although a "build" folder is created with a subfolder for the machine inside it, the .ipk files are nowhere to be found as the bitbake command is not fully completed.

Contemplate more automation for this repo

It's still largely a manual process, and there's not an easy way to stage stuff for testing without wiping out the production repo. One potential solution is using docker + my ODROID C2 as a build slave that auto builds on pushes to master (gitlab-ci actually makes this sort of thing easier).

Rename python packages

I propose that:

• python2 becomes python27
• python3 becomes python35

Let's take python35-numpy as an example. Because of the way our ipks work, numpy's files are always deployed to /usr/local/lib/python3.5, which means a 3.6 version of python3 wouldn't work correctly.

At the end of the day it doesn't really matter anyways, as we're not upgrading versions in the middle of the season anyways.

Update ATLAS et al

I think the RoboRIO toolchain was upgraded, so this will need to be done.

Integrate compileall patch

Patch from python/cpython#4856 needs to be integrated into python package for 2018.

Then update pyfrc: robotpy/pyfrc#90

Get rid of 'Compiling...' messages from python package

Slows down the installation considerably.

2020 RoboRIO uses xz compression for ipkg

.tar.gz still works, but there's definitely some space savings to be had if we switched.

Would need to update the ipkg.py tool and some of the makefiles.

Upgrade to Python 3.6

This will come out in mid December, and in particular there's a few performance improvements they've implemented. If it's like any other python upgrades, this shouldn't be a big deal.

Strip debug symbols from wheels?

Add this to whl2ipk?

Remove mjpg-streamer in favor of using robotpy-cscore

Need to adjust the installer so it can install it.

whl2ipk package versions not compatible with opkg

Needs some investigation, but see robotpy/robotpy-installer#12