Decouple external build configuration (Travis CI/YCM) from local settings about questions-and-answers

[PeterBowman]

This problem has been approached in some valid ways that are close enough to the intended result. I'm going to close it at last. A few remarks regarding what was discussed here:

• initial-cache.cmake: doable, but we don't sacrifice that much thanks to the adoption of ccache in CI builds
• feature_summary(): doable, but we tend to minimize console output, and YARP devices are already logged (this can be disabled, though)
• find_package(): I was considering the removal of the QUIET option; sadly, config-type packages (as opposed to find modules) are a bit too verbose on failure
• #18 (comment): not perfect, but it has served well all these years

Therefore, the original issue was already addressed quite a while ago. We ended up spreading boilerplate as detailed in previous comments (e.g. roboticslab-uc3m/vision@f5c89fb). Some people choose to see the ugliness in this code. The disarray. I choose to see the practicality. To believe there is an order to our CMake list files, a purpose.

[PeterBowman]

CMake 3.19 introduced presets which mostly behave like a JSON-formatted initial cache.

[PeterBowman]

Warnings may be annoying, but help to keep users aware of which components could not be successfully configured. Setting everything to OFF essentially hides the functionality of a repo.

Idea: try the FeatureSummary CMake module. It was adopted some time ago in amor-api, then in project-generator. Devs may use it to register all available CMake options and provide short descriptions. Then, CMake generates a report after the configuration step. Example output of amor-api:

-- YCM found in /usr/local/share/YCM.
-- AMOR API version: 0.2.1 (0.2.1)
-- Boost version: 1.58.0
-- Found the following Boost libraries:
--   thread
--   date_time
--   chrono
--   system
--   atomic
-- Could NOT find Doxygen (missing:  DOXYGEN_EXECUTABLE) 
-- 
-- The following features have been enabled:

 * AMOR_API_extended , Advanced AMOR API functions.

-- The following REQUIRED packages have been found:

 * Eigen3
 * Boost

-- The following features have been disabled:

 * AMOR_API_cartesian_rate , cartesian_rate program (AMOR+YARP).

-- The following OPTIONAL packages have not been found:

 * Doxygen

-- Configuring done
-- Generating done
-- Build files have been written to: /home/bartek/wingit/amor-api/build-wsl

Starting from The following features have been enabled, we see the result of invoking feature_summary(). Moreover, it lists satisfied and missing package dependencies (find_package() calls).

[PeterBowman]

Proposed roadmap (I'll replicate and expand this in the issue description if nobody opposes):

• Apply improvements explained in #18 (comment) in local CMake list files, and simplify the option list passed to the cmake .. command in local .travis.yml. All options (except ENABLE_coverage and, perhaps, ENABLE_tests (see #41)) set to ON.
• Create an initial-cache.cmake in scripts/ (#18 (comment)) with all (relevant) options set to OFF, use this in foreign .travis.yml files. See last paragraph in previous comment for reasons.
• Use a single find_package(YARP REQUIRED) call in root CMakeLists.txt - except wherever a specific version of YARP is needed.

TODO: I need to check this ideas on a superbuild-like repo, especially regarding CMake options and BuildPkg.cmake files.

[PeterBowman]

In order to alleviate configuration efforts (particularly with installation guides in mind), we could avoid passing several -Dxxx CL parameters (one per CMake option, e.g. -DENABLE_fancy_program and so on) by introducing a single -Cxxx parameter pointing at the most convenient .cmake intended for initializing the cache during the first run of cmake. Examples at robotology/yarp: CL invocation, initial-cache.cmake.

[PeterBowman]

Another ridiculous example (scroll right): roboticslab-uc3m/kinematics-dynamics@d5a9599.

[PeterBowman]

I did some CMake refactorization work in a bunch of repos. Namely:

• roboticslab-uc3m/kinematics-dynamics#14
• roboticslab-uc3m/yarp-devices#26
• roboticslab-uc3m/openrave-yarp-plugins#56

Setting all options to OFF may create a twofold problem:

• If not handled correctly, CMake could be asked to process an export set with no actual targets linked to it. See roboticslab-uc3m/project-generator/issues/19 and related issues.
• In order to prevent from building targets if their local dependencies are disabled, I resorted to using cmake_dependent_option() and the DEPENDS parameter of yarp_prepare_plugin() (examples: one, two). However, users might get tired of switching options to ON with ccmake and repeatedly configuring the project - relevant options won't show up until all their dependencies are previously enabled by hand.

[PeterBowman]

@jgvictores suggests to keep all/most options enabled and warn the user if it cannot get built for whatever reason (missing package, unsatisfied dependencies, etc.).

[jgvictores]

Yes, essentially all enabled, and automatically disabling (without breaking, just messages/warnings) if requirements are not met.

[PeterBowman]

YARP devices already provide some sort of summarization mechanism, check this cmake .. sample run:

--  --- plugin AmorCartesianControl: disabled
--  +++ plugin AsibotSolver: enabled
--  +++ plugin BasicCartesianControl: enabled
--  --- plugin BasicTwoLimbCartesianControl: disabled (dependencies unsatisfied)
--  +++ plugin CartesianControlClient: enabled
--  +++ plugin CartesianControlServer: enabled
--  +++ plugin KdlSolver: enabled
--  +++ plugin TwoLimbCartesianControlServer: enabled

[PeterBowman]

keep all/most options enabled and warn the user if it cannot get built for whatever reason (missing package, unsatisfied dependencies, etc.)

Perhaps we should move most find_package() calls to an upper level (root CMakeLists.txt?) and then use the corresponding _FOUND variables in cmake_dependent_option()/yarp_prepare_plugin() (e.g. AmorCartesianControl/CMakeLists.txt@kinematics-dynamics).

[jgvictores]

Perhaps we should move most find_package() calls to an upper level (root CMakeLists.txt?)

Advantage: Faster and cleaner output of cmake.
Disadvantage: Goes a bit against philosophy of everything near where it is used...

Seeing the disadvantage... Maybe only in the case a dep is detected to be used on more than one component, then move up one level? I have no idea....

[PeterBowman]

Idea: place find_package() calls before the opening if() clause in cases such as:

yarp_prepare_plugin(AmorCartesianControl
                    CATEGORY device
                    TYPE roboticslab::AmorCartesianControl
                    INCLUDE AmorCartesianControl.hpp
                    DEFAULT ON
                    DEPENDS ENABLE_KinematicRepresentationLib)

if(NOT SKIP_AmorCartesianControl)

find_package(AMOR_API REQUIRED)

# ...

Obviously, the REQUIRED parameter should be removed. Then, custom code (more ifs, message(), etc.) would take care of warning the user about unsatisfied deps and reflecting that in the GUI.

[PeterBowman]

Working example at roboticslab-uc3m/kinematics-dynamics@14cf9cb. Template:

find_package(3rdparty_pkg QUIET)

if(NOT 3rdparty_pkg_FOUND AND (NOT DEFINED ENABLE_MyComponent OR ENABLE_MyComponent))
    message(WARNING "3rdparty_pkg package not found, disabling MyComponent")
endif()

# same with yarp_prepare_plugin()
cmake_dependent_option(ENABLE_MyComponent "Enable/disable MyComponent" ON
                       3rdparty_pkg_FOUND OFF)

if(ENABLE_MyComponent) # 'NOT SKIP_MyComponent' for YARP devices
    # ...
else()
    set(ENABLE_MyComponent OFF CACHE BOOL "Enable/disable MyComponent" FORCE)
endif()

Travis YAML files may be updated so that local repository options are now omitted (see .travis.yml).

Regarding non-local CMake options (as described in the first comment in this issue, i.e. remote repos pulled via .travis.yml with lots of disabled components), we'll want to provide a means to disable all/most non-vital options. The reason is that downstreams should not be forced to compile unnecesary programs and libraries, especially whenever we just want to get access to a header file. To achieve that, the initial-cache.cmake way may be a valid solution (#18 (comment)).

[PeterBowman]

Expanding on my previous comment, such refactorization is only intended for devices/programs/libraries that depend on REQUIRED 3rd-party packages - except YARP. As spoken with @jgvictores, YARP itself should be considered a hard dependency in our repos, that is, we'll always instruct CMake to search for it with find_package(YARP REQUIRED) (note the second parameter). I'd go a bit further and pull this call to the root CMakeLists.txt, before traversing any subdirectories.

[PeterBowman]

Another TODO: if I place find_package(YARP REQUIRED) at the top, and then proceed as described earlier (3rd-party dep) with a YARP version that is not available on my system (e.g. find_package(YARP 2.3.70 QUIET)), what will happen with directories processed after that which depend on previous (supported) YARP releases?

(I was thinking about certain YARP devices in openrave-yarp-plugins)

[jgvictores]

I think I've found an issue with the current proposal.

Looking at these lines, cmake always goes through find_package(ARAVIS QUIET). Now, if we look at FindARAVIS.cmake, it has a find_package(GLib REQUIRED) which makes everything break, always, even if we are not using AravisGigE... :-(

[PeterBowman]

That find_package(GLib REQUIRED) should be moved to the caller list file (e.g. AravisGigE/CMakeLists.txt). Similarly, you have to explicitly call both find_package(OpenRAVE REQUIRED) and find_package(Boost REQUIRED) in the openrave-yarp-plugins repo.

However, I'd personally give a try to the find_dependency() command (ref) within FindARAVIS.cmake. I used to have one at ROBOTICSLAB_KINEMATICS_DYNAMICSConfig.cmake.in.

[jgvictores]

Okay, we can open a new issue for this, for now hacked via sudo apt install libglib2.0-dev.

[PeterBowman]

Another example: disable component if YARP's math lib cannot be found (roboticslab-uc3m/kinematics-dynamics@76eb0d7).

Edit: follow-up at roboticslab-uc3m/kinematics-dynamics@b93ff7d (avoid continuous generation of warning messages).

[PeterBowman]

Sometimes, a component depends on another. If the latter is not enabled, the former (usually) does not show up in the GUI at all. However, my proposal prints a warning anyway and I noticed that dependers do still show up (vision repo, check TravisLib and colorDetection(2D)).

[PeterBowman]

More examples:

• unsuitable Python version: roboticslab-uc3m/yarp-devices@b6fe2b9
• unsuitable YARP version: roboticslab-uc3m/openrave-yarp-plugins@18291fa

[PeterBowman]

This mechanism (#18 (comment)) is not aware of 3rd-party deps being available upon successive configure runs. Example:

• make sure that orocos_kdl is not installed on system nor registered at ~/.cmake/packages
• clone and configure https://github.com/roboticslab-uc3m/kinematics-dynamics/
• several components (e.g. KdlSolver) should be automatically disabled (and respond with a warning when manually switched to ON)
• configure and install orocos_kdl
• run cmake on kinematics-dynamics
  • expected: previously disabled options (depending on orocos_kdl) should be now enabled
  • result: nothing changes
• (we should not be here, but previous step failed...) enable KdlSolver via ccmake
  • expected: ...well, it's set to ON in the GUI
  • result: upon pressing c (configure), KdlSolver is switched back to OFF with no warning

[PeterBowman]

YARP itself should be considered a hard dependency in our repos, that is, we'll always instruct CMake to search for it with find_package(YARP REQUIRED) (note the second parameter)

Example: roboticslab-uc3m/yarp-devices@2fdc802.

[PeterBowman]

#18 (comment):

Idea: try the FeatureSummary CMake module.

I think it's already supported in custom YARP devices (ref, available since 2.3.68.1), that is, add a feature_summary(...) call at the end of root CMakeLists.txt and output should be automatically generated along with a list of available components.

[PeterBowman]

Another example (test two conditions): roboticslab-uc3m/vision@f5c89fb.

[PeterBowman]

Note: new YARP components may require additional attention when working with RL projects depending upon each other, see roboticslab-uc3m/kinematics-dynamics@273ac97 and #65.

[PeterBowman]

TODO: reflect this new dependency+warning mechanism in project-generator and repositories derived from this one (mainly tools and asibot-hmi).

[PeterBowman]

More ideas: robotology/yarp@758d23c (Plugins not enabled due to missing dependencies are now shown in ccmake andcmake-gui together with a list of dependencies that are not satisfied.).

[PeterBowman]

Somewhat a follow-up to #48 motivated by roboticslab-uc3m/openrave-yarp-plugins#91: if we just need a header file (usually, a YARP device interface) or a CMake module (rare) from a dependency, then clone/download, run cmake .., skip the compile part altogether and adjust the <project>_DIR variable.

[PeterBowman]

Perhaps we should move most find_package() calls to an upper level (root CMakeLists.txt?)

YARP itself should be considered a hard dependency in our repos

For future reference, this was done at #54.

[PeterBowman]

(#18 (comment))

This mechanism (...) is not aware of 3rd-party deps being available upon successive configure runs.

YARP v3.0.x suffers from this same problem, too, after robotology/yarp@758d23c was introduced. Looks like the final part of my proposed solution (#18 (comment)), which also resembles what YARP aims to achieve, is colliding with cmake_dependent_option's internals called either explicitly or from within yarp_prepare_plugin():

set(ENABLE_MyComponent OFF CACHE BOOL "Enable/disable MyComponent" FORCE)

[PeterBowman]

Note for travelers:

if(${var} MATCHES "^${var}$")

is a legacy way of saying:

if(NOT DEFINED ${var})

Found in CMakeDependentOption.cmake.

[PeterBowman]

CMake 3.13 introduces policy CMP0077. Not sure if we want this new behavior, though.