Allowing EXCLUDE_FROM_ALL about cpm.cmake

[TheLartians]

Thanks for the response, I'm glad it has resolved your issue!

The option does sound like a good idea. I think I'll do some experimenting with the option on my own and evaluate if it makes sense to include it as a default. Thanks for bringing it up!

[alandefreitas]

That's beautiful! 😀 It's a single step from a requirements.txt file.
Imagine C++ projects being part of the GitHub dependency graphs.

Unfortunately, GitHub won't pick up the dependencies unless it's in one of the usual formats.

[iboB]

If CPM becomes popular enough, it will be one of the usual formats 😄

[alandefreitas]

It seems like they're just not really interested in common C++ formats because not even vcpkg and conan files are supported there.

It's a shame because being part of the graph has lots of advantages, like automatically documenting dependencies, inferring how far your project has reached and being recommended for sponsorships. Until they are interested, a file in one of the existing formats might be very useful, instead of reinventing the wheel.

Like CPM already has a package-lock.cmake, I was thinking of writing some script to annotate whatever dependencies I included as a pip requirements.txt/pipfile.lock or a npm package-lock.json/package.json.

A requirements.txt file might be easier because it's just a list of repositories. Just a matter of appending a new line to requirements.txt file whenever we call CPMAddPackages. For instance, if we call CPMAddPackage("gh:catchorg/[email protected]"), we would just append a line like

-e git+https://github.com/catchorg/[email protected]#egg= Catch2

to requirements.txt.

A package-lock.json file might be more complicated because writing a json file from cmake must be very clumsy.

The only reason I haven't moved forward with this is that I would still need to compare the supported formats, and look into the syntactic details of pip requirements.txt.

[alandefreitas]

good idea to see if there's any Github staff member who is enthusiastic about C++ enough

That would be great. If anyone is interested in that, things should be much easier. However, even if no one is interested, I think it's still worth working on this.

So there is the discussion: #221

This would be great.

[alandefreitas]

Setting DOWNLOAD_ONLY to TRUE and calling add_subdirectory manually is something I didn't think about. This should be able to achieve something more similar to find_package. It's a great option. What a Columbus' egg. It's not a replacement for find_package in the sense that I'll need some conditionals after that but it's more than enough for what I need. Feel free to close this issue if you want.

If you're still interested in the idea anyway, using EXCLUDE_FROM_ALL could be a good practice when using add_subdirectory as a way to emulate what find_package does. I think that's why they created this option. Because when you use find_package it doesn't mess with your install commands and these imported targets don't get built when you build all. Without EXCLUDE_FROM_ALL, add_subdirectory would never be able to replace find_package properly in any situation because find_package never puts its own targets in your install commands. Some libraries include an option to not build the install target for that reason, but not many libraries do that.

Regarding your question, as far as I understand, EXCLUDE_FROM_ALL does not forbid us to install these imported targets in any way. It just stops these scripts in the subdirectory from messing with the install commands in the main project by default with no way to opt-out. Putting external packages in your own package becomes an option rather than a requirement for all external packages. By default, it behaves a little as if you were using find_package. It's also great not to have lots of useless targets in your "build all" when you just want to use the subdirectory as if it were a package. I'm sorry if I didn't understand your question properly.

[TheLartians]

Hey @saxbophone, thanks for bringing the issue up again and creating an example implementation. I've done some testing and believe the best approach for now is to add an optional flag EXCLUDE_FROM_ALL that can be added to the CPMAddPackage call. I've created an example implementation in #198 and published a pre-release for easy testing of this feature.

[TheLartians]

Yeah, gotcha there, if you don't need any of the additional CPM.cmake features, this would give you most control. FYI in the current implementation, the API would be as

CPMAddPackage(
  NAME googletest
  GITHUB_REPOSITORY google/googletest
  GIT_TAG release-1.8.0
  EXCLUDE_FROM_ALL YES
)

I prefer the explicit EXCLUDE_FROM_ALL option vs enabling it as a default for now, as I've discovered that even my own projects rely on implicit installation of added dependencies and this would be a breaking feature. I still have to add a test case, but would love to hear any feedback you have in the meantime.

[TheLartians]

Awesome, thanks for the feedback and testing! I've just released the feature in v0.29.0. 🎉

[saxbophone]

No problem, thank you very much for cutting a new release with this feature so quickly, much appreciated! 🤝

[alandefreitas]

That makes sense. Thanks again for including this option.

[TheLartians]

FYI, EXCLUDE_FROM_ALL is now the default behaviour for the new shorthand-syntax introduced in v0.31.0, allowing concise definitions such as:

CPMAddPackage("gh:jarro2783/[email protected]")

instead of the way more verbose previous version

CPMAddPackage(
  NAME cxxopts
  GITHUB_REPOSITORY jarro2783/cxxopts
  VERSION 2.2.0
  OPTIONS
    "CXXOPTS_BUILD_EXAMPLES Off"
    "CXXOPTS_BUILD_TESTS Off"
)

Thanks again for the suggestion, this is a great improvement in usability!

[alandefreitas]

I wonder if it would be worthwhile choosing a more unique common filename for the requirements file

That's what I would like, but I think GitHub would ignore the file if we did that.

Some alternatives would be (i) telling GitHub to look into another file, (ii) to annotate which dependencies are python or C++ dependencies, even if it's just a comment, (iii) to use a pip syntax to differentiate build modes, or to (iv) leave it to the lock file only.

The pros/cons seem like (i) Maybe there's a way to tell GitHub to look into another file, but I doubt that. (ii) Including it as just a comment is a workaround that doesn't really solve the problem, but it's better than nothing. (iii) I'm not sure pip provides a syntax for doing that outside comments, but I guess it must be possible somehow since even Python projects might have different dependencies depending on how you want to use it. (iv) I'm not sure if leaving it to lock file solves the problem or doesn't cause any conflict with python projects.

The third alternative seems most promising in that case.

But that's a problem only assuming the project is mixing C++ and Python, and the project cannot use different directories for these things. It's more like a feature than a problem because even a python project might need two different sets of requirements. It should be ok to not have this feature in the initial implementation.

Another interesting thing is that CPM could annotate itself in every project that uses CPM, so it would also have the benefits of being part of the dependency graph and it would make clear which CPM version that project is using.

I apologize if these implications are going too far from the original issue. We could open a new discussion to compare alternatives and possibilities if you want. Both seem fine since this issue is closed anyway.

[TheLartians]

Hey, that's a good question. Currently I don't think there is a good way to achieve this behaviour besides setting the DOWNLOAD_ONLY flag and calling add_subdirectory manually.

I actually wasn't really aware of this option until now, but it does sound like it could be the desired behaviour for most use-cases (given the option to opt-out). Will setting EXCLUDE_FROM_ALL on a directory still allow us to install all targets that are used normally or are there some caveats?

For context, when cross-compiling a project, I find it extremely useful to be able to "install" all relevant targets to a single location for deployment.

[saxbophone]

+1 on this. I have been encountering these same issues with using other CMake-based projects as sub-projects.

I've seen some CMake projects get around this by wrapping their install(export) commands with a condition to check if it's being built as a sub-project or not (such as https://github.com/catchorg/Catch2), but providing an option in CPM to expose the EXCLUDE_FROM_ALL behaviour is IMHO a better solution (wrapping install commands prevents everyone from being able to install from sub-project, which is not what everyone needs: catchorg/Catch2#2164)

[saxbophone]

As you may see from the referenced commit, I have had a go hacking this feature into CPM, with some success. In that commit, I've just set EXCLUDE_FROM_ALL always for targets imported with CPM when FetchContent() is used, however I have been unable to add this as a new option for CPM, nor write any new tests for it or fix the tests that the change broke.

If anyone else wanted to take a look at refining my hack, of course that commit is available as a starting point if needed...

[alandefreitas]

Since then, I've been using:

FetchContent_Declare(
  googletest
  GIT_REPOSITORY https://github.com/google/googletest.git
  GIT_TAG        release-1.8.0
)
FetchContent_GetProperties(googletest)
if(NOT googletest_POPULATED)
  FetchContent_Populate(googletest)
  add_subdirectory(${googletest_SOURCE_DIR} ${googletest_BINARY_DIR} EXCLUDE_FROM_ALL)
endif()

because without the EXCLUDE_FROM_ALL option, both snippets would be almost as verbose anyway.

[saxbophone]

Hi @TheLartians thanks for taking a further look and making that pre-release, I'll take a look at using it for my use case and see if it does the job.

I prefer the explicit EXCLUDE_FROM_ALL option vs enabling it as a default for now, as I've discovered that even my own projects rely on implicit installation of added dependencies and this would be a breaking feature. I still have to add a test case, but would love to hear any feedback you have in the meantime.

I completely agree that EXCLUDE_FROM_ALL should be an explicit option rather than enabled by default, as it would be unfortunate to break existing projects that might rely on the current implicit installation and it's likely anyone who needs the new behaviour, will know they need it and not miss out on finding it when documented.

[saxbophone]

I've just tested the new pre-release v0.29.0-preview-4 and it works just fine for my use case, where I need to find SDL and want to use installed versions where available, or source versions where not. I've tested the pre-release using CPMFindPackage() for this reason:

CPMFindPackage(
    NAME sdl2
    GIT_REPOSITORY https://github.com/libsdl-org/SDL.git
    GIT_TAG main
    VERSION 2.0.15
    OPTIONS "SDL_TEST OFF" "SDL_SHARED OFF" "SDL_STATIC ON"
    EXCLUDE_FROM_ALL YES
)

Note that this call will currently always retrieve SDL from source unless you've compiled your own copy from bleeding-edge, because v2.0.15 hasn't been released yet.

[alandefreitas]

Wouldn't it be possible to have flags without parameters?

CPMFindPackage(
    NAME sdl2
    GIT_REPOSITORY https://github.com/libsdl-org/SDL.git
    GIT_TAG main
    VERSION 2.0.15
    OPTIONS "SDL_TEST OFF" "SDL_SHARED OFF" "SDL_STATIC ON"
    EXCLUDE_FROM_ALL # YES
)

That would make it compatible with the syntax for add_subdirectory (add_subdirectory(_______ EXCLUDE_FROM_ALL) instead add_subdirectory(_______ EXCLUDE_FROM_ALL TRUE)). And the usual syntax for most cmake functions: (find_package(_______ QUIET) instead of find_package(_______ QUIET TRUE), etc). It seems more intuitive.

[TheLartians]

I did have that in the first implementation, but decided against the flag syntax, as

• it's consistent with current CPM.cmake syntax: all current arguments use a KEY VALUE declaration, including on/off parameters such as DOWNLOAD_ONLY, GIT_SHALLOW, etc. IMO it would be more confusing if some parameters were with value and others without.
• it's easier to extend: say for some reason in the future we want to add an additional setting for EXCLUDE_FROM_ALL besides on/off, we could just re-use the existing parameter, e.g. EXCLUDE_FROM_ALL sometimes.

[saxbophone]

FYI, EXCLUDE_FROM_ALL is now the default behaviour for the new shorthand-syntax introduced in v0.31.0, allowing concise definitions such as:

CPMAddPackage("gh:jarro2783/[email protected]")

instead of the way more verbose previous version
...

Wow! Thank you for the update!

It seems like they're just not really interested in common C++ formats because not even vcpkg and conan files are supported there.

It's a shame because being part of the graph has lots of advantages, like automatically documenting dependencies,
...

I too lament at the lack of inclusion of C++ projects in the dependency tracker. I did respond once to a GH survey about future expansion of the feature. My guess would be it's not on GH's priorities list in part due to focusing more on web-dev popular languages, as well as the fact that C++ unfortunately does not have wide consensus on build systems (although I understand CMake is a popular contender).

Like CPM already has a package-lock.cmake, I was thinking of writing some script to annotate whatever dependencies I included as a pip requirements.txt/pipfile.lock or a npm package-lock.json/package.json.

A requirements.txt file might be easier because it's just a list of repositories. Just a matter of appending a new line to requirements.txt file whenever we call CPMAddPackages. For instance, if we call CPMAddPackage("gh:catchorg/[email protected]"), we would just append a line like
...

This is an exciting idea! I wonder if it would be worthwhile choosing a more unique common filename for the requirements file (that way, one could mix C++ and Python in the same project without the requirements files conflicting)?

[saxbophone]

I apologize if these implications are going too far from the original issue. We could open a new discussion to compare alternatives and possibilities if you want. Both seem fine since this issue is closed anyway.

Good points all around, I am glad we have this opportunity to discuss these ideas with others who are interested in them. Please invite me to the discussion if it moves elsewhere!

...I wonder if it could be a good idea to see if there's any Github staff member who is enthusiastic about C++ enough that it could be worth tagging them in a conversation about C++ project dependency tracking, maybe with a view to making the case for such a feature to Github?

[iboB]

Yeah, this belongs in discussions. It's a very interesting topic