Build fails, protobuf dependency about autofdo

[feng-qi]

I had the same problem when building in a Ubuntu 22.04.4 container.

And cmake found this protobuf library:

-- Found Protobuf: /usr/lib/x86_64-linux-gnu/libprotobuf.so (found version "3.12.4")

[michaelrun]

I encountered this issue too, tried both 'master' and 'v0.20', both failed

[erozenfeld]

@shenhanc78 @dhoekwater After recent changes (most likely the submodule perf_data_converter update) the build fails both on Ubuntu 20.04 and 22.04. It looks like the latest protoc and libprotobuf.so available on those versions of Ubuntu is not compatible with the version of perf_data_converter this repo is including as a submodule. What's the proper way to fix this?

[2/279] Building CXX object CMakeFiles/quipper_perf.dir/third_party/perf_data_converter/src/quipper/perf_serializer.cc.o
FAILED: CMakeFiles/quipper_perf.dir/third_party/perf_data_converter/src/quipper/perf_serializer.cc.o
/usr/bin/c++  -I/home/erozen/autofdo/third_party/perf_data_converter/src -I/home/erozen/autofdo/third_party/perf_data_converter/src/quipper -I/home/erozen/autofdo -I/home/erozen/autofdo/third_party/glog/src -I/home/erozen/autofdo/third_party/abseil -I/home/erozen/autofdo/util -I/home/erozen/autofdo/build_debug -I/home/erozen/autofdo/build_debug/third_party/glog -g -std=gnu++17 -MD -MT CMakeFiles/quipper_perf.dir/third_party/perf_data_converter/src/quipper/perf_serializer.cc.o -MF CMakeFiles/quipper_perf.dir/third_party/perf_data_converter/src/quipper/perf_serializer.cc.o.d -o CMakeFiles/quipper_perf.dir/third_party/perf_data_converter/src/quipper/perf_serializer.cc.o -c /home/erozen/autofdo/third_party/perf_data_converter/src/quipper/perf_serializer.cc
/home/erozen/autofdo/third_party/perf_data_converter/src/quipper/perf_serializer.cc: In member function ‘bool quipper::PerfSerializer::SerializeHybridTopologyMetadata(const quipper::PerfHybridTopologyMetadata&, quipper::PerfDataProto_PerfHybridTopologyMetadata*) const’:
/home/erozen/autofdo/third_party/perf_data_converter/src/quipper/perf_serializer.cc:1453:39: error: ‘class google::protobuf::RepeatedField<unsigned int>’ has no member named ‘Assign’
 1453 |   proto_metadata->mutable_cpu_list()->Assign(metadata.cpu_list.begin(),
      |                                       ^~~~~~
[11/279] Building CXX object CMakeFiles/dump_gcov_lib.dir/symbol_map.cc.o
ninja: build stopped: subcommand failed.

[dinyy]

I encountered this issue too , has anyone solved this issue as well？

[snehasish]

@erozenfeld

What's the proper way to fix this?

Discussed this with @dhoekwater and he will look into bundling the appropriate version of protobuf as a submodule.

[dhoekwater]

I'll take a look and see if we can use older versions of some dependencies to get the same functionality.

If not, we can bundle necessary dependencies. Failing that, I'll update the cmake to give more descriptive errors.

[dhoekwater]

Unfortunately, it looks like the protobuf dependency was updated from 3.19.x to 3.20.1 in 261b96f, just two commits before the most recent fix to SPE decoder in b665ece (I suppose this means that any version prior to 3.19.x was already out of date, but that's on us as maintainers to clearly communicate).

Thankfully, none of the intermediate commits change the way they interact with the proto library; the breaking change was introduced in d95f790, so we should be fine downgrading our dependency to any time before then. I'll try that now.

[snehasish]

This should be fixed in #197 and released as 0.20.1.

Thanks for the quick fix @dhoekwater !

[dhoekwater]

No problem, thanks for the review!

[algrant-arm]

Apologies but I'm still struggling with this. With a clean checkout and following the "build with LLVM" commands in the README, cmake prints

-- Found Protobuf: /usr/lib/x86_64-linux-gnu/libprotobuf.so;-lpthread (found version "3.6.1")
-- Configuring done
-- Generating done
CMake Warning:
  Manually-specified variables were not used by the project:

    WITH_LLVM

then the build fails:

ninja: Entering directory `Build'
[20/2580] Building CXX object CMakeFiles/perfdata_reader.dir/perfdata_reader.cc.o
FAILED: CMakeFiles/perfdata_reader.dir/perfdata_reader.cc.o
/usr/bin/c++ -DHAVE_LLVM=1 -I/home/agrant/build/autofdo/autofdo -I/home/agrant/build/autofdo/autofdo/third_party/glog/src -I/home/agrant/build/autofdo/autofdo/third_party/abseil -I/home/agrant/build/autofdo/autofdo/third_party/llvm-project/llvm/include -I/home/agrant/build/autofdo/autofdo/third_party/perf_data_converter/src -I/home/agrant/build/autofdo/autofdo/third_party/perf_data_converter/src/quipper -I/home/agrant/build/autofdo/Build -I/home/agrant/build/autofdo/Build/third_party/glog -I/home/agrant/build/autofdo/Build/third_party/llvm-project/llvm/include -I/home/agrant/build/autofdo/autofdo/third_party/googletest/googletest/include -I/home/agrant/build/autofdo/autofdo/third_party/googletest/googlemock/include -I/home/agrant/build/autofdo/autofdo/util/regexp -O3 -DNDEBUG -std=gnu++17 -MD -MT CMakeFiles/perfdata_reader.dir/perfdata_reader.cc.o -MF CMakeFiles/perfdata_reader.dir/perfdata_reader.cc.o.d -o CMakeFiles/perfdata_reader.dir/perfdata_reader.cc.o -c /home/agrant/build/autofdo/autofdo/perfdata_reader.cc
In file included from /home/agrant/build/autofdo/autofdo/perfdata_reader.cc:16:
/home/agrant/build/autofdo/autofdo/spe_tid_pid_provider.h:14:10: fatal error: google/protobuf/repeated_ptr_field.h: No such file or directory
   14 | #include "google/protobuf/repeated_ptr_field.h"
      |          ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
compilation terminated.

apt reports

libprotobuf-dev is already the newest version (3.6.1.3-2ubuntu5.2).

[dhoekwater]

Looks like the breakage is due to RepeatedPtrField being separated into its own header in v3.19.0. We should be able to fix this one by making the function take an absl::span<quipper::PerfDataProto_PerfEvent*> instead of a const RepeatedPtrField<quipper::PerfDataProto_PerfEvent>&. I'll have to plumb this through a couple places, which I'll get to in the next couple days.

This change does put us a bit further out of sync with the internal version. That's fine for AutoFDO, but once we split Propeller tooling into its own repository, it will probably have a more recent version of google/protobuf as a dependency.

[dhoekwater]

Since repeated_field.h transitively includes repeated_ptr_field in newer versions, swapping the header we include is sufficient.

#197 fixed the breakage for stock builds (non-LLVM), and #200 fixes it for WITH_LLVM builds.