googleapis / google-cloud-cpp Goto Github PK

C++ Client Libraries for Google Cloud Services

License: Apache License 2.0

CMake 0.73% C++ 97.88% Shell 0.24% Python 0.01% C 0.16% Makefile 0.16% Awk 0.01% Starlark 0.70% Dockerfile 0.11%

google cloud google-cloud google-cloud-platform cpp google-cloud-storage bigtable google-cloud-spanner google-cloud-bigtable spanner

google-cloud-cpp's Introduction

Google Cloud Platform C++ Client Libraries

This repository contains idiomatic C++ client libraries for the following Google Cloud Platform services.

Please check the CHANGELOG for important announcements and upcoming changes.

Quickstart

Each library (see below) contains a directory named quickstart/ that's intended to help you get up and running in a matter of minutes. This quickstart/ directory contains a minimal "Hello World" program demonstrating how to use the library, along with minimal build files for common build systems, such as CMake and Bazel.

As an example, the following code snippet, taken from Google Cloud Storage, should give you a sense of what it's like to use one of these C++ libraries.

#include "google/cloud/storage/client.h"
#include <iostream>

int main(int argc, char* argv[]) {
  if (argc != 2) {
    std::cerr << "Missing bucket name.\n";
    std::cerr << "Usage: quickstart <bucket-name>\n";
    return 1;
  }
  std::string const bucket_name = argv[1];

  // Create aliases to make the code easier to read.
  namespace gcs = ::google::cloud::storage;

  // Create a client to communicate with Google Cloud Storage. This client
  // uses the default configuration for authentication and project id.
  auto client = gcs::Client();

  auto writer = client.WriteObject(bucket_name, "quickstart.txt");
  writer << "Hello World!";
  writer.Close();
  if (!writer.metadata()) {
    std::cerr << "Error creating object: " << writer.metadata().status()
              << "\n";
    return 1;
  }
  std::cout << "Successfully created object: " << *writer.metadata() << "\n";

  auto reader = client.ReadObject(bucket_name, "quickstart.txt");
  if (!reader) {
    std::cerr << "Error reading object: " << reader.status() << "\n";
    return 1;
  }

  std::string contents{std::istreambuf_iterator<char>{reader}, {}};
  std::cout << contents << "\n";

  return 0;
}

GA Libraries

See each library's README.md file for more information about:

Where to find the documentation for the library and the service.
How to get started using the library.
How to incorporate the library into your build system.
The library's support status if not Generally Available (GA); unless noted in a library's README.md, these libraries are all GA and supported by Google.

Expand to see the full list of GA libraries

Building and Installing

This is a quickstart guide for developers wanting to compile the libraries and run the examples included with the libraries.

Packaging maintainers or developers who prefer to install the library in a fixed directory (such as /usr/local or /opt) should consult the packaging guide.
Developers who prefer using a package manager such as vcpkg, or Conda, should follow the instructions for their package manager.
Developers wanting to use the libraries as part of a larger CMake or Bazel project should consult the quickstart guides for the library or libraries they want to use.
Developers wanting to compile the library just to run some examples or tests should read the current document.
Contributors and developers to google-cloud-cpp should consult the guide to set up a development workstation.

Building with Bazel

This library requires Bazel >= 6.0. From the top-level directory, run the usual commands.

bazel build //...

Building with CMake

This library requires CMake >= 3.10. If you are planning to install the libraries please consult the packaging guide, these instructions will NOT produce artifacts that you can put in /usr/local, or share with your colleagues.

From the top-level directory of google-cloud-cpp run these commands:

git -C $HOME clone https://github.com/microsoft/vcpkg.git
env VCPKG_ROOT=$HOME/vcpkg $HOME/vcpkg/bootstrap-vcpkg.sh
cmake -S . -B cmake-out/ -DCMAKE_TOOLCHAIN_FILE=$HOME/vcpkg/scripts/buildsystems/vcpkg.cmake
cmake --build cmake-out -- -j $(nproc)

The binary artifacts, such as examples, will be placed in cmake-out/.

Support

This project follows Google's Foundational C++ Support Policy, which is summarized in the Foundational C++ Support Matrix.
- Language Version (>= C++14).
- Operating Systems: Windows, macOS, and Linux.
- Build Systems: Bazel (>= 6.0), CMake (>= 3.10).
- Compilers: GCC (>= 7.3), Clang (>= 6.0), MSVC (>= 2022), Apple Clang (>= 12).
This project uses dependencies described in doc/packaging.md.
This project works with or without exceptions enabled.
This project cuts monthly releases with detailed release notes.

Public API and API Breaking Changes

This project follows Google's OSS Library Breaking Change Policy.

In general, we avoid making backwards incompatible changes to our public API. Sometimes such changes yield benefits to our customers, in the form of better performance, easier-to-understand APIs, and/or more consistent APIs across services. When these benefits warrant it, we will:

Announce these changes prominently in our CHANGELOG.md file and in the affected release's notes.
Increase the major version for google-cloud-cpp.

Nevertheless, though we take commercially reasonable efforts to prevent this, it is possible that backwards incompatible changes go undetected and, therefore, undocumented. We apologize if this is the case and welcome feedback or bug reports to rectify the problem.

Contact us

GitHub Discussions -- For questions and general comments
GitHub Issues -- For reporting bugs and feature requests

Contributing changes

See CONTRIBUTING.md for details on how to contribute to this project, including how to build and test your changes as well as how to properly format your code.

Licensing

Apache 2.0; see LICENSE for details.

google-cloud-cpp's People

Contributors

Stargazers

Watchers

Forkers

coryan mbrukman chemelnucfin danielmahu manish-qlogic hemant-qlogic ajaaym roopak-qlogic avaksman deepankarsharma paprat linecode cschuet rcleveng houglum saeta jc-pixit gibbdash aviakasurvivor kyledj shubhamshubhankar frankyn csumathdyj perfinion phillip-elm aspineon dopiera omunroe-com sargishayrapetyan nettyh tfgt devjgm akater320 benjaminr rsimion yuhonghong7035 jsoref mr-salty bhanditz devbww yiakwy nickolasrossi lian-hu huangtimlinfeng past-due scotthart reivash billyjacobson vahagsaribekyan kaustubhshamshery wisehub-team limalucas96 bzztbomb grant jseph vperus dexterb oneoftheirs flj emkornfield crackwei exhorder cclauss renovate-bot alichnewsky tonysu7 shanebishop dvmorris johnedmonds jkuszmaul coincashbank aocsa crwilcox peteypii aryann jiefei-wang larriojas azarel1996 themidwestcanapps helmithejoe fangtanchen timford fbrmedia daniel457 m7md889 kajithirive leal1981 alexyigidey8 gberesford rajpratik71 mcharsley silvrwolfboy mark-akaon wegorekm xiaoliuzi mtrippled millerantonio810 clementperon ihnorton 2014cocos

google-cloud-cpp's Issues

Implement ClientConfig class

Implement a class to configure the bigtable::Table client. Write down what it does, separate the code to its own files, write tests, etc.

Add dynamic test instrumentation via Clang plugins

Clang supports a number of plugins for detecting code issues at runtime, such as:

We can enable these individually when tests are compiled with Clang, and then we can just run them as usual, so it could just be a compile-time flag to enable any of these.

If we're going to run a bunch of these options on Travis CI, it would be much better for us to run them all on Linux, rather than on macOS, because Travis CI can spin up any number of Linux VMs or containers (as they use cloud infrastructure, which is elastic), while macOS is required to run on Apple hardware and is thus significantly limited, and thus may incur delays because this project (and this entire org) will be throttled on parallel macOS builds as a result.

Implement Apply for multiple rows

At the end of this task the bigtable::Table class will have a member function to apply multiple mutations to multiple rows in a single call. A good name might be BulkApply(), but Apply() overloaded for some class that encapsulates multiple mutations is also acceptable.

The function should follow the same policies defined in #12 to control what operations are idempotent, how to backoff when retrying RPC operations, which error codes are retryable, etc.

Compile msan-enabled library to avoid false positives in MemorySanitizer

Clang's MemorySanitizer requires an instrumented version of the standard library. We should change the build system to create such a library when needed, and use it in the build. There are details on how to do that here:

https://github.com/google/sanitizers/wiki/MemorySanitizerLibcxxHowTo

Test if disabling all caching accelerates docker builds

Since all we do is download some packages before compiling gRPC and the client library it might be faster to download the packages every time instead of trying to cache the build results. Test it out and report the results here, if faster, change the main build.

Add build to validate formatting.

We need to configure one build to run clang-format and to fail if there are any differences between the code in a PR (or commit!) vs. the code formatted with clang-format.

File bug and/or submit patch for vcpkg

Microsoft has a project to support open source packages under MSVC, much like the FreeBSD ports collection. They have patches / scripts to compile protobuf, grpc, abseil, etc. We should submit a bug and/or patches to compile the client library.

Add continuous integration and testing

We should add continuous integration and testing such that each PR and the master branch is regularly tested.

Travis CI can be used for Linux and OS X; AppVeyor can be used for Windows builds.

Autogenerate and host API documentation

Users shouldn't have to read the header files to understand the API; this project should provide the equivalent of Javadoc for users to explore easily via their browser.

C++ has several API documentation autogeneration options:

Doxygen, possibly the most popular / well-known for C++
- Moxygen which appears to depend on Doxygen, seems to help generate nice Markdown pages
- Sphinx appears to only work with C++ via Doxygen + Breathe (source: Stack Overflow)
- Exhale depends on Doxygen, Sphinx, and Breathe
cldoc, based on Clang, rumored to be unmaintained based on commit history
Standardese — claims to be "nextgen Doxygen" but is experimental, so maybe not ready for us to use yet
other options listed on Wikipedia

Also, other Google Cloud client library projects have a landing page, and publish docs to a central location, so this repo should also follow suit. For example, here are a few sites for existing APIs:

Create an example and CI build to test the 'install' target

At the end of this task we will have:

A test for the install target: this is a program that uses the Cloud Bigtable C++ client as installed by make install.
A CI build that exercises this test.

This is because it is too easy to forget header files, or libraries in the install rules and then the library is unusable.

Implement (sync) ReadRow() a read for a single row

At the end of this task the bigtable::Table class will have a member function to read a single row, and the unit tests associated with it.
The member function will use the existing policies to determine whether to retry any RPC requests, and how to backoff the RPC requests.
The member function will return a Row class with all the data in memory.
This function should take both a absl::string_view parameter for the row key and a bigtable::Filter parameter to control the filter expression associated with the request.

Consider splitting this issue if theFilter class is not ready.

Implement examples that run against the production Cloud Bigtable.

We need examples showing how the client library is used against Cloud Bigtable. Ideally the examples should manipulate non-trivial amounts of data (a few GiB at least).

Investigate posting relevant static and dynamic analysis findings as PR comments

It should be possible to use the github APIs to post some of the errors from our builds directly as PR comments, for example:

The code formatting errors have line numbers and can be flagged as comments in the PR.
Any clang-tidy errors (in a future clang-tidy build) have line numbers and can be flagged as comments in the PR.
The errors from the sanitizers often have line numbers and could be posted to the review too.

Create a build that runs clang-tidy

At the end of this task we will have a build in the matrix that runs clang-tidy. Ideally the results will be pushed as comments to PRs, but even if they are in the build log that would be good enough.

Randomize backoff delay in ExponentialBackoffPolicy.

The current backoff policy is too naive, it just uses exponential backoff (with a limit), we need to add randomization to avoid multiple clients synchronizing their retries.

Consider separating image creation and running build inside the image.

Currently we run the CI builds while building a container, in a single docker build command. This downloads all the dependencies (compilers, build tools, libraries), copies the code from the CI workspace to the container, and then run the builds inside the container.

We should consider separating this in two steps. First create a Docker image with all the dependencies, then run a container (with docker run) with that image that runs the build.

The advantage is that we might be able to cache the build images, that saves some time. We can also mount the git workspace instead of using a COPY Docker command to copy the contents (around 250MiB) into the container.

The disadvantage is that when running locally the build script would leave build-output owned by root in your workspace.

Add policy to control how does the Table class backoff on RPC failures

At the end of this task it will be possible to configure the bigtable::Table class with a policy to control how does the class backoff before retrying failed RPC operations. The default policy should use exponential backoff.

Decide if CentOS 7 is a target platform.

Decide if it is (very likely) and if so, get the code to compile on it (very hard).

Cleanup comments with ... in them.

Change them to assertions and remove the ... play-by-play commentary.

Create pkg-config files during install

Many users would like to integrate with their build systems using pkg-config. We need to create the right *.pc files as we install the libraries so the users can take advantage of this configuration.

See the protobuf install.cmake file for an example.

Track code coverage metrics

While code coverage is an imperfect metric of test quality (necessary, but not sufficient), it's an additional useful signal to keep in mind when assessing the quality of the overall test suite.

There are several SaaS options which are free for open-source projects:

Codecov
Coveralls
Circle CI (code coverage + CI in one)

I've heard good things about Codecov, and we're using it in another project and it works well, and is easy to set up.

Here are the C++ setup instructions for Codecov, which seem quite simple and straight-forward.

As an aside: Codecov uses gcov for C++ coverage (via Clang's -coverage flag), which is also what the Google blog post on code coverage from 2014 talks about, so seems like this is a good choice.

Write down design doc for Row Range and Row Set wrappers

Write down a design document for row ranges and row sets. Document the implementation steps as issues in this repository.

Add policy to control which RPC operations are re-tryable.

At the end of this task it will be possible to configure the bigtable::Table class with a policy to control which RPC operations can be retried based on their status codes. The default policy should only retry operations that fail with grpc::StatusCode::ABORTED, grpc::StatusCode::UNAVAILABLE, or grpc::StatusCode::DEADLINE_EXCEEDED.

Design wrappers for filter expressions

At the end of this task we will have classes (and standalone functions) to create filter expressions.
The first step is to write a design doc for how they would work, and break down the work into smaller issues.

Design full build matrix and implement it.

We need to gather some data and then complete the definition of the build matrix. There is an skeleton in #8, but that won't be enough. And a full NxMxK matrix of all combinations of compiler, platform, and build options would be overkill.

The design should answer these questions:

What version of gRPC we support?
What distributions of Linux we support?
What compilers and versions we plan to support?
For Windows, we need to get specific with MSVC versions

Implement (sync) Table::Apply(SingleRowMutation)

At the end of this task bigtable::Table will provide an Apply() member function to apply mutations to a single row. The member functions should:

Use the policies in effect to retry any recoverable RPC failures (#14)
Use the policies in effect to determine if a mutation is idempotent (#13)
Report an error when the operation fails after the policies "give up" (#16)

Do not run the macOS build for every PR; just run on master

The macOS build rarely finds any problems, and it is unlikely to be an important platform for the target libraries. Certainly some of our users and developers will take advantage of it, but it is unlikely to be the main platform.

It is often very delayed in Travis CI though, if we run it on a nightly build we will get most of the benefits, and save ourselves a lot of waiting.

Add EditorConfig support

EditorConfig provides a universal way to configure editor preferences (e.g., indentation rules, file encoding, EOL style, etc.), which can then be easily included by most editors and enforced on a per-repo basis, even if they may differ from the user's global preferences, and make it easy for a user to work with multiple projects, without having to manage their own custom editor configuration on a per-project basis.

Note that EditorConfig can support any files, so this isn't limited to just the C++ source code, but can include, e.g., CMakeLists.txt, README.md, *.proto files, text protobufs for testing, etc.

Refactor testing classes to a testing/ subdirectory

We have a small number of classes that are used only for testing, for example, declarations of mock objects, the chrono literals, etc. We should refactor those to a testing/ subdirectory (and namespace?) first to be able to share them, and second to avoid duplication.

Implement Table::ReadRows() wrapper

We need to write a brief design down for the ReadRows() and its associated wrappers. We also need a breakdown of the tasks associated with this work as issues in this repository.

Add static analysis to find bugs earlier

There are a number of options for finding issues via static analysis, whether via self-run tooling or via SaaS, e.g.,

Specific tools

CppCheck — open-source, self-managed
Coverity Scan — commercial SaaS, free for open-source GitHub repos — most of their success stories are for C/C++ repositories!
Clang static analyzer — open-source, self-managed
FlawFinder — a new one I found, but haven't heard much about it in the past

Collections, lists, reviews, etc.

a list of static analysis tools (Wikipedia)
another list of tools (via GitHub repo)
yet another list of tools which separates tools into 3 categories: (i) free/open-source, (ii) free for academic use, and (iii) commercial
a Stack Overflow discussion about C++ static analysis tools
a review of C++ static analyzers (June 2017)
static code analysis comparison by John Carmack (of Quake fame); posted on the PVS-Studio blog because his original article is no longer available

I think we should start with enabling Coverity Scan and integrating Clang analyzers (since we're going to use Clang for building our code anyway).

Adding CppCheck and FlawFinders are also a good idea, but if they don't support C++11 (or C++14, if/when we upgrade this repo's minimum C++ requirements), that might become an issue.

Configure issue template for github

We need to capture key information, such as:

The version of the library.
Compiler name and version.
Operating system version.
gRPC version.
protobuf version.

Create a static function for ClientOptions(Data)

After resolving this issue we should be able to get a configured client using this function that can be used with emulator.

Implement integration test for Apply(Single) against Cloud Bigtable Emulator

At the end of this task we will have an integration test that checks the Apply(Single) member function against the Cloud Bigtable Emulator

Decide if these tests should run on the CI build after each commit, or nightly, document why and set it up.

Refactor wrappers without exceptions to separate namespace.

We need to write separate wrappers that do not use exceptions to report errors. The higher level wrappers will be written in term of these internal wrappers to be used only by Google.

Implement a policy to control which mutations are idempotent

At the end of this task it will be possible to configure the bigtable::Table class with a policy to control which operations are idempotent. The default policy only accepts safe (true idempotent) mutations.

Refactor implementation detail classes to internal/ subdirectory.

Right now they are in the main directory and that makes it hard for users to navigate the code.

Create cmake module config files during install

Some of our users may want to use cmake for their builds, we should create module files during the installation so the find_package(bigtable-client) (or whatever we call our library) function works correctly.

See the protobuf install.cmake file for an example.

Separate getters/setters for different Channel Argument functions available.

After resolving this, user should be able to get/set different channel arguments using the functions available.

Compile with clang-4.0

The build fails with clang-4.0 (and clang-3.9) because the version of protobuf included with the grpc version we are using has a bug. This is a big rabbit hole, we may need to fix protobuf first (file a bug, submit a PR), then change the version of protobuf that grpc has pinned (file a bug, submit a PR), and then change our version of grpc (this bug, a future PR).

I am not sure what this means for our plans for he build matrix in #19, it might mean that we do not support older versions of grpc (e.g. v1.6.x), or that we do not support using the particular compiler / grpc version combos that are broken.

Convert any existing TODO items pointing at people into github issues

Instead of TODO(coryan) it should be TODO(#NNN).

Compile with gcc-7.x

This is similar to #58 but for gcc. The build fails with gcc-7.0 because one or more of our dependencies have warnings in their exported headers with gcc-7.0. This is a big rabbit hole, we may need to fix protobuf or zlib or boringssl first (file a bug, submit a PR), then change the version of that dependency that grpc has pinned (file a bug, submit a PR), and then change our version of grpc (this bug, a future PR).

Create a build for Windows 64 using appveyor

At the end of this task we will have a CI build for appveyor using Windows 64. There is some prior art for things using protobuf + grpc in:

https://github.com/coryan/cpp-docs-samples

Consider enabling codecov.io comments on PRs

Currently posting on PRs is disabled, we may want to enable it to make sure no PR reduces code coverage significantly.

The RPC policies should be able to set parameters in the ClientContext

The RPC policies should be able to set the grpc::ClientContext used for each request, for example, they may want to control the deadline based on how much time is left overall.

Setup an ABI compliance tracker before releasing v1.0

The ABI will change dramatically during the initial development, but should be fixed after v1.0.0 and this should be enforced using a compliance tool such as:

https://github.com/lvc/abi-compliance-checker

Consider making some policies optional in Table constructor.

One of the constructors for bigtable::Table takes the error handling policies in effect for the table class. It would be desirable (for example in some of our tests) to only set some of the policies and use the default for the rest.
Figure out if it is possible to do that (variadic args and some clever type matching?)

Add policy to control when the Table class stops retrying

At the end of this task it will be possible to configure the bigtable::Table class with a policy to decide when to give up retrying RPC operations. The default should use a long timeout, but other policies (maximum number of failures, combinations of failure counts and time) are possible.

Use absl::string_view where appropriate

Make one pass on the initial code and make sure we are using absl::string_view where appropriate.

Complete implementation of Client class

The bigtable::Client class is kind of a skeleton right now. Needs to be completed.