Ruby | Support installing the gem via git and some other small build tweaks (#21061) ## Overview This PR introduces a couple of small tweaks to the Ruby `google-protobuf` gem build. We've confirmed that the packaged `.gem` that gets created as part of `bazel build //ruby:release` is identical to that in the latest `google-protobuf` Ruby gem release. We also had an internal review of these changes, and if it's helpful, that can be found here: https://github.com/Shopify/protobuf/pull/12 ## Make the gem installable via git Ruby `Gemfile`s typically allow you to define a `git` source for gems to easily be able to test custom branches. However, given the following in a `Gemfile`: ``` # Gemfile gem "google-protobuf", git: "https://github.com/Shopify/protobuf.gif", branch: "master", submodules: true ``` we get the following build output when we try to `bundle install`: ``` Fetching https://github.com/Shopify/protobuf.git Fetching gem metadata from https://pkgs.shopify.io/basic/gems/ruby/... Fetching gem metadata from https://rubygems.org/...... Resolving dependencies... Resolving dependencies... Gem::Ext::BuildError: ERROR: Failed to build gem native extension. current directory: /Users/dave/.gem/ruby/3.4.1/bundler/gems/protobuf-8b63023562e0/ruby/ext/google/protobuf_c /opt/rubies/3.4.1/bin/ruby extconf.rb creating Makefile current directory: /Users/dave/.gem/ruby/3.4.1/bundler/gems/protobuf-8b63023562e0/ruby/ext/google/protobuf_c make DESTDIR\= sitearchdir\=./.gem.20250331-44786-nchu9i sitelibdir\=./.gem.20250331-44786-nchu9i clean current directory: /Users/dave/.gem/ruby/3.4.1/bundler/gems/protobuf-8b63023562e0/ruby/ext/google/protobuf_c make DESTDIR\= sitearchdir\=./.gem.20250331-44786-nchu9i sitelibdir\=./.gem.20250331-44786-nchu9i compiling protobuf.c In file included from protobuf.c:8: In file included from ./protobuf.h:23: In file included from ./defs.h:12: ./ruby-upb.h:1018:18: warning: implicit conversion loses integer precision: 'long' to 'int' [-Wshorten-64-to-32] 1018 | *overrun = ptr - e->end; | ~ ~~~~^~~~~~~~ ./ruby-upb.h:1151:64: warning: implicit conversion loses integer precision: 'size_t' (aka 'unsigned long') to 'int' [-Wshorten-64-to-32] 1151 | upb_EpsCopyInputStream_CheckDataSizeAvailable(e, ptr, size); | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ ^~~~ ./ruby-upb.h:1222:65: warning: implicit conversion loses integer precision: 'size_t' (aka 'unsigned long') to 'int' [-Wshorten-64-to-32] 1222 | if (!upb_EpsCopyInputStream_CheckDataSizeAvailable(e, *ptr, size)) { | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ ^~~~ ./ruby-upb.h:1228:66: warning: implicit conversion loses integer precision: 'size_t' (aka 'unsigned long') to 'int' [-Wshorten-64-to-32] 1228 | const char* ret = upb_EpsCopyInputStream_Copy(e, *ptr, data, size); | ~~~~~~~~~~~~~~~~~~~~~~~~~~~ ^~~~ ./ruby-upb.h:1314:27: warning: implicit conversion loses integer precision: 'long' to 'int' [-Wshorten-64-to-32] 1314 | e->limit = e->limit_ptr - e->end; | ~ ~~~~~~~~~~~~~^~~~~~~~ ./ruby-upb.h:1390:19: warning: implicit conversion loses integer precision: 'size_t' (aka 'unsigned long') to 'int' [-Wshorten-64-to-32] 1390 | return a.size - b.size; | ~~~~~~ ~~~~~~~^~~~~~~~ ./ruby-upb.h:4358:19: warning: implicit conversion loses integer precision: 'size_t' (aka 'unsigned long') to 'int' [-Wshorten-64-to-32] 4357 | array = UPB_PRIVATE(_upb_Array_New)( | ~~~~~~~~~~~~~~~~~~~~~~~~~~~ 4358 | arena, 4, UPB_PRIVATE(_upb_MiniTableField_ElemSizeLg2)(f)); | ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ ./ruby-upb.h:275:24: note: expanded from macro 'UPB_PRIVATE' 275 | #define UPB_PRIVATE(x) x##_dont_copy_me__upb_internal_use_only | ^ <scratch space>:19:1: note: expanded from here 19 | _upb_MiniTableField_ElemSizeLg2_dont_copy_me__upb_internal_use_only | ^ In file included from protobuf.c:8: In file included from ./protobuf.h:23: In file included from ./defs.h:12: ./ruby-upb.h:14850:10: warning: implicit conversion loses integer precision: 'uint64_t' (aka 'unsigned long long') to 'uint32_t' (aka 'unsigned int') [-Wshorten-64-to-32] 14850 | *tag = val; | ~ ^~~ ./ruby-upb.h:14886:11: warning: implicit conversion loses integer precision: 'uint64_t' (aka 'unsigned long long') to 'int' [-Wshorten-64-to-32] 14886 | *size = size64; | ~ ^~~~~~ ./ruby-upb.h:15299:10: fatal error: 'utf8_range.h' file not found 15299 | #include "utf8_range.h" | ^~~~~~~~~~~~~~ 9 warnings and 1 error generated. make: *** [protobuf.o] Error 1 make failed, exit code 2 Gem files will remain installed in /Users/dave/.gem/ruby/3.4.1/bundler/gems/protobuf-8b63023562e0/ruby for inspection. Results logged to /Users/dave/.gem/ruby/3.4.1/bundler/gems/extensions/arm64-darwin-23/3.4.0/protobuf-8b63023562e0-ruby/gem_make.out /opt/rubies/3.4.1/lib/ruby/3.4.0/rubygems/ext/builder.rb:126:in 'Gem::Ext::Builder.run' /opt/rubies/3.4.1/lib/ruby/3.4.0/rubygems/ext/builder.rb:52:in 'block in Gem::Ext::Builder.make' /opt/rubies/3.4.1/lib/ruby/3.4.0/rubygems/ext/builder.rb:44:in 'Array#each' /opt/rubies/3.4.1/lib/ruby/3.4.0/rubygems/ext/builder.rb:44:in 'Gem::Ext::Builder.make' /opt/rubies/3.4.1/lib/ruby/3.4.0/rubygems/ext/ext_conf_builder.rb:44:in 'Gem::Ext::ExtConfBuilder.build' /opt/rubies/3.4.1/lib/ruby/3.4.0/rubygems/ext/builder.rb:195:in 'Gem::Ext::Builder#build_extension' /opt/rubies/3.4.1/lib/ruby/3.4.0/rubygems/ext/builder.rb:229:in 'block in Gem::Ext::Builder#build_extensions' /opt/rubies/3.4.1/lib/ruby/3.4.0/rubygems/ext/builder.rb:226:in 'Array#each' /opt/rubies/3.4.1/lib/ruby/3.4.0/rubygems/ext/builder.rb:226:in 'Gem::Ext::Builder#build_extensions' /opt/rubies/3.4.1/lib/ruby/3.4.0/rubygems/installer.rb:844:in 'Gem::Installer#build_extensions' /Users/dave/.gem/ruby/3.4.1/gems/bundler-2.6.3/lib/bundler/rubygems_gem_installer.rb:111:in 'Bundler::RubyGemsGemInstaller#build_extensions' /Users/dave/.gem/ruby/3.4.1/gems/bundler-2.6.3/lib/bundler/source/path/installer.rb:28:in 'Bundler::Source::Path::Installer#post_install' /Users/dave/.gem/ruby/3.4.1/gems/bundler-2.6.3/lib/bundler/source/path.rb:234:in 'Bundler::Source::Path#generate_bin' /Users/dave/.gem/ruby/3.4.1/gems/bundler-2.6.3/lib/bundler/source/git.rb:212:in 'Bundler::Source::Git#install' /Users/dave/.gem/ruby/3.4.1/gems/bundler-2.6.3/lib/bundler/installer/gem_installer.rb:55:in 'Bundler::GemInstaller#install' /Users/dave/.gem/ruby/3.4.1/gems/bundler-2.6.3/lib/bundler/installer/gem_installer.rb:17:in 'Bundler::GemInstaller#install_from_spec' /Users/dave/.gem/ruby/3.4.1/gems/bundler-2.6.3/lib/bundler/installer/parallel_installer.rb:133:in 'Bundler::ParallelInstaller#do_install' /Users/dave/.gem/ruby/3.4.1/gems/bundler-2.6.3/lib/bundler/installer/parallel_installer.rb:124:in 'block in Bundler::ParallelInstaller#worker_pool' /Users/dave/.gem/ruby/3.4.1/gems/bundler-2.6.3/lib/bundler/worker.rb:62:in 'Bundler::Worker#apply_func' /Users/dave/.gem/ruby/3.4.1/gems/bundler-2.6.3/lib/bundler/worker.rb:57:in 'block in Bundler::Worker#process_queue' <internal:kernel>:168:in 'Kernel#loop' /Users/dave/.gem/ruby/3.4.1/gems/bundler-2.6.3/lib/bundler/worker.rb:54:in 'Bundler::Worker#process_queue' /Users/dave/.gem/ruby/3.4.1/gems/bundler-2.6.3/lib/bundler/worker.rb:90:in 'block (2 levels) in Bundler::Worker#create_threads' An error occurred while installing google-protobuf (4.31.0), and Bundler cannot continue. In Gemfile: grpc was resolved to 1.71.0, which depends on googleapis-common-protos-types was resolved to 1.19.0, which depends on google-protobuf ``` This is because building the `protobuf_c` Ruby C extension requires the `copy_third_party` rake task to copy the `third_party/utf8_range` library into `ruby/ext/google/protobuf_c`. Fortunately, the `extensions` array in a `Gemfile` can accept either a relative path to an `extconf.rb`, or a `Rakefile`. In the case of the latter, it runs the default `rake`, which in the case of `google-protobuf` is what we need to successfully build the native bindings. The `bazel` build process takes care of these steps which is why `bazel build //ruby:release` works as expected. However, this `Rakefile` contains tasks that reference files outside of `ruby/`, which won't exist when installing the packaged `.gem` (the artifact that gets published to rubygems.org). The changes here will conditionally set the relevant `extensions` entry to either `ext/google/protobuf_c/extconf.rb` or `Rakefile` depending on the broader build context. This also required a small tweak in the `Rakefile` to modify the `Gem::PackageTask`. ## Missing `google/protobuf/plugin_pb.rb` in `rake gem` artifact Another issue we noticed is the compiled `src/google/protobuf/compiler/plugin.proto` is absent in the packaged `.gem` that gets outputted when following the [Ruby development build instructions](https://github.com/protocolbuffers/protobuf/tree/main/ruby#installation-from-source-building-gem). This is because this particular proto file was absent from the `well_known_protos` definition in the `Rakefile`. This adds that missing proto to the list of well known protos and makes a small tweak to ensure it ends up in the gem's `lib/google/protobuf` directory instead of `lib/google/protobuf/compiler`. I'm not entirely sure why that directory tree gets a bit flattened, but we wanted to ensure the output remained consistent. Again, this is another issue that the `bazel` build takes care of. ## Tweak `ruby/BUILD.bazel` to also work on macOS Finally, this PR introduces a small change to `ruby/BUILD.bazel` so `bazel build //ruby:release` can run on macOS as well as Linux. `cp --parents` is not supported in BSD's it seems. Closes #21061 COPYBARA_INTEGRATE_REVIEW=https://github.com/protocolbuffers/protobuf/pull/21061 from Shopify:ruby-installable-from-git e6b1f0cdad6a42952cd5aa4f6b185fcebaf50191 PiperOrigin-RevId: 744700849
Copyright 2023 Google LLC
Protocol Buffers (a.k.a., protobuf) are Google's language-neutral, platform-neutral, extensible mechanism for serializing structured data. You can learn more about it in protobuf's documentation.
This README file contains protobuf installation instructions. To install protobuf, you need to install the protocol compiler (used to compile .proto files) and the protobuf runtime for your chosen programming language.
Most users will find working from supported releases to be the easiest path.
If you choose to work from the head revision of the main branch your build will occasionally be broken by source-incompatible changes and insufficiently-tested (and therefore broken) behavior.
If you are using C++ or otherwise need to build protobuf from source as a part of your project, you should pin to a release commit on a release branch.
This is because even release branches can experience some instability in between release commits.
Protobuf supports Bzlmod with Bazel 7 +. Users should specify a dependency on protobuf in their MODULE.bazel file as follows.
bazel_dep(name = "protobuf", version = <VERSION>)
Users can optionally override the repo name, such as for compatibility with WORKSPACE.
bazel_dep(name = "protobuf", version = <VERSION>, repo_name = "com_google_protobuf")
Users can also add the following to their legacy WORKSPACE file.
Note that with the release of 30.x there are a few more load statements to properly set up rules_java and rules_python.
http_archive(
name = "com_google_protobuf",
strip_prefix = "protobuf-VERSION",
sha256 = ...,
url = ...,
)
load("@com_google_protobuf//:protobuf_deps.bzl", "protobuf_deps")
protobuf_deps()
load("@rules_java//java:rules_java_deps.bzl", "rules_java_dependencies")
rules_java_dependencies()
load("@rules_java//java:repositories.bzl", "rules_java_toolchains")
rules_java_toolchains()
load("@rules_python//python:repositories.bzl", "py_repositories")
py_repositories()
The protobuf compiler is written in C++. If you are using C++, please follow the C++ Installation Instructions to install protoc along with the C++ runtime.
For non-C++ users, the simplest way to install the protocol compiler is to download a pre-built binary from our GitHub release page.
In the downloads section of each release, you can find pre-built binaries in zip packages: protoc-$VERSION-$PLATFORM.zip. It contains the protoc binary as well as a set of standard .proto files distributed along with protobuf.
If you are looking for an old version that is not available in the release page, check out the Maven repository.
These pre-built binaries are only provided for released versions. If you want to use the github main version at HEAD, or you need to modify protobuf code, or you are using C++, it's recommended to build your own protoc binary from source.
If you would like to build protoc binary from source, see the C++ Installation Instructions.
Protobuf supports several different programming languages. For each programming language, you can find instructions in the corresponding source directory about how to install protobuf runtime for that specific language:
| Language | Source |
|---|---|
| C++ (include C++ runtime and protoc) | src |
| Java | java |
| Python | python |
| Objective-C | objectivec |
| C# | csharp |
| Ruby | ruby |
| Go | protocolbuffers/protobuf-go |
| PHP | php |
| Dart | dart-lang/protobuf |
| JavaScript | protocolbuffers/protobuf-javascript |
The best way to learn how to use protobuf is to follow the tutorials in our developer guide.
If you want to learn from code examples, take a look at the examples in the examples directory.
The complete documentation is available at the Protocol Buffers doc site.
Read about our version support policy to stay current on support timeframes for the language libraries.
To be alerted to upcoming changes in Protocol Buffers and connect with protobuf developers and users, join the Google Group.