tree 338f9e12aca2a492ddfed7c397a9a0887b795f33
parent 4ce6d364131bec88f93739c5f32608b2d27583c1
author Dave Benvenuti <davebenvenuti@gmail.com> 1744032544 -0700
committer Copybara-Service <copybara-worker@google.com> 1744032704 -0700

Ruby | Support installing the gem via git and some other small build tweaks (#21061)

## Overview

This PR introduces a couple of small tweaks to the Ruby `google-protobuf` gem build.  We've confirmed that the packaged `.gem` that gets created as part of `bazel build //ruby:release` is identical to that in the latest `google-protobuf` Ruby gem release.  We also had an internal review of these changes, and if it's helpful, that can be found here: https://github.com/Shopify/protobuf/pull/12

## Make the gem installable via git

Ruby `Gemfile`s typically allow you to define a `git` source for gems to easily be able to test custom branches.  However, given the following in a `Gemfile`:

```
# Gemfile
gem "google-protobuf", git: "https://github.com/Shopify/protobuf.gif", branch: "master", submodules: true
```

we get the following build output when we try to `bundle install`:

```
Fetching https://github.com/Shopify/protobuf.git
Fetching gem metadata from https://pkgs.shopify.io/basic/gems/ruby/...
Fetching gem metadata from https://rubygems.org/......
Resolving dependencies...
Resolving dependencies...
Gem::Ext::BuildError: ERROR: Failed to build gem native extension.

    current directory: /Users/dave/.gem/ruby/3.4.1/bundler/gems/protobuf-8b63023562e0/ruby/ext/google/protobuf_c
/opt/rubies/3.4.1/bin/ruby extconf.rb
creating Makefile

current directory: /Users/dave/.gem/ruby/3.4.1/bundler/gems/protobuf-8b63023562e0/ruby/ext/google/protobuf_c
make DESTDIR\= sitearchdir\=./.gem.20250331-44786-nchu9i sitelibdir\=./.gem.20250331-44786-nchu9i clean

current directory: /Users/dave/.gem/ruby/3.4.1/bundler/gems/protobuf-8b63023562e0/ruby/ext/google/protobuf_c
make DESTDIR\= sitearchdir\=./.gem.20250331-44786-nchu9i sitelibdir\=./.gem.20250331-44786-nchu9i
compiling protobuf.c
In file included from protobuf.c:8:
In file included from ./protobuf.h:23:
In file included from ./defs.h:12:
./ruby-upb.h:1018:18: warning: implicit conversion loses integer precision: 'long' to 'int' [-Wshorten-64-to-32]
 1018 |   *overrun = ptr - e->end;
      |            ~ ~~~~^~~~~~~~
./ruby-upb.h:1151:64: warning: implicit conversion loses integer precision: 'size_t' (aka 'unsigned long') to 'int' [-Wshorten-64-to-32]
 1151 |          upb_EpsCopyInputStream_CheckDataSizeAvailable(e, ptr, size);
      |          ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~         ^~~~
./ruby-upb.h:1222:65: warning: implicit conversion loses integer precision: 'size_t' (aka 'unsigned long') to 'int' [-Wshorten-64-to-32]
 1222 |     if (!upb_EpsCopyInputStream_CheckDataSizeAvailable(e, *ptr, size)) {
      |          ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~          ^~~~
./ruby-upb.h:1228:66: warning: implicit conversion loses integer precision: 'size_t' (aka 'unsigned long') to 'int' [-Wshorten-64-to-32]
 1228 |     const char* ret = upb_EpsCopyInputStream_Copy(e, *ptr, data, size);
      |                       ~~~~~~~~~~~~~~~~~~~~~~~~~~~                ^~~~
./ruby-upb.h:1314:27: warning: implicit conversion loses integer precision: 'long' to 'int' [-Wshorten-64-to-32]
 1314 |   e->limit = e->limit_ptr - e->end;
      |            ~ ~~~~~~~~~~~~~^~~~~~~~
./ruby-upb.h:1390:19: warning: implicit conversion loses integer precision: 'size_t' (aka 'unsigned long') to 'int' [-Wshorten-64-to-32]
 1390 |     return a.size - b.size;
      |     ~~~~~~ ~~~~~~~^~~~~~~~
./ruby-upb.h:4358:19: warning: implicit conversion loses integer precision: 'size_t' (aka 'unsigned long') to 'int' [-Wshorten-64-to-32]
 4357 |     array = UPB_PRIVATE(_upb_Array_New)(
      |             ~~~~~~~~~~~~~~~~~~~~~~~~~~~
 4358 |         arena, 4, UPB_PRIVATE(_upb_MiniTableField_ElemSizeLg2)(f));
      |                   ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
./ruby-upb.h:275:24: note: expanded from macro 'UPB_PRIVATE'
  275 | #define UPB_PRIVATE(x) x##_dont_copy_me__upb_internal_use_only
      |                        ^
<scratch space>:19:1: note: expanded from here
   19 | _upb_MiniTableField_ElemSizeLg2_dont_copy_me__upb_internal_use_only
      | ^
In file included from protobuf.c:8:
In file included from ./protobuf.h:23:
In file included from ./defs.h:12:
./ruby-upb.h:14850:10: warning: implicit conversion loses integer precision: 'uint64_t' (aka 'unsigned long long') to 'uint32_t' (aka 'unsigned int') [-Wshorten-64-to-32]
 14850 |   *tag = val;
       |        ~ ^~~
./ruby-upb.h:14886:11: warning: implicit conversion loses integer precision: 'uint64_t' (aka 'unsigned long long') to 'int' [-Wshorten-64-to-32]
 14886 |   *size = size64;
       |         ~ ^~~~~~
./ruby-upb.h:15299:10: fatal error: 'utf8_range.h' file not found
 15299 | #include "utf8_range.h"
       |          ^~~~~~~~~~~~~~
9 warnings and 1 error generated.
make: *** [protobuf.o] Error 1

make failed, exit code 2

Gem files will remain installed in /Users/dave/.gem/ruby/3.4.1/bundler/gems/protobuf-8b63023562e0/ruby for inspection.
Results logged to /Users/dave/.gem/ruby/3.4.1/bundler/gems/extensions/arm64-darwin-23/3.4.0/protobuf-8b63023562e0-ruby/gem_make.out

  /opt/rubies/3.4.1/lib/ruby/3.4.0/rubygems/ext/builder.rb:126:in 'Gem::Ext::Builder.run'
  /opt/rubies/3.4.1/lib/ruby/3.4.0/rubygems/ext/builder.rb:52:in 'block in Gem::Ext::Builder.make'
  /opt/rubies/3.4.1/lib/ruby/3.4.0/rubygems/ext/builder.rb:44:in 'Array#each'
  /opt/rubies/3.4.1/lib/ruby/3.4.0/rubygems/ext/builder.rb:44:in 'Gem::Ext::Builder.make'
  /opt/rubies/3.4.1/lib/ruby/3.4.0/rubygems/ext/ext_conf_builder.rb:44:in 'Gem::Ext::ExtConfBuilder.build'
  /opt/rubies/3.4.1/lib/ruby/3.4.0/rubygems/ext/builder.rb:195:in 'Gem::Ext::Builder#build_extension'
  /opt/rubies/3.4.1/lib/ruby/3.4.0/rubygems/ext/builder.rb:229:in 'block in Gem::Ext::Builder#build_extensions'
  /opt/rubies/3.4.1/lib/ruby/3.4.0/rubygems/ext/builder.rb:226:in 'Array#each'
  /opt/rubies/3.4.1/lib/ruby/3.4.0/rubygems/ext/builder.rb:226:in 'Gem::Ext::Builder#build_extensions'
  /opt/rubies/3.4.1/lib/ruby/3.4.0/rubygems/installer.rb:844:in 'Gem::Installer#build_extensions'
  /Users/dave/.gem/ruby/3.4.1/gems/bundler-2.6.3/lib/bundler/rubygems_gem_installer.rb:111:in 'Bundler::RubyGemsGemInstaller#build_extensions'
  /Users/dave/.gem/ruby/3.4.1/gems/bundler-2.6.3/lib/bundler/source/path/installer.rb:28:in 'Bundler::Source::Path::Installer#post_install'
  /Users/dave/.gem/ruby/3.4.1/gems/bundler-2.6.3/lib/bundler/source/path.rb:234:in 'Bundler::Source::Path#generate_bin'
  /Users/dave/.gem/ruby/3.4.1/gems/bundler-2.6.3/lib/bundler/source/git.rb:212:in 'Bundler::Source::Git#install'
  /Users/dave/.gem/ruby/3.4.1/gems/bundler-2.6.3/lib/bundler/installer/gem_installer.rb:55:in 'Bundler::GemInstaller#install'
  /Users/dave/.gem/ruby/3.4.1/gems/bundler-2.6.3/lib/bundler/installer/gem_installer.rb:17:in 'Bundler::GemInstaller#install_from_spec'
  /Users/dave/.gem/ruby/3.4.1/gems/bundler-2.6.3/lib/bundler/installer/parallel_installer.rb:133:in 'Bundler::ParallelInstaller#do_install'
  /Users/dave/.gem/ruby/3.4.1/gems/bundler-2.6.3/lib/bundler/installer/parallel_installer.rb:124:in 'block in Bundler::ParallelInstaller#worker_pool'
  /Users/dave/.gem/ruby/3.4.1/gems/bundler-2.6.3/lib/bundler/worker.rb:62:in 'Bundler::Worker#apply_func'
  /Users/dave/.gem/ruby/3.4.1/gems/bundler-2.6.3/lib/bundler/worker.rb:57:in 'block in Bundler::Worker#process_queue'
  <internal:kernel>:168:in 'Kernel#loop'
  /Users/dave/.gem/ruby/3.4.1/gems/bundler-2.6.3/lib/bundler/worker.rb:54:in 'Bundler::Worker#process_queue'
  /Users/dave/.gem/ruby/3.4.1/gems/bundler-2.6.3/lib/bundler/worker.rb:90:in 'block (2 levels) in Bundler::Worker#create_threads'

An error occurred while installing google-protobuf (4.31.0), and Bundler cannot continue.

In Gemfile:
  grpc was resolved to 1.71.0, which depends on
    googleapis-common-protos-types was resolved to 1.19.0, which depends on
      google-protobuf
```

This is because building the `protobuf_c` Ruby C extension requires the `copy_third_party` rake task to copy the `third_party/utf8_range` library into `ruby/ext/google/protobuf_c`.  Fortunately, the `extensions` array in a `Gemfile` can accept either a relative path to an `extconf.rb`, or a `Rakefile`.  In the case of the latter, it runs the default `rake`, which in the case of `google-protobuf` is what we need to successfully build the native bindings.  The `bazel` build process takes care of these steps which is why `bazel build //ruby:release` works as expected.

However, this `Rakefile` contains tasks that reference files outside of `ruby/`, which won't exist when installing the packaged `.gem` (the artifact that gets published to rubygems.org).  The changes here will conditionally set the relevant `extensions` entry to either `ext/google/protobuf_c/extconf.rb` or `Rakefile` depending on the broader build context.  This also required a small tweak in the `Rakefile` to modify the `Gem::PackageTask`.

## Missing `google/protobuf/plugin_pb.rb` in `rake gem` artifact

Another issue we noticed is the compiled `src/google/protobuf/compiler/plugin.proto` is absent in the packaged `.gem` that gets outputted when following the [Ruby development build instructions](https://github.com/protocolbuffers/protobuf/tree/main/ruby#installation-from-source-building-gem).  This is because this particular proto file was absent from the `well_known_protos` definition in the `Rakefile`.  This adds that missing proto to the list of well known protos and makes a small tweak to ensure it ends up in the gem's `lib/google/protobuf` directory instead of `lib/google/protobuf/compiler`.  I'm not entirely sure why that directory tree gets a bit flattened, but we wanted to ensure the output remained consistent.  Again, this is another issue that the `bazel` build takes care of.

## Tweak `ruby/BUILD.bazel` to also work on macOS

Finally, this PR introduces a small change to `ruby/BUILD.bazel` so `bazel build //ruby:release` can run on macOS as well as Linux.  `cp --parents` is not supported in BSD's it seems.

Closes #21061

COPYBARA_INTEGRATE_REVIEW=https://github.com/protocolbuffers/protobuf/pull/21061 from Shopify:ruby-installable-from-git e6b1f0cdad6a42952cd5aa4f6b185fcebaf50191
PiperOrigin-RevId: 744700849