Ported Ruby extension to upb_msg (#8184)

* WIP.

* WIP.

* WIP.

* WIP.

* WIP.

* WIP.

* Added some missing files.

* WIP.

* WIP.

* Updated upb.

* Extension loads, but crashes immediately.

* Gets through the test suite without SEGV!

Still a lot of bugs to fix, but it is a major step!

214 tests, 378 assertions, 37 failures, 147 errors, 0 pendings, 0 omissions, 0 notifications
14.0187% passed

* Test and build for Ruby 3.0

* Fixed a few more bugs, efficient #inspect is almost done.

214 tests, 134243 assertions, 30 failures, 144 errors, 0 pendings, 0 omissions, 0 notifications
18.6916% passed

* Fixed message hash initialization and encode depth checking.

214 tests, 124651 assertions, 53 failures, 70 errors, 0 pendings, 0 omissions, 0 notifications
42.5234% passed

* A bunch of fixes to failing tests, now 70% passing.

214 tests, 202091 assertions, 41 failures, 23 errors, 0 pendings, 0 omissions, 0 notifications
70.0935% passed

* More than 80% of tests are passing now.

214 tests, 322331 assertions, 30 failures, 9 errors, 0 pendings, 0 omissions, 0 notifications
81.7757% passed

Unfortunately there is also a sporadic bug/segfault hanging around
that appears to be GC-related.

* Add linux/ruby30 and macos/ruby30

* Use rvm master for 3.0.0-preview2

* Over 90% of tests are passing!

214 tests, 349898 assertions, 15 failures, 1 errors, 0 pendings, 0 omissions, 0 notifications
92.5234% passed

* Passes all tests!

214 tests, 369388 assertions, 0 failures, 0 errors, 0 pendings, 0 omissions, 0 notifications
100% passed

* A bunch of cleanup.

1. Removed a bunch of internal-only symbols from headers.
2. Required a frozen check to get a non-const pointer to a map or array.
3. De-duplicated the code to get a type argument for Map/RepeatedField.

* Removed a bunch more stuff from protobuf.h.  There is an intermittent assert failure.

Intermittent failure:

ruby: ../../../../ext/google/protobuf_c/protobuf.c:263: ObjectCache_Add: Assertion `rb_funcall(obj_cache2, (__builtin_constant_p("[]") ? __extension__ ({ static ID rb_intern_id_cache; if (!rb_intern_id_cache) rb_intern_id_cache = rb_intern2((("[]")
), (long)strlen(("[]"))); (ID) rb_intern_id_cache; }) : rb_intern("[]")), 1, key_rb) == val' failed

* Removed a few more things from protobuf.h.

* Ruby 3.0.0-preview2 to 3.0.0

* Require rake-compiler-dock >= 1.1.0

* More progress, fighting with the object cache.

* Passes on all Ruby versions!

* Updated and clarified comment regarding WeakMap.

* Fixed the wyhash compile.

* Fixed conformance tests for Ruby.

Conformance results now look like:

RUBYLIB=../ruby/lib:. ./conformance-test-runner --enforce_recommended --failure_list failure_list_ruby.txt --text_format_failure_list text_format_failure_list_ruby.txt ./conformance_ruby.rb

CONFORMANCE TEST BEGIN ====================================

CONFORMANCE SUITE PASSED: 1955 successes, 0 skipped, 58 expected failures, 0 unexpected failures.

CONFORMANCE TEST BEGIN ====================================

CONFORMANCE SUITE PASSED: 0 successes, 111 skipped, 8 expected failures, 0 unexpected failures.

Fixes include:

- Changed Ruby compiler to no longer reject proto2 maps.
- Changed Ruby compiler to emit a warning when proto2 extensions are
  present instead of rejecting the .proto file completely.
- Fixed conformance tests to allow proto2 and look up message by name
  instead of hardcoding a specific list of messages.
- Fixed conformance test to support the "ignore unknown" option for
  JSON.
- Fixed conformance test to properly report serialization errors.

* Removed debug printf and fixed #inspect for floats.

* Fixed compatibility test to have proper semantics for #to_json.

* Updated Makefile.am with new file list.

* Don't try to copy wyhash when inside Docker.

* Fixed bug where we would forget that a sub-object is frozen in Ruby >=2.7.

* Avoid exporting unneeded symbols and refactored a bit of code.

* Some more refactoring.

* Simplified and added more comments.

* Some more comments and simplification. Added a missing license block.

Co-authored-by: Masaki Hara <hara@wantedly.com>
43 files changed
tree: ef9511085eaaae2a233b1ce1d33d8a4d58098ac3
  1. .github/
  2. benchmarks/
  3. cmake/
  4. conformance/
  5. csharp/
  6. docs/
  7. editors/
  8. examples/
  9. java/
  10. js/
  11. kokoro/
  12. m4/
  13. objectivec/
  14. php/
  15. protoc-artifacts/
  16. python/
  17. ruby/
  18. src/
  19. third_party/
  20. util/
  21. .bazelignore
  22. .gitignore
  23. .gitmodules
  24. .readthedocs.yml
  25. appveyor.bat
  26. appveyor.yml
  27. autogen.sh
  28. BUILD
  29. build_files_updated_unittest.sh
  30. cc_proto_blacklist_test.bzl
  31. CHANGES.txt
  32. compiler_config_setting.bzl
  33. composer.json
  34. configure.ac
  35. CONTRIBUTING.md
  36. CONTRIBUTORS.txt
  37. fix_permissions.sh
  38. generate_changelog.py
  39. generate_descriptor_proto.sh
  40. global.json
  41. LICENSE
  42. Makefile.am
  43. post_process_dist.sh
  44. Protobuf-C++.podspec
  45. protobuf-lite.pc.in
  46. protobuf.bzl
  47. protobuf.pc.in
  48. Protobuf.podspec
  49. protobuf_deps.bzl
  50. README.md
  51. tests.sh
  52. update_compatibility_version.py
  53. update_file_lists.sh
  54. update_version.py
  55. WORKSPACE
README.md

Protocol Buffers - Google's data interchange format

Copyright 2008 Google Inc.

https://developers.google.com/protocol-buffers/

Overview

Protocol Buffers (a.k.a., protobuf) are Google's language-neutral, platform-neutral, extensible mechanism for serializing structured data. You can find protobuf's documentation on the Google Developers site.

This README file contains protobuf installation instructions. To install protobuf, you need to install the protocol compiler (used to compile .proto files) and the protobuf runtime for your chosen programming language.

Protocol Compiler Installation

The protocol compiler is written in C++. If you are using C++, please follow the C++ Installation Instructions to install protoc along with the C++ runtime.

For non-C++ users, the simplest way to install the protocol compiler is to download a pre-built binary from our release page:

https://github.com/protocolbuffers/protobuf/releases

In the downloads section of each release, you can find pre-built binaries in zip packages: protoc-$VERSION-$PLATFORM.zip. It contains the protoc binary as well as a set of standard .proto files distributed along with protobuf.

If you are looking for an old version that is not available in the release page, check out the maven repo here:

https://repo1.maven.org/maven2/com/google/protobuf/protoc/

These pre-built binaries are only provided for released versions. If you want to use the github master version at HEAD, or you need to modify protobuf code, or you are using C++, it's recommended to build your own protoc binary from source.

If you would like to build protoc binary from source, see the C++ Installation Instructions.

Protobuf Runtime Installation

Protobuf supports several different programming languages. For each programming language, you can find instructions in the corresponding source directory about how to install protobuf runtime for that specific language:

LanguageSourceUbuntuMacOSWindows
C++ (include C++ runtime and protoc)srcBuild status
Build status
Build status
Build status
Build status
Build status
JavajavaBuild status
Build status
Build status
Build status
PythonpythonBuild status
Build status
Build status
Build status
Build status
Build status
Build status
Build status
Build status
Build status
Build status
Build status
Build status
Build status
Objective-CobjectivecBuild status
Build status
Build status
Build status
C#csharpBuild statusBuild status
Build status
JavaScriptjsBuild statusBuild status
RubyrubyBuild status
Build status
Build status
Build status
Build status
Build status
Build status
Build status
Build status
Build status
Goprotocolbuffers/protobuf-go
PHPphpBuild status
Build status
Build status
Build status
Build status
Dartdart-lang/protobufBuild Status

Quick Start

The best way to learn how to use protobuf is to follow the tutorials in our developer guide:

https://developers.google.com/protocol-buffers/docs/tutorials

If you want to learn from code examples, take a look at the examples in the examples directory.

Documentation

The complete documentation for Protocol Buffers is available via the web at:

https://developers.google.com/protocol-buffers/