Merge pull request #5109 from haberman/integrate
Down-integrate from google3.
diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md
new file mode 100644
index 0000000..49b1941
--- /dev/null
+++ b/CONTRIBUTING.md
@@ -0,0 +1,85 @@
+# Contributing to Protocol Buffers
+
+We welcome your contributions to protocol buffers. This doc describes the
+process to contribute patches to protobuf and the general guidelines we
+expect contributors to follow.
+
+## Before You Start
+
+We accept patches in the form of github pull requests. If you are new to
+github, please read [How to create github pull requests](https://help.github.com/articles/about-pull-requests/)
+first.
+
+### Contributor License Agreements
+
+Contributions to this project must be accompanied by a Contributor License
+Agreement. You (or your employer) retain the copyright to your contribution,
+this simply gives us permission to use and redistribute your contributions
+as part of the project.
+
+* If you are an individual writing original source code and you're sure you
+ own the intellectual property, then you'll need to sign an [individual CLA](https://cla.developers.google.com/about/google-individual?csw=1).
+* If you work for a company that wants to allow you to contribute your work,
+ then you'll need to sign a [corporate CLA](https://cla.developers.google.com/about/google-corporate?csw=1).
+
+### Coding Style
+
+This project follows [Google’s Coding Style Guides](https://github.com/google/styleguide).
+Before sending out your pull request, please familiarize yourself with the
+corresponding style guides and make sure the proposed code change is style
+conforming.
+
+## Contributing Process
+
+Most pull requests should go to the master branch and the change will be
+included in the next major/minor version release (e.g., 3.6.0 release). If you
+need to include a bug fix in a patch release (e.g., 3.5.2), make sure it’s
+already merged to master, and then create a pull request cherry-picking the
+commits from master branch to the release branch (e.g., branch 3.5.x).
+
+For each pull request, a protobuf team member will be assigned to review the
+pull request. For minor cleanups, the pull request may be merged right away
+after an initial review. For larger changes, you will likely receive multiple
+rounds of comments and it may take some time to complete. We will try to keep
+our response time within 7-days but if you don’t get any response in a few
+days, feel free to comment on the threads to get our attention. We also expect
+you to respond to our comments within a reasonable amount of time. If we don’t
+hear from you for 2 weeks or longer, we may close the pull request. You can
+still send the pull request again once you have time to work on it.
+
+Once a pull request is merged, we will take care of the rest and get it into
+the final release.
+
+## Pull Request Guidelines
+
+* If you are a Googler, it is preferable to first create an internal CL and
+ have it reviewed and submitted. The code propagation process will deliver the
+ change to GitHub.
+* Create small PRs that are narrowly focused on addressing a single concern.
+ We often receive PRs that are trying to fix several things at a time, but if
+ only one fix is considered acceptable, nothing gets merged and both author's
+ & review's time is wasted. Create more PRs to address different concerns and
+ everyone will be happy.
+* For speculative changes, consider opening an issue and discussing it first.
+ If you are suggesting a behavioral or API change, make sure you get explicit
+ support from a protobuf team member before sending us the pull request.
+* Provide a good PR description as a record of what change is being made and
+ why it was made. Link to a GitHub issue if it exists.
+* Don't fix code style and formatting unless you are already changing that
+ line to address an issue. PRs with irrelevant changes won't be merged. If
+ you do want to fix formatting or style, do that in a separate PR.
+* Unless your PR is trivial, you should expect there will be reviewer comments
+ that you'll need to address before merging. We expect you to be reasonably
+ responsive to those comments, otherwise the PR will be closed after 2-3 weeks
+ of inactivity.
+* Maintain clean commit history and use meaningful commit messages. PRs with
+ messy commit history are difficult to review and won't be merged. Use rebase
+ -i upstream/master to curate your commit history and/or to bring in latest
+ changes from master (but avoid rebasing in the middle of a code review).
+* Keep your PR up to date with upstream/master (if there are merge conflicts,
+ we can't really merge your change).
+* All tests need to be passing before your change can be merged. We recommend
+ you run tests locally before creating your PR to catch breakages early on.
+ Ultimately, the green signal will be provided by our testing infrastructure.
+ The reviewer will help you if there are test failures that seem not related
+ to the change you are making.
diff --git a/Makefile.am b/Makefile.am
index 6694dff..b0913a5 100644
--- a/Makefile.am
+++ b/Makefile.am
@@ -1043,6 +1043,7 @@
WORKSPACE \
cmake/CMakeLists.txt \
cmake/README.md \
+ cmake/conformance.cmake \
cmake/examples.cmake \
cmake/extract_includes.bat.in \
cmake/install.cmake \
diff --git a/cmake/CMakeLists.txt b/cmake/CMakeLists.txt
index ece39f7..06436c3 100644
--- a/cmake/CMakeLists.txt
+++ b/cmake/CMakeLists.txt
@@ -30,6 +30,7 @@
# Options
option(protobuf_BUILD_TESTS "Build tests" ON)
+option(protobuf_BUILD_CONFORMANCE "Build conformance tests" OFF)
option(protobuf_BUILD_EXAMPLES "Build examples" OFF)
option(protobuf_BUILD_PROTOC_BINARIES "Build libprotoc and protoc compiler" ON)
if (BUILD_SHARED_LIBS)
@@ -218,6 +219,10 @@
include(tests.cmake)
endif (protobuf_BUILD_TESTS)
+if (protobuf_BUILD_CONFORMANCE)
+ include(conformance.cmake)
+endif (protobuf_BUILD_CONFORMANCE)
+
include(install.cmake)
if (protobuf_BUILD_EXAMPLES)
diff --git a/cmake/conformance.cmake b/cmake/conformance.cmake
new file mode 100644
index 0000000..ec1f9e0
--- /dev/null
+++ b/cmake/conformance.cmake
@@ -0,0 +1,48 @@
+
+add_custom_command(
+ OUTPUT ${protobuf_source_dir}/conformance/conformance.pb.cc
+ DEPENDS protoc ${protobuf_source_dir}/conformance/conformance.proto
+ COMMAND protoc ${protobuf_source_dir}/conformance/conformance.proto
+ --proto_path=${protobuf_source_dir}/conformance
+ --cpp_out=${protobuf_source_dir}/conformance
+)
+
+add_custom_command(
+ OUTPUT ${protobuf_source_dir}/src/google/protobuf/test_messages_proto3.pb.cc
+ ${protobuf_source_dir}/src/google/protobuf/test_messages_proto2.pb.cc
+ DEPENDS protoc ${protobuf_source_dir}/src/google/protobuf/test_messages_proto3.proto
+ protoc ${protobuf_source_dir}/src/google/protobuf/test_messages_proto2.proto
+ COMMAND protoc ${protobuf_source_dir}/src/google/protobuf/test_messages_proto3.proto
+ ${protobuf_source_dir}/src/google/protobuf/test_messages_proto2.proto
+ --proto_path=${protobuf_source_dir}/src
+ --cpp_out=${protobuf_source_dir}/src
+)
+
+add_executable(conformance_test_runner
+ ${protobuf_source_dir}/conformance/conformance.pb.cc
+ ${protobuf_source_dir}/conformance/conformance_test.cc
+ ${protobuf_source_dir}/conformance/conformance_test_impl.cc
+ ${protobuf_source_dir}/conformance/conformance_test_runner.cc
+ ${protobuf_source_dir}/conformance/third_party/jsoncpp/json.h
+ ${protobuf_source_dir}/conformance/third_party/jsoncpp/jsoncpp.cpp
+ ${protobuf_source_dir}/src/google/protobuf/test_messages_proto3.pb.cc
+ ${protobuf_source_dir}/src/google/protobuf/test_messages_proto2.pb.cc
+)
+
+add_executable(conformance_cpp
+ ${protobuf_source_dir}/conformance/conformance.pb.cc
+ ${protobuf_source_dir}/conformance/conformance_cpp.cc
+ ${protobuf_source_dir}/src/google/protobuf/test_messages_proto3.pb.cc
+ ${protobuf_source_dir}/src/google/protobuf/test_messages_proto2.pb.cc
+)
+
+target_include_directories(
+ conformance_test_runner
+ PUBLIC ${protobuf_source_dir}/conformance)
+
+target_include_directories(
+ conformance_cpp
+ PUBLIC ${protobuf_source_dir}/conformance)
+
+target_link_libraries(conformance_test_runner libprotobuf)
+target_link_libraries(conformance_cpp libprotobuf)
diff --git a/conformance/Makefile.am b/conformance/Makefile.am
index e51ab80..29d9a98 100644
--- a/conformance/Makefile.am
+++ b/conformance/Makefile.am
@@ -206,6 +206,7 @@
conformance_test_runner_LDADD = $(top_srcdir)/src/libprotobuf.la
conformance_test_runner_SOURCES = conformance_test.h conformance_test.cc \
+ conformance_test_impl.cc \
conformance_test_runner.cc \
third_party/jsoncpp/json.h \
third_party/jsoncpp/jsoncpp.cpp
diff --git a/conformance/conformance_test.cc b/conformance/conformance_test.cc
index 0a45f9b..2d822c2 100644
--- a/conformance/conformance_test.cc
+++ b/conformance/conformance_test.cc
@@ -57,34 +57,6 @@
namespace google {
namespace protobuf {
-std::set<ConformanceTestSuite*> *conformance_test_suite_set;
-GOOGLE_PROTOBUF_DECLARE_ONCE(conformance_test_suite_set_init_);
-
-void DeleteConformanceTestSuiteSet() {
- delete conformance_test_suite_set;
-}
-
-static void InitConformanceTestSuiteSet() {
- conformance_test_suite_set = new std::set<ConformanceTestSuite*>();
- internal::OnShutdown(&DeleteConformanceTestSuiteSet);
-}
-
-static void InitConformanceTestSuiteSetOnce() {
- ::google::protobuf::GoogleOnceInit(
- &conformance_test_suite_set_init_,
- &InitConformanceTestSuiteSet);
-}
-
-void AddTestSuite(ConformanceTestSuite* suite) {
- InitConformanceTestSuiteSetOnce();
- conformance_test_suite_set->insert(suite);
-}
-
-const std::set<ConformanceTestSuite*>& GetTestSuiteSet() {
- InitConformanceTestSuiteSetOnce();
- return *conformance_test_suite_set;
-}
-
ConformanceTestSuite::ConformanceRequestSetting::ConformanceRequestSetting(
ConformanceLevel level,
conformance::WireFormat input_format,
diff --git a/conformance/conformance_test.h b/conformance/conformance_test.h
index 8738735..d5c2f3d 100644
--- a/conformance/conformance_test.h
+++ b/conformance/conformance_test.h
@@ -62,6 +62,8 @@
namespace google {
namespace protobuf {
+class ConformanceTestSuite;
+
class ConformanceTestRunner {
public:
virtual ~ConformanceTestRunner() {}
@@ -78,6 +80,54 @@
std::string* output) = 0;
};
+// Test runner that spawns the process being tested and communicates with it
+// over a pipe.
+class ForkPipeRunner : public ConformanceTestRunner {
+ public:
+ static int Run(int argc, char *argv[],
+ ConformanceTestSuite* suite);
+
+ private:
+ ForkPipeRunner(const std::string &executable)
+ : child_pid_(-1), executable_(executable) {}
+
+ virtual ~ForkPipeRunner() {}
+
+ void RunTest(const std::string& test_name,
+ const std::string& request,
+ std::string* response);
+
+ // TODO(haberman): make this work on Windows, instead of using these
+ // UNIX-specific APIs.
+ //
+ // There is a platform-agnostic API in
+ // src/google/protobuf/compiler/subprocess.h
+ //
+ // However that API only supports sending a single message to the subprocess.
+ // We really want to be able to send messages and receive responses one at a
+ // time:
+ //
+ // 1. Spawning a new process for each test would take way too long for thousands
+ // of tests and subprocesses like java that can take 100ms or more to start
+ // up.
+ //
+ // 2. Sending all the tests in one big message and receiving all results in one
+ // big message would take away our visibility about which test(s) caused a
+ // crash or other fatal error. It would also give us only a single failure
+ // instead of all of them.
+ void SpawnTestProgram();
+
+ void CheckedWrite(int fd, const void *buf, size_t len);
+ bool TryRead(int fd, void *buf, size_t len);
+ void CheckedRead(int fd, void *buf, size_t len);
+
+ int write_fd_;
+ int read_fd_;
+ pid_t child_pid_;
+ std::string executable_;
+ std::string current_test_name_;
+};
+
// Class representing the test suite itself. To run it, implement your own
// class derived from ConformanceTestRunner, class derived from
// ConformanceTestSuite and then write code like:
@@ -89,28 +139,20 @@
// }
// };
//
-// // Force MyConformanceTestSuite to be added at dynamic initialization
-// // time.
-// struct StaticTestSuiteInitializer {
-// StaticTestSuiteInitializer() {
-// AddTestSuite(new MyConformanceTestSuite());
-// }
-// } static_test_suite_initializer;
-//
// class MyConformanceTestRunner : public ConformanceTestRunner {
// public:
+// static int Run(int argc, char *argv[],
+// ConformanceTestSuite* suite);
+//
+// private:
// virtual void RunTest(...) {
// // INSERT YOUR FRAMEWORK-SPECIFIC CODE HERE.
// }
// };
//
// int main() {
-// MyConformanceTestRunner runner;
-// const std::set<ConformanceTestSuite*>& test_suite_set =
-// ::google::protobuf::GetTestSuiteSet();
-// for (auto suite : test_suite_set) {
-// suite->RunSuite(&runner, &output);
-// }
+// MyConformanceTestSuite suite;
+// MyConformanceTestRunner::Run(argc, argv, &suite);
// }
//
class ConformanceTestSuite {
@@ -259,9 +301,6 @@
std::string type_url_;
};
-void AddTestSuite(ConformanceTestSuite* suite);
-const std::set<ConformanceTestSuite*>& GetTestSuiteSet();
-
} // namespace protobuf
} // namespace google
diff --git a/conformance/conformance_test_impl.cc b/conformance/conformance_test_impl.cc
index acd0f25..a884dea 100644
--- a/conformance/conformance_test_impl.cc
+++ b/conformance/conformance_test_impl.cc
@@ -2358,11 +2358,10 @@
"");
}
-struct StaticTestSuiteInitializer {
- StaticTestSuiteInitializer() {
- AddTestSuite(new ConformanceTestSuiteImpl());
- }
-} static_test_suite_initializer;
-
} // namespace protobuf
} // namespace google
+
+int main(int argc, char *argv[]) {
+ google::protobuf::ConformanceTestSuiteImpl suite;
+ return google::protobuf::ForkPipeRunner::Run(argc, argv, &suite);
+}
diff --git a/conformance/conformance_test_runner.cc b/conformance/conformance_test_runner.cc
index 3340f1c..9c37712 100644
--- a/conformance/conformance_test_runner.cc
+++ b/conformance/conformance_test_runner.cc
@@ -80,162 +80,31 @@
exit(1); \
}
-// Test runner that spawns the process being tested and communicates with it
-// over a pipe.
-class ForkPipeRunner : public google::protobuf::ConformanceTestRunner {
- public:
- ForkPipeRunner(const std::string &executable)
- : child_pid_(-1), executable_(executable) {}
+namespace google {
+namespace protobuf {
- virtual ~ForkPipeRunner() {}
+void ParseFailureList(const char *filename,
+ std::vector<string>* failure_list) {
+ std::ifstream infile(filename);
- void RunTest(const std::string& test_name,
- const std::string& request,
- std::string* response) {
- if (child_pid_ < 0) {
- SpawnTestProgram();
- }
-
- current_test_name_ = test_name;
-
- uint32_t len = request.size();
- CheckedWrite(write_fd_, &len, sizeof(uint32_t));
- CheckedWrite(write_fd_, request.c_str(), request.size());
-
- if (!TryRead(read_fd_, &len, sizeof(uint32_t))) {
- // We failed to read from the child, assume a crash and try to reap.
- GOOGLE_LOG(INFO) << "Trying to reap child, pid=" << child_pid_;
-
- int status;
- waitpid(child_pid_, &status, WEXITED);
-
- string error_msg;
- if (WIFEXITED(status)) {
- StringAppendF(&error_msg,
- "child exited, status=%d", WEXITSTATUS(status));
- } else if (WIFSIGNALED(status)) {
- StringAppendF(&error_msg,
- "child killed by signal %d", WTERMSIG(status));
- }
- GOOGLE_LOG(INFO) << error_msg;
- child_pid_ = -1;
-
- conformance::ConformanceResponse response_obj;
- response_obj.set_runtime_error(error_msg);
- response_obj.SerializeToString(response);
- return;
- }
-
- response->resize(len);
- CheckedRead(read_fd_, (void*)response->c_str(), len);
+ if (!infile.is_open()) {
+ fprintf(stderr, "Couldn't open failure list file: %s\n", filename);
+ exit(1);
}
- private:
- // TODO(haberman): make this work on Windows, instead of using these
- // UNIX-specific APIs.
- //
- // There is a platform-agnostic API in
- // src/google/protobuf/compiler/subprocess.h
- //
- // However that API only supports sending a single message to the subprocess.
- // We really want to be able to send messages and receive responses one at a
- // time:
- //
- // 1. Spawning a new process for each test would take way too long for thousands
- // of tests and subprocesses like java that can take 100ms or more to start
- // up.
- //
- // 2. Sending all the tests in one big message and receiving all results in one
- // big message would take away our visibility about which test(s) caused a
- // crash or other fatal error. It would also give us only a single failure
- // instead of all of them.
- void SpawnTestProgram() {
- int toproc_pipe_fd[2];
- int fromproc_pipe_fd[2];
- if (pipe(toproc_pipe_fd) < 0 || pipe(fromproc_pipe_fd) < 0) {
- perror("pipe");
- exit(1);
- }
+ for (string line; getline(infile, line);) {
+ // Remove whitespace.
+ line.erase(std::remove_if(line.begin(), line.end(), ::isspace),
+ line.end());
- pid_t pid = fork();
- if (pid < 0) {
- perror("fork");
- exit(1);
- }
+ // Remove comments.
+ line = line.substr(0, line.find("#"));
- if (pid) {
- // Parent.
- CHECK_SYSCALL(close(toproc_pipe_fd[0]));
- CHECK_SYSCALL(close(fromproc_pipe_fd[1]));
- write_fd_ = toproc_pipe_fd[1];
- read_fd_ = fromproc_pipe_fd[0];
- child_pid_ = pid;
- } else {
- // Child.
- CHECK_SYSCALL(close(STDIN_FILENO));
- CHECK_SYSCALL(close(STDOUT_FILENO));
- CHECK_SYSCALL(dup2(toproc_pipe_fd[0], STDIN_FILENO));
- CHECK_SYSCALL(dup2(fromproc_pipe_fd[1], STDOUT_FILENO));
-
- CHECK_SYSCALL(close(toproc_pipe_fd[0]));
- CHECK_SYSCALL(close(fromproc_pipe_fd[1]));
- CHECK_SYSCALL(close(toproc_pipe_fd[1]));
- CHECK_SYSCALL(close(fromproc_pipe_fd[0]));
-
- std::unique_ptr<char[]> executable(new char[executable_.size() + 1]);
- memcpy(executable.get(), executable_.c_str(), executable_.size());
- executable[executable_.size()] = '\0';
-
- char *const argv[] = {executable.get(), NULL};
- CHECK_SYSCALL(execv(executable.get(), argv)); // Never returns.
+ if (!line.empty()) {
+ failure_list->push_back(line);
}
}
-
- void CheckedWrite(int fd, const void *buf, size_t len) {
- if (write(fd, buf, len) != len) {
- GOOGLE_LOG(FATAL) << current_test_name_
- << ": error writing to test program: "
- << strerror(errno);
- }
- }
-
- bool TryRead(int fd, void *buf, size_t len) {
- size_t ofs = 0;
- while (len > 0) {
- ssize_t bytes_read = read(fd, (char*)buf + ofs, len);
-
- if (bytes_read == 0) {
- GOOGLE_LOG(ERROR) << current_test_name_
- << ": unexpected EOF from test program";
- return false;
- } else if (bytes_read < 0) {
- GOOGLE_LOG(ERROR) << current_test_name_
- << ": error reading from test program: "
- << strerror(errno);
- return false;
- }
-
- len -= bytes_read;
- ofs += bytes_read;
- }
-
- return true;
- }
-
- void CheckedRead(int fd, void *buf, size_t len) {
- if (!TryRead(fd, buf, len)) {
- GOOGLE_LOG(FATAL) << current_test_name_
- << ": error reading from test program: "
- << strerror(errno);
- }
- }
-
- int write_fd_;
- int read_fd_;
- pid_t child_pid_;
- std::string executable_;
- std::string current_test_name_;
-};
+}
void UsageError() {
fprintf(stderr,
@@ -263,33 +132,51 @@
exit(1);
}
-void ParseFailureList(const char *filename, std::vector<string>* failure_list) {
- std::ifstream infile(filename);
-
- if (!infile.is_open()) {
- fprintf(stderr, "Couldn't open failure list file: %s\n", filename);
- exit(1);
+void ForkPipeRunner::RunTest(
+ const std::string& test_name,
+ const std::string& request,
+ std::string* response) {
+ if (child_pid_ < 0) {
+ SpawnTestProgram();
}
- for (string line; getline(infile, line);) {
- // Remove whitespace.
- line.erase(std::remove_if(line.begin(), line.end(), ::isspace),
- line.end());
+ current_test_name_ = test_name;
- // Remove comments.
- line = line.substr(0, line.find("#"));
+ uint32_t len = request.size();
+ CheckedWrite(write_fd_, &len, sizeof(uint32_t));
+ CheckedWrite(write_fd_, request.c_str(), request.size());
- if (!line.empty()) {
- failure_list->push_back(line);
+ if (!TryRead(read_fd_, &len, sizeof(uint32_t))) {
+ // We failed to read from the child, assume a crash and try to reap.
+ GOOGLE_LOG(INFO) << "Trying to reap child, pid=" << child_pid_;
+
+ int status;
+ waitpid(child_pid_, &status, WEXITED);
+
+ string error_msg;
+ if (WIFEXITED(status)) {
+ StringAppendF(&error_msg,
+ "child exited, status=%d", WEXITSTATUS(status));
+ } else if (WIFSIGNALED(status)) {
+ StringAppendF(&error_msg,
+ "child killed by signal %d", WTERMSIG(status));
}
+ GOOGLE_LOG(INFO) << error_msg;
+ child_pid_ = -1;
+
+ conformance::ConformanceResponse response_obj;
+ response_obj.set_runtime_error(error_msg);
+ response_obj.SerializeToString(response);
+ return;
}
+
+ response->resize(len);
+ CheckedRead(read_fd_, (void*)response->c_str(), len);
}
-int main(int argc, char *argv[]) {
+int ForkPipeRunner::Run(
+ int argc, char *argv[], ConformanceTestSuite* suite) {
char *program;
- const std::set<ConformanceTestSuite*>& test_suite_set =
- ::google::protobuf::GetTestSuiteSet();
-
string failure_list_filename;
std::vector<string> failure_list;
@@ -299,13 +186,9 @@
failure_list_filename = argv[arg];
ParseFailureList(argv[arg], &failure_list);
} else if (strcmp(argv[arg], "--verbose") == 0) {
- for (auto *suite : test_suite_set) {
- suite->SetVerbose(true);
- }
+ suite->SetVerbose(true);
} else if (strcmp(argv[arg], "--enforce_recommended") == 0) {
- for (auto suite : test_suite_set) {
- suite->SetEnforceRecommended(true);
- }
+ suite->SetEnforceRecommended(true);
} else if (argv[arg][0] == '-') {
fprintf(stderr, "Unknown option: %s\n", argv[arg]);
UsageError();
@@ -318,18 +201,97 @@
}
}
- for (auto suite : test_suite_set) {
- suite->SetFailureList(failure_list_filename, failure_list);
- }
+ suite->SetFailureList(failure_list_filename, failure_list);
ForkPipeRunner runner(program);
std::string output;
- bool ok = true;
- for (auto suite : test_suite_set) {
- ok &= suite->RunSuite(&runner, &output);
- }
+ bool ok = suite->RunSuite(&runner, &output);
fwrite(output.c_str(), 1, output.size(), stderr);
return ok ? EXIT_SUCCESS : EXIT_FAILURE;
}
+
+void ForkPipeRunner::SpawnTestProgram() {
+ int toproc_pipe_fd[2];
+ int fromproc_pipe_fd[2];
+ if (pipe(toproc_pipe_fd) < 0 || pipe(fromproc_pipe_fd) < 0) {
+ perror("pipe");
+ exit(1);
+ }
+
+ pid_t pid = fork();
+ if (pid < 0) {
+ perror("fork");
+ exit(1);
+ }
+
+ if (pid) {
+ // Parent.
+ CHECK_SYSCALL(close(toproc_pipe_fd[0]));
+ CHECK_SYSCALL(close(fromproc_pipe_fd[1]));
+ write_fd_ = toproc_pipe_fd[1];
+ read_fd_ = fromproc_pipe_fd[0];
+ child_pid_ = pid;
+ } else {
+ // Child.
+ CHECK_SYSCALL(close(STDIN_FILENO));
+ CHECK_SYSCALL(close(STDOUT_FILENO));
+ CHECK_SYSCALL(dup2(toproc_pipe_fd[0], STDIN_FILENO));
+ CHECK_SYSCALL(dup2(fromproc_pipe_fd[1], STDOUT_FILENO));
+
+ CHECK_SYSCALL(close(toproc_pipe_fd[0]));
+ CHECK_SYSCALL(close(fromproc_pipe_fd[1]));
+ CHECK_SYSCALL(close(toproc_pipe_fd[1]));
+ CHECK_SYSCALL(close(fromproc_pipe_fd[0]));
+
+ std::unique_ptr<char[]> executable(new char[executable_.size() + 1]);
+ memcpy(executable.get(), executable_.c_str(), executable_.size());
+ executable[executable_.size()] = '\0';
+
+ char *const argv[] = {executable.get(), NULL};
+ CHECK_SYSCALL(execv(executable.get(), argv)); // Never returns.
+ }
+}
+
+void ForkPipeRunner::CheckedWrite(int fd, const void *buf, size_t len) {
+ if (write(fd, buf, len) != len) {
+ GOOGLE_LOG(FATAL) << current_test_name_
+ << ": error writing to test program: "
+ << strerror(errno);
+ }
+}
+
+bool ForkPipeRunner::TryRead(int fd, void *buf, size_t len) {
+ size_t ofs = 0;
+ while (len > 0) {
+ ssize_t bytes_read = read(fd, (char*)buf + ofs, len);
+
+ if (bytes_read == 0) {
+ GOOGLE_LOG(ERROR) << current_test_name_
+ << ": unexpected EOF from test program";
+ return false;
+ } else if (bytes_read < 0) {
+ GOOGLE_LOG(ERROR) << current_test_name_
+ << ": error reading from test program: "
+ << strerror(errno);
+ return false;
+ }
+
+ len -= bytes_read;
+ ofs += bytes_read;
+ }
+
+ return true;
+}
+
+void ForkPipeRunner::CheckedRead(int fd, void *buf, size_t len) {
+ if (!TryRead(fd, buf, len)) {
+ GOOGLE_LOG(FATAL) << current_test_name_
+ << ": error reading from test program: "
+ << strerror(errno);
+ }
+}
+
+} // namespace protobuf
+} // namespace google
diff --git a/conformance/failure_list_php_c.txt b/conformance/failure_list_php_c.txt
index bb32ace..17e0fda 100644
--- a/conformance/failure_list_php_c.txt
+++ b/conformance/failure_list_php_c.txt
@@ -93,8 +93,6 @@
Required.Proto3.JsonInput.ValueAcceptInteger.ProtobufOutput
Required.Proto3.JsonInput.ValueAcceptList.JsonOutput
Required.Proto3.JsonInput.ValueAcceptList.ProtobufOutput
-Required.Proto3.JsonInput.ValueAcceptListWithNull.JsonOutput
-Required.Proto3.JsonInput.ValueAcceptListWithNull.ProtobufOutput
Required.Proto3.JsonInput.ValueAcceptNull.JsonOutput
Required.Proto3.JsonInput.ValueAcceptNull.ProtobufOutput
Required.Proto3.JsonInput.ValueAcceptObject.JsonOutput
diff --git a/conformance/failure_list_ruby.txt b/conformance/failure_list_ruby.txt
index 1bab135..53b4970 100644
--- a/conformance/failure_list_ruby.txt
+++ b/conformance/failure_list_ruby.txt
@@ -133,3 +133,9 @@
Required.Proto3.ProtobufInput.ValidDataRepeated.FLOAT.JsonOutput
Required.TimestampProtoInputTooLarge.JsonOutput
Required.TimestampProtoInputTooSmall.JsonOutput
+Required.Proto3.JsonInput.IgnoreUnknownJsonFalse.ProtobufOutput
+Required.Proto3.JsonInput.IgnoreUnknownJsonNull.ProtobufOutput
+Required.Proto3.JsonInput.IgnoreUnknownJsonNumber.ProtobufOutput
+Required.Proto3.JsonInput.IgnoreUnknownJsonObject.ProtobufOutput
+Required.Proto3.JsonInput.IgnoreUnknownJsonString.ProtobufOutput
+Required.Proto3.JsonInput.IgnoreUnknownJsonTrue.ProtobufOutput
diff --git a/docs/options.md b/docs/options.md
index 70db4d6..700fef6 100644
--- a/docs/options.md
+++ b/docs/options.md
@@ -14,7 +14,7 @@
## Existing Registered Extensions
1. C# port of protocol buffers
- * Website: http://github.com/jskeet/dotnet-protobufs
+ * Website: https://github.com/jskeet/protobuf-csharp-port
* Extensions: 1000
1. Perl/XS port of protocol buffers
@@ -50,7 +50,7 @@
* Extensions: 1008
1. Goby Underwater Autonomy Project
- * Website: https://launchpad.net/goby
+ * Website: https://github.com/GobySoft/goby
* Extensions: 1009
1. Nanopb
@@ -62,7 +62,7 @@
* Extensions: 1011
1. Dynamic Compact Control Language
- * Website: https://launchpad.net/dccl
+ * Website: http://github.com/GobySoft/dccl
* Extensions: 1012
1. ScaleOut StateServer® Native C++ API
@@ -188,3 +188,9 @@
1. Netifi Proteus
* Website: https://github.com/netifi-proteus
* Extensions: 1057
+
+1. CGSN Mooring Project
+ * Website: https://bitbucket.org/ooicgsn/cgsn-mooring
+ * Extensions: 1058
+
+
diff --git a/java/README.md b/java/README.md
index 5e4fb8b..def4fd9 100644
--- a/java/README.md
+++ b/java/README.md
@@ -13,13 +13,17 @@
$ protoc --java_out=${OUTPUT_DIR} path/to/your/proto/file
Include the generated Java files in your project and add a dependency on the
-protobuf Java runtime. If you are using Maven, use the following:
+protobuf Java runtime.
+
+### Maven
+
+If you are using Maven, use the following:
```xml
<dependency>
<groupId>com.google.protobuf</groupId>
<artifactId>protobuf-java</artifactId>
- <version>3.5.1</version>
+ <version>3.6.1</version>
</dependency>
```
@@ -33,10 +37,18 @@
<dependency>
<groupId>com.google.protobuf</groupId>
<artifactId>protobuf-java-util</artifactId>
- <version>3.5.1</version>
+ <version>3.6.1</version>
</dependency>
```
+### Gradle
+
+If you are using Gradle, add the following to your `build.gradle` file's dependencies:
+```
+ compile 'com.google.protobuf:protobuf-java:3.6.1'
+```
+Again, be sure to check that the version number maches (or is newer than) the version number of protoc that you are using.
+
### Use Java Protocol Buffers on Android
For Android users, it's recommended to use protobuf Java Lite runtime because
diff --git a/kokoro/release/collect_all_artifacts.cfg b/kokoro/release/collect_all_artifacts.cfg
new file mode 100644
index 0000000..3da1a7c
--- /dev/null
+++ b/kokoro/release/collect_all_artifacts.cfg
@@ -0,0 +1,7 @@
+build_file: "protobuf/kokoro/release/collect_all_artifacts.sh"
+
+action {
+ define_artifacts {
+ regex: "github/protobuf/artifacts/**"
+ }
+}
diff --git a/kokoro/release/collect_all_artifacts.sh b/kokoro/release/collect_all_artifacts.sh
new file mode 100755
index 0000000..0023937
--- /dev/null
+++ b/kokoro/release/collect_all_artifacts.sh
@@ -0,0 +1,62 @@
+#!/bin/bash
+
+set -ex
+
+# Change to repo root.
+cd $(dirname $0)/../..
+
+# Initialize any submodules.
+git submodule update --init --recursive
+
+# The directory with all resulting artifacts
+mkdir -p artifacts
+
+# Artifacts from all predecessor jobs get copied to this directory by kokoro
+INPUT_ARTIFACTS_DIR="${KOKORO_GFILE_DIR}/github/protobuf"
+
+# TODO(jtattermusch): remove listing the files, but for now it make it easier
+# to iterate on the script.
+ls -R ${INPUT_ARTIFACTS_DIR}
+
+# ====================================
+# Copy to expose all the artifacts from the predecessor jobs to the output
+# TODO(jtattermusch): the directory layout of the artifact builds is pretty messy,
+# so will be the output artifacts of this job.
+cp -r ${INPUT_ARTIFACTS_DIR}/* artifacts
+
+# ====================================
+# Build Google.Protobuf.Tools C# nuget
+# The reason it's being done in this script is that we need access to protoc binaries
+# built on multiple platform (the build is performed by the "build artifact" step)
+# and adding and extra chained build just for building the Google.Protobuf.Tools
+# nuget seems like an overkill.
+cd csharp
+mkdir -p protoc/windows_x86
+mkdir -p protoc/windows_x64
+cp ${INPUT_ARTIFACTS_DIR}/build32/Release/protoc.exe protoc/windows_x86/protoc.exe
+cp ${INPUT_ARTIFACTS_DIR}/build64/Release/protoc.exe protoc/windows_x64/protoc.exe
+
+mkdir -p protoc/linux_x86
+mkdir -p protoc/linux_x64
+# Because of maven unrelated reasonse the linux protoc binaries have a dummy .exe extension.
+# For the Google.Protobuf.Tools nuget, we don't want that expection, so we just remove it.
+cp ${INPUT_ARTIFACTS_DIR}/protoc-artifacts/target/linux/x86_32/protoc.exe protoc/linux_x86/protoc
+cp ${INPUT_ARTIFACTS_DIR}/protoc-artifacts/target/linux/x86_64/protoc.exe protoc/linux_x64/protoc
+
+mkdir -p protoc/macosx_x86
+mkdir -p protoc/macosx_x64
+cp ${INPUT_ARTIFACTS_DIR}/build32/src/protoc protoc/macosx_x86/protoc
+cp ${INPUT_ARTIFACTS_DIR}/build64/src/protoc protoc/macosx_x64/protoc
+
+# Install nuget (will also install mono)
+# TODO(jtattermusch): use "mono:5.14" docker image instead so we don't have to apt-get install
+sudo apt-key adv --keyserver hkp://keyserver.ubuntu.com:80 --recv-keys 3FA7E0328081BFF6A14DA29AA6A19B38D3D831EF
+sudo apt install apt-transport-https
+echo "deb https://download.mono-project.com/repo/ubuntu stable-trusty main" | sudo tee /etc/apt/sources.list.d/mono-official-stable.list
+sudo apt update
+sudo apt-get install -y nuget
+
+nuget pack Google.Protobuf.Tools.nuspec
+
+# Copy the nupkg to the output artifacts
+cp Google.Protobuf.Tools.*.nupkg ../artifacts
diff --git a/php/ext/google/protobuf/protobuf.h b/php/ext/google/protobuf/protobuf.h
index 9cb36dc..a84feec 100644
--- a/php/ext/google/protobuf/protobuf.h
+++ b/php/ext/google/protobuf/protobuf.h
@@ -138,7 +138,8 @@
INIT_CLASS_ENTRY_EX(class_type, CLASSNAME, strlen(CLASSNAME), \
LOWWERNAME##_methods); \
LOWWERNAME##_type = zend_register_internal_class(&class_type TSRMLS_CC); \
- LOWWERNAME##_type->create_object = message_create;
+ LOWWERNAME##_type->create_object = message_create; \
+ zend_do_inheritance(LOWWERNAME##_type, message_type TSRMLS_CC);
#define PHP_PROTO_INIT_SUBMSGCLASS_END \
}
@@ -404,7 +405,7 @@
INIT_CLASS_ENTRY_EX(class_type, CLASSNAME, strlen(CLASSNAME), \
LOWWERNAME##_methods); \
LOWWERNAME##_type = zend_register_internal_class(&class_type TSRMLS_CC); \
- LOWWERNAME##_type->create_object = message_create;
+ zend_do_inheritance(LOWWERNAME##_type, message_type TSRMLS_CC);
#define PHP_PROTO_INIT_SUBMSGCLASS_END \
}
diff --git a/php/tests/well_known_test.php b/php/tests/well_known_test.php
index 9f2661f..6a788df 100644
--- a/php/tests/well_known_test.php
+++ b/php/tests/well_known_test.php
@@ -42,6 +42,7 @@
public function testEmpty()
{
$msg = new GPBEmpty();
+ $this->assertTrue($msg instanceof \Google\Protobuf\Internal\Message);
}
public function testImportDescriptorProto()
diff --git a/src/google/protobuf/util/internal/protostream_objectwriter.cc b/src/google/protobuf/util/internal/protostream_objectwriter.cc
index cf0fa45..9c52116 100644
--- a/src/google/protobuf/util/internal/protostream_objectwriter.cc
+++ b/src/google/protobuf/util/internal/protostream_objectwriter.cc
@@ -76,6 +76,18 @@
ProtoStreamObjectWriter::ProtoStreamObjectWriter(
const TypeInfo* typeinfo, const google::protobuf::Type& type,
+ strings::ByteSink* output, ErrorListener* listener,
+ const ProtoStreamObjectWriter::Options& options)
+ : ProtoWriter(typeinfo, type, output, listener),
+ master_type_(type),
+ current_(nullptr),
+ options_(options) {
+ set_ignore_unknown_fields(options_.ignore_unknown_fields);
+ set_use_lower_camel_for_enums(options.use_lower_camel_for_enums);
+}
+
+ProtoStreamObjectWriter::ProtoStreamObjectWriter(
+ const TypeInfo* typeinfo, const google::protobuf::Type& type,
strings::ByteSink* output, ErrorListener* listener)
: ProtoWriter(typeinfo, type, output, listener),
master_type_(type),
@@ -343,7 +355,7 @@
// Create our object writer and initialize it with the first StartObject
// call.
ow_.reset(new ProtoStreamObjectWriter(parent_->typeinfo(), *type, &output_,
- parent_->listener()));
+ parent_->listener(), parent_->options_));
// Don't call StartObject() for well-known types yet. Depending on the
// type of actual data, we may not need to call StartObject(). For
diff --git a/src/google/protobuf/util/internal/protostream_objectwriter.h b/src/google/protobuf/util/internal/protostream_objectwriter.h
index f865cc1..fe8170d 100644
--- a/src/google/protobuf/util/internal/protostream_objectwriter.h
+++ b/src/google/protobuf/util/internal/protostream_objectwriter.h
@@ -330,6 +330,11 @@
const google::protobuf::Type& type,
strings::ByteSink* output, ErrorListener* listener);
+ ProtoStreamObjectWriter(const TypeInfo* typeinfo,
+ const google::protobuf::Type& type,
+ strings::ByteSink* output, ErrorListener* listener,
+ const ProtoStreamObjectWriter::Options& options);
+
// Returns true if the field is a map.
inline bool IsMap(const google::protobuf::Field& field);
diff --git a/src/google/protobuf/util/json_util_test.cc b/src/google/protobuf/util/json_util_test.cc
index cbfc7ad..125d4d4 100644
--- a/src/google/protobuf/util/json_util_test.cc
+++ b/src/google/protobuf/util/json_util_test.cc
@@ -56,6 +56,7 @@
using proto3::TestMap;
using proto3::TestMessage;
using proto3::TestOneof;
+using proto3::TestAny;
static const char kTypeUrlPrefix[] = "type.googleapis.com";
@@ -357,6 +358,28 @@
EXPECT_EQ(ToJson(generated, options), ToJson(*message, options));
}
+TEST_F(JsonUtilTest, TestParsingUnknownAnyFields) {
+ string input =
+ "{\n"
+ " \"value\": {\n"
+ " \"@type\": \"type.googleapis.com/proto3.TestMessage\",\n"
+ " \"unknown_field\": \"UNKOWN_VALUE\",\n"
+ " \"string_value\": \"expected_value\"\n"
+ " }\n"
+ "}";
+
+ TestAny m;
+ JsonParseOptions options;
+ EXPECT_FALSE(FromJson(input, &m, options));
+
+ options.ignore_unknown_fields = true;
+ EXPECT_TRUE(FromJson(input, &m, options));
+
+ TestMessage t;
+ EXPECT_TRUE(m.value().UnpackTo(&t));
+ EXPECT_EQ("expected_value", t.string_value());
+}
+
TEST_F(JsonUtilTest, TestParsingUnknownEnumsProto2) {
string input =
"{\n"
