docs/design/editions/editions-feature-visibility.md - third_party/github/protocolbuffers/protobuf - Git at Google

 # Editions Feature Visibility

 **Authors:** [@mkruskal-google](https://github.com/mkruskal-google)

 **Approved:** 2023-09-08

 ## Background

 While [Editions: Life of a FeatureSet](editions-life-of-a-featureset.md) handles
 how we propagate features *to* runtimes, what's left under-specified is how the
 runtimes should expose features to their users. *Exposing Editions Feature Sets*
 (not available externally) was an initial attempt to cover both these topics
 (specifically the C++ API section), but much of it has been redesigned since.
 This is a much more targeted document laying out how features should be treated
 by runtimes.

 ## Problem Description

 There are two main concerns from a runtime's perspective:

 1.  **Direct access to resolved features protos** - While runtime decisions
     *should* be made based on the data in these protos, their struct-like nature
     makes them very rigid. Once users start to depend on the proto API, it makes
     it very difficult for us to do internal refactoring. These protos are also
     naturally structured based on how feature *specification* is done in proto
     files, rather than the actual behaviors they represent. This makes it
     difficult to guarantee that complex relationships between features and other
     conditions are being uniformly handled.

 2.  **Accidental use of unresolved features** - Unresolved features represent a
     clear foot-gun for users, that could also cause issues for us. Since they
     share the same type as resolved features, it's not always easy to tell the
     two apart. If runtime decisions are made using unresolved features, it's
     very plausible that everything will work as expected in a given edition by
     coincidence. However, when the proto's edition is bumped, it will very
     likely break this code unexpectedly.

 Some concrete examples to help illustrate these concerns:

 *   **Remodeling features** - We've bounced back and forth on how UTF8
     validation should be modeled as a feature. None of the proposals resulted in
     any functional changes, since edition zero preserves all proto2/proto3
     behavior, the question was just about what features should be used to
     control them. While the `.proto` file large-scale change to bump them to the
     next edition containing these changes is unavoidable, we'd like to avoid
     having to update any code simultaneously. If everyone is directly inspecting
     the `utf8_validation` feature, we would need to do both.

 *   **Incomplete features** - Looking at a feature like `packed`, it's really
     more of a contextual *suggestion* than a strict rule. If it's set at the
     file level, **all** fields will have the feature even though only packable
     ones will actually respect it. Giving users direct access to this feature
     would be problematic, because they would *also* need to check if it's
     packable before making decisions based on it. Field presence is an even more
     complicated example, where the logic we want people making runtime decisions
     based on is distinct from what's specified in the proto file.

 *   **Optimizations** - One of the major considerations in *Exposing Editions
     Feature Sets* (not available externally) was whether or not it would be
     possible to reduce the cost of editions later. Every descriptor is going to
     contain two separate features protos, and it's likely this will end up
     getting expensive as we roll out edition zero. We could decide to optimize
     this by storing them as a custom class with a much more compact memory
     layout. This is similar to other optimizations we've done to descriptor
     classes, where we have the freedom to *because* we don't generally expose
     them as protos.

 *   **Bumpy Edition Large-scale Change** - The proto team is going to be
     responsible for rolling out the next edition to internal Google repositories
     every year (at least 80% of it per our churn policy). We *expect* that
     people are only making decisions based on resolved features, and therefore
     that Prototiller transformations are behavior-preserving (despite changing
     the unresolved features). If people have easy access to unresolved features
     though, we can expect a lot of Hyrum's law issues slowing down these
     large-scale changes.

 ## Recommended Solution

 We recommend a conservative approach of hiding all `FeatureSet` protos from
 public APIs whenever possible. This means that there should be no public
 `features()` getter, and that features should be stripped from any descriptor
 options. All `options()` getters should have an unset features field. Instead,
 helper methods should be provided on the relevant descriptors to encapsulate the
 behaviors users care about. This has already been done for edition zero features
 (e.g. `has_presence`, `requires_utf8_validation`, etc), and we should continue
 this model.

 The one notable place where we *can't* completely hide features is in
 reflection. Most of our runtimes provide APIs for converting descriptors back to
 their original state at runtime (e.g. `CopyTo` and `DebugString` in C++). In
 order to give a faithful representation of the original proto file in these
 cases, we should include the **unresolved** features here. Given how inefficient
 these methods are and how hard the resulting protos are to work with, we expect
 misuse of these unresolved features to be rare.

 **Note:** While we may need to adjust this approach in the future, this is the
 one that gives us the most flexibility to do so. Adding a new API when we have
 solid use-cases for it is easy to do. Removing an existing one when we decide we
 don't want it has proven to be very difficult.

 ### Enforcement

 While we make the recommendation above, ultimately this decision should be up to
 the runtime owners. Outside of Google we can't enforce it, and the cost would be
 a worse experience for *their* users (not the entire protobuf ecosystem). Inside
 of Google, we should be more diligent about this, since the cost mostly falls on
 us.

 ### μpb

 One notable standout here is μpb, which is a runtime *implementation*, but not a
 full runtime. Since μpb only provides APIs to the wrapping runtime in a target
 language, it's free to expose features anywhere it wants. The wrapping language
 should be responsible for stripping them out where appropriate.

 #### Pros

 *   Prevents any direct access to resolved feature protos

     *   Gives us freedom to do internal refactoring
     *   Allows us to encapsulate more complex relationships
     *   Users don't have to distinguish between resolved/unresolved features

 *   Limits access to unresolved features

     *   Accidental usage of these is less likely (especially considering the
         above)

 *   This should be easy to loosen in the future if we find a real use-case for
     `features()` getters.

 *   More inline with our descriptor APIs, which wrap descriptor protos but
     aren't strictly 1:1 with them. Options are more an exception here, mostly
     due to the need to expose extensions.

 #### Cons

 *   There's no precedent for modifying `options()` like this. Up until now it
     represented a faithful clone of what was specified in the proto file.

 *   Deciding to loosen this in the future would be a bit awkward for
     `options()`. If we stop stripping it, people will suddenly start seeing a
     new field and Hyrum's law might result in breakages.

 *   Requires duplicating high-level feature behaviors across every language. For
     example, `has_presence` will need to be implemented identically in every
     language. We will likely need some kind of conformance test to make sure
     these all agree.

 ## Considered Alternatives

 ### Expose Features

 This is the simplest implementation, and was the initial approach taken in
 prototypes. We would just have public `features()` getters in our descriptor
 APIs, and keep the unresolved features in `options()`.

 #### Pros

 *   Very easy to implement

 #### Cons

 *   Doesn't solve any of the problems laid out above

 *   Difficult to reverse later

 ### Hide Features in Generated Options

 This is a tweak of the recommended solution where we add a hack to the generated
 options messages. Instead of just stripping the features out and leaving an
 empty field, we could give the `features` fields "package-scoped" visibility
 (e.g. access tokens in C++). We would still strip them, but nobody outside of
 our runtimes could even access them to see that they're empty. This eliminates
 the Hyrum's law concern above.

 #### Pros

 *   Resolves one of the cons in the recommended approach.

 #### Cons

 *   We'd have to do this separately for each runtime, meaning specific hacks in
     *every* code generator

 *   No clear benefit. This only helps **if** we decide to expose features and
     **if** a bunch of people start depending on the fact that `features` are
     always empty.

 ### ClangTidy warning Options Features

 Similar to the above alternative, but leverages ClangTidy to warn users against
 checking `options().features()`.

 #### Pros

 *   Resolves one of the cons in the recommended approach.

 #### Cons

 *   Doesn't work in every language

 *   Doesn't work in OSS
	# Editions Feature Visibility

	Authors: [@mkruskal-google](https://github.com/mkruskal-google)

	Approved: 2023-09-08

	## Background

	While [Editions: Life of a FeatureSet](editions-life-of-a-featureset.md) handles
	how we propagate features to runtimes, what's left under-specified is how the
	runtimes should expose features to their users. Exposing Editions Feature Sets
	(not available externally) was an initial attempt to cover both these topics
	(specifically the C++ API section), but much of it has been redesigned since.
	This is a much more targeted document laying out how features should be treated
	by runtimes.

	## Problem Description

	There are two main concerns from a runtime's perspective:

	1. Direct access to resolved features protos - While runtime decisions
	should be made based on the data in these protos, their struct-like nature
	makes them very rigid. Once users start to depend on the proto API, it makes
	it very difficult for us to do internal refactoring. These protos are also
	naturally structured based on how feature specification is done in proto
	files, rather than the actual behaviors they represent. This makes it
	difficult to guarantee that complex relationships between features and other
	conditions are being uniformly handled.

	2. Accidental use of unresolved features - Unresolved features represent a
	clear foot-gun for users, that could also cause issues for us. Since they
	share the same type as resolved features, it's not always easy to tell the
	two apart. If runtime decisions are made using unresolved features, it's
	very plausible that everything will work as expected in a given edition by
	coincidence. However, when the proto's edition is bumped, it will very
	likely break this code unexpectedly.

	Some concrete examples to help illustrate these concerns:

	* Remodeling features - We've bounced back and forth on how UTF8
	validation should be modeled as a feature. None of the proposals resulted in
	any functional changes, since edition zero preserves all proto2/proto3
	behavior, the question was just about what features should be used to
	control them. While the `.proto` file large-scale change to bump them to the
	next edition containing these changes is unavoidable, we'd like to avoid
	having to update any code simultaneously. If everyone is directly inspecting
	the `utf8_validation` feature, we would need to do both.

	* Incomplete features - Looking at a feature like `packed`, it's really
	more of a contextual suggestion than a strict rule. If it's set at the
	file level, all fields will have the feature even though only packable
	ones will actually respect it. Giving users direct access to this feature
	would be problematic, because they would also need to check if it's
	packable before making decisions based on it. Field presence is an even more
	complicated example, where the logic we want people making runtime decisions
	based on is distinct from what's specified in the proto file.

	* Optimizations - One of the major considerations in *Exposing Editions
	Feature Sets* (not available externally) was whether or not it would be
	possible to reduce the cost of editions later. Every descriptor is going to
	contain two separate features protos, and it's likely this will end up
	getting expensive as we roll out edition zero. We could decide to optimize
	this by storing them as a custom class with a much more compact memory
	layout. This is similar to other optimizations we've done to descriptor
	classes, where we have the freedom to because we don't generally expose
	them as protos.

	* Bumpy Edition Large-scale Change - The proto team is going to be
	responsible for rolling out the next edition to internal Google repositories
	every year (at least 80% of it per our churn policy). We expect that
	people are only making decisions based on resolved features, and therefore
	that Prototiller transformations are behavior-preserving (despite changing
	the unresolved features). If people have easy access to unresolved features
	though, we can expect a lot of Hyrum's law issues slowing down these
	large-scale changes.

	## Recommended Solution

	We recommend a conservative approach of hiding all `FeatureSet` protos from
	public APIs whenever possible. This means that there should be no public
	`features()` getter, and that features should be stripped from any descriptor
	options. All `options()` getters should have an unset features field. Instead,
	helper methods should be provided on the relevant descriptors to encapsulate the
	behaviors users care about. This has already been done for edition zero features
	(e.g. `has_presence`, `requires_utf8_validation`, etc), and we should continue
	this model.

	The one notable place where we can't completely hide features is in
	reflection. Most of our runtimes provide APIs for converting descriptors back to
	their original state at runtime (e.g. `CopyTo` and `DebugString` in C++). In
	order to give a faithful representation of the original proto file in these
	cases, we should include the unresolved features here. Given how inefficient
	these methods are and how hard the resulting protos are to work with, we expect
	misuse of these unresolved features to be rare.

	Note: While we may need to adjust this approach in the future, this is the
	one that gives us the most flexibility to do so. Adding a new API when we have
	solid use-cases for it is easy to do. Removing an existing one when we decide we
	don't want it has proven to be very difficult.

	### Enforcement

	While we make the recommendation above, ultimately this decision should be up to
	the runtime owners. Outside of Google we can't enforce it, and the cost would be
	a worse experience for their users (not the entire protobuf ecosystem). Inside
	of Google, we should be more diligent about this, since the cost mostly falls on
	us.

	### μpb

	One notable standout here is μpb, which is a runtime implementation, but not a
	full runtime. Since μpb only provides APIs to the wrapping runtime in a target
	language, it's free to expose features anywhere it wants. The wrapping language
	should be responsible for stripping them out where appropriate.

	#### Pros

	* Prevents any direct access to resolved feature protos

	* Gives us freedom to do internal refactoring
	* Allows us to encapsulate more complex relationships
	* Users don't have to distinguish between resolved/unresolved features

	* Limits access to unresolved features

	* Accidental usage of these is less likely (especially considering the
	above)

	* This should be easy to loosen in the future if we find a real use-case for
	`features()` getters.

	* More inline with our descriptor APIs, which wrap descriptor protos but
	aren't strictly 1:1 with them. Options are more an exception here, mostly
	due to the need to expose extensions.

	#### Cons

	* There's no precedent for modifying `options()` like this. Up until now it
	represented a faithful clone of what was specified in the proto file.

	* Deciding to loosen this in the future would be a bit awkward for
	`options()`. If we stop stripping it, people will suddenly start seeing a
	new field and Hyrum's law might result in breakages.

	* Requires duplicating high-level feature behaviors across every language. For
	example, `has_presence` will need to be implemented identically in every
	language. We will likely need some kind of conformance test to make sure
	these all agree.

	## Considered Alternatives

	### Expose Features

	This is the simplest implementation, and was the initial approach taken in
	prototypes. We would just have public `features()` getters in our descriptor
	APIs, and keep the unresolved features in `options()`.

	#### Pros

	* Very easy to implement

	#### Cons

	* Doesn't solve any of the problems laid out above

	* Difficult to reverse later

	### Hide Features in Generated Options

	This is a tweak of the recommended solution where we add a hack to the generated
	options messages. Instead of just stripping the features out and leaving an
	empty field, we could give the `features` fields "package-scoped" visibility
	(e.g. access tokens in C++). We would still strip them, but nobody outside of
	our runtimes could even access them to see that they're empty. This eliminates
	the Hyrum's law concern above.

	#### Pros

	* Resolves one of the cons in the recommended approach.

	#### Cons

	* We'd have to do this separately for each runtime, meaning specific hacks in
	every code generator

	* No clear benefit. This only helps if we decide to expose features and
	if a bunch of people start depending on the fact that `features` are
	always empty.

	### ClangTidy warning Options Features

	Similar to the above alternative, but leverages ClangTidy to warn users against
	checking `options().features()`.

	#### Pros

	* Resolves one of the cons in the recommended approach.

	#### Cons

	* Doesn't work in every language

	* Doesn't work in OSS