Add a few design sketches for requested features.
diff --git a/doc/design_docs/default_values.md b/doc/design_docs/default_values.md new file mode 100644 index 0000000..66a41fc --- /dev/null +++ b/doc/design_docs/default_values.md
@@ -0,0 +1,138 @@ +# Design Sketch: Initializer Values for Fields + +## Motivation + +It is often useful to initialize a structure to some "un-set" value before +starting to modify it. For many structures, a simple `memset(buffer, sizeof +*buffer, 0)` suffices, but other structures require specific values at specific +locations, which are tedious to write out in user code. + +This design proposes two main elements: + +1. A way to specify initializer values for specific fields. +2. A new feature in the generated C++ code to set the memory underlying a view + to those initializer values, or to 0 if no initializer was specified. + + +## Initializer Value Syntax + +### Attribute + +The most straightforward option is to use the existing attribute syntax: + + struct Foo: + 0 [+2] UInt bar + [initialize_to: 7] + +The exact name TBD, but it should be somewhat visually distinct from the +existing `$default` keyword for attributes, which specifies that an attribute +should be used for all descendants of the current node, unless overridden: + + [$default byte_order = "LittleEndian"] + + +### New Syntax + +Other options might add new syntax. + + +#### Suffix `= value` + +Suffix `= value` looks somewhat similar to the "initialize on construction" +syntax in languages like C++: + + struct Foo: + 0 [+2] UInt bar = 7 + + class Foo { + int bar = 7; + }; + +However, it also looks somewhat confusingly similar to the field number +specifiers in Proto: + + message Foo { + optional uint32 bar = 7; + } + + +#### Something Else? + +It is difficult to come up with a syntax that is clear and concise, especially +to a reader who is not particularly familiar with Emboss: + + struct Foo: + 0 [+2] UInt bar := 7 + + struct Foo: + 0 [+2] UInt bar [7] + + struct Foo: + 0 [+2] UInt [initialize to 7] bar + +Feel free to propose other options. + + +## `Initialize()` Method + +Emboss *views* do not own their backing storage: creating a view does not +allocate memory, it just provides a structured, well, view of existing bytes. +This means that there is not a natural place to automatically initialize a +struct, the way that there is for an object in a typical programming language. + +Instead, I propose adding an `Initialize()` method (name TBD) to each view, +which can be called to explicitly initialize the underlying memory. + +For `external` (`UInt`, `Bcd`, etc.) and `enum` views, `Initialize()` should +just set the initializer value specified in the `.emb` file, or `0` if none was +specified. + +For structure views (`struct` and `bits`), `Initialize()` should set the +initializer values of each of their fields, recursively. + +TBD whether `Initialize()` should also zero out any bytes that are not part +of any concrete field. + + +## Implementation Notes + +This is a moderately complex change, touching both the front end and C++ back +end of the compiler, as well as the C++ runtime. + + +### Front End Changes + +On the front end, adding a new attribute is definitely the most straightforward +change: mostly just adding the new attribute to `attributes.py`, and updating a +few things in `dependency_checker.py`, `expression_bounds.py`, and possibly +`constraints.py` to inspect the new attribute. + + +### C++ Back End Changes + +On the back end, there are a couple of implementation strategies. + +The easiest strategy is to generate an `Initialize()` method on each structure +type that recursively calls `Initialize()` on each field within the structure +(in dependency-safe order, similar to `WriteToTextStream()`). + +An alternate strategy would be to generate an "empty image" for each structure, +and have `Initialize()` `memcpy()` the image into its backing storage. This +seems like it would be faster at runtime, but may bloat binary size quite a bit +-- a 4kb `struct` would need 4kb of const data in your binary to support +`Initialize()`, whereas iteratively calling `Initialize()` on each element of +an array does not require any extra code space for each element of the array. +For this reason, there would likely still need to be a fallback to recursively +calling `Initialize()`. + +Either way, fields with an explicit initializer value need to be wrapped in a +view adapter when accessed, similar to how `[requires]` is handled now. + +Enum views can be generated with a simple `Initialize()` method that just sets +their backing storage to 0. + + +### C++ Runtime Changes + +Each of the views for Prelude types (`UInt`, `Int`, `Flag`, etc.) in +`runtime/cpp/emboss_prelude.h` will need to have the new `Initialize()` method.
diff --git a/doc/design_docs/explicitly_size_enums.md b/doc/design_docs/explicitly_size_enums.md new file mode 100644 index 0000000..31b5106 --- /dev/null +++ b/doc/design_docs/explicitly_size_enums.md
@@ -0,0 +1,106 @@ +# Design Sketch: Explicit Enum Sizes + +## Overview + +Currently in Emboss, when rendering an Emboss `enum`, the corresponding C++ +`enum` always has either `int64_t` or `uint64_t` as its storage class, e.g.: + + enum class Foo : int64_t { + // ... values ... + }; + +This is because Emboss `enum`s are open (can hold any value, even if it is not +a known enum value) and can be placed in fields of any size up to 64 bits. +E.g.: + + enum Foo: + AA = 1 + BB = 2 + CC = 3 + + struct Bar: + 0 [+1] Foo short_foo + 1 [+8] Foo long_foo + +Since `Foo` doesn't know how big of a field it will eventually be used for -- +and it can vary! -- Emboss takes the safe approach of using a 64-bit type in +C++-land. + +(Aside: yes, there are cases in real message formats where the same enum is used +in fields of different sizes. In message formats from multiple manufacturers. +Generally in some overly-clever way.) + +However, this means that anyone who directly uses that `enum` type in their own +C++ `class` has to pay for the entire 8 bytes, even if the `enum` would fit in +a much smaller type, and larger values are never needed. + + +## `[max_value_size_in_bits]` Attribute + +This design proposes adding a `[max_value_size_in_bits]` (name TBD) attribute +on `enum`s, which specifies that an `enum` may not be used in a field larger +than a certain size. This gives the C++ back end (and others) the freedom to +use a smaller underlying type. For example: + + enum Foo: + [max_value_size_in_bits: 16] + AA = 1 + BB = 2 + CC = 3 + + struct Bar: + 0 [+1] Foo short_foo + 1 [+8] Foo long_foo # Now an error + +In the generated C++, this would now produce: + + enum class Foo : uint16_t { + // ... values ... + }; + +Now any C++ classes storing `Foo` would only need to allocate 2 bytes. + + +## `[signedness]` Attribute (Optional) + +Currently, Emboss uses the absence or presence of any negative values to +determine whether an `enum` type uses `uint64_t` or `int64_t` storage: + + enum Unsigned: + AA = 1 + + enum Signed: + AA = -1 + +Produces: + + enum class Unsigned : uint64_t { + AA = 1, + }; + + enum class Signed : int64_t { + AA = -1, + }; + +It may make sense to add an explicit attribute to control signedness, instead +of relying on the "negative value" heuristic. + + +## Implementation Notes + +This is a relatively simple change, even though it touches places throughout +the Emboss codebase. + + +### Compiler Front End + +`attributes.py` needs to be updated with the new attribute. +`_check_that_enum_values_are_representable()` in `constraints.py` needs to be +updated to check the constrained sizes. + + +### C++ Back End + +`_generate_enum_definition()` in `header_generator.py` needs to be updated to +check the new attribute and use it to pick a type, instead of always picking +`::std::int64_t` or `::std::uint64_t`.
diff --git a/doc/design_docs/field_packing_notation.md b/doc/design_docs/field_packing_notation.md new file mode 100644 index 0000000..77ae16e --- /dev/null +++ b/doc/design_docs/field_packing_notation.md
@@ -0,0 +1,94 @@ +# Design Sketch: Packed Field Notation + +## Motivation + +Many structures have many or most fields laid out consecutively, possibly with +padding for alignment. For example: + + struct Simple: + 0 [+2] UInt field_1 + 2 [+4] UInt field_2 + 6 [+2] UInt field_3 + +For simple structures of fixed-size fields, the main issue is unchecked +redundancy: it is relatively easy to enter the wrong value for the field +offset, and no compiler checks will help. + +For more complex structures with multiple variable-sized fields, this can lead +to unwieldy offsets: + + struct Complex: + 0 [+2] UInt header_length (h) + 2 [+h] Header header + 2+h [+2] UInt body_length (b) + 4+h [+b] Body body + 4+h+b [+4] UInt crc + +In both cases, there is some benefit to a shorthand notation that says 'this +field should be placed immediately after the end of the lexically-previous +field. + + +## Example + +(Exact syntax TBD.) + + struct Complex: + 0 [+2] UInt header_length (h) + $next [+h] Header header + $next [+2] UInt body_length (b) + $next [+b] Body body + $next [+4] UInt crc + +It is tempting to use some more specialized, terser syntax, like: + + struct Complex: + 0 [+2] UInt header_length (h) + ^^ [+h] Header header + ^^ [+2] UInt body_length (b) + ^^ [+b] Body body + ^^ [+4] UInt crc + +However, an explicit symbol has the advantage that you can use it in +expressions, if needed: + + struct ComplexWithGap: + 0 [+2] UInt header_length (h) + $next [+h] Header header + $next [+2] UInt body_length (b) + # 2-byte reserved gap. + $next+2 [+b] Body body + $next [+4] UInt crc + +Or, with a (not-yet-implemented) `$align()` function: + + struct ComplexWithAlignment: + 0 [+2] UInt header_length (h) + $align($next, 4) [+h] Header header + $align($next, 4) [+2] UInt body_length (b) + $align($next, 4) [+b] Body body + $align($next, 4) [+4] UInt crc + + +## Implementation + +Assuming the "new symbol" approach: + +1. Pick a new symbol name -- preferably, come up with a few alternatives and + do a quick survey. By convention, Emboss built-in symbols start with `$`. +2. Add the new name to `LITERAL_TOKEN_PATTERNS` in + `compiler/front_end/tokenizer.py`. +3. Add a new production for `builtin-word -> "$new_symbol"` to the `_word()` + function in `module_ir.py`. +4. Add a new compiler pass before `synthetics.synthesize_fields`, to replace + the new symbol with the expanded representation. This should be relatively + straightforward -- something that uses `fast_traverse_ir_top_down()` to + find all `ir_pb2.Structure` elements in the IR, then iterates over the + field offsets within each structure, and recursively replaces any + `ir_pb2.Expression`s with a + `builtin_reference.canonical_name.object_path[0]` equal to + `"$new_symbol"`. It would probably be useful to make + `traverse_ir._fast_traverse_proto_top_down()` into a public function, so + that you do not have to re-write `Expression` traversal. + +For this change, the back end should not need any modifications.
diff --git a/doc/design_docs/value_of_enum_function.md b/doc/design_docs/value_of_enum_function.md new file mode 100644 index 0000000..4b73016 --- /dev/null +++ b/doc/design_docs/value_of_enum_function.md
@@ -0,0 +1,79 @@ +# Design Sketch: Integer-Value-of-Enum Function + +## Overview + +It is sometimes useful to use the integer value of an enumerated name: + + enum Foo: + ZZ = 17 + + struct Bar: + [requires: id == Foo.ZZ] # Type error + 0 [+4] UInt id + +In the current Emboss expression language, there is no way to perform this +comparison. + + +## `$to_int()` Function + +A `$to_int()` function (name TBD), taking an `enum` value and returning the +same numeric value with type integer, would fix this problem: + + enum Foo: + ZZ = 17 + + struct Bar: + [requires: id == $to_int(Foo.ZZ)] + 0 [+4] UInt id + + +## `$from_int()` Function (Optional) + +The opposite function would also be useful in some circumstances, but would +take a lot more effort: new syntax would be needed for a type-parameterized +function. + +A couple of possible syntaxes: + + $int_to<EnumType>(7) # 1 + EnumType.$from_int(7) # 2 + EnumType(7) # 3 + +The first option resembles type-parameterized functions in many languages, but +may require some tricky modifications to Emboss's strict LR(1) grammar. + +The second option looks like a class method (a la Python) or a static method (a +la C++/Java/C#), and *may* require a less-difficult change to the Emboss +grammar... but there are some messy bits in the grammar around how `.` is +handled, and the notation does not scale to multiple types. + +The third option is more C-like, *still* requires grammar updates, and does not +provide any obvious solution for any other, future type-parameterized functions. + + +## Implementation Notes + +`$to_int()` would require changes in a lot of places, though each change should +be small. + +`$from_int()` would require changes in pretty much the same places, but a few +of them would be significantly more complex. + +Basically anywhere that walks or evaluates an `ir_pb2.Expression` would need to +be updated to know about the new function. A probably-incomplete list: + + compiler/back_end/header_generator.py + compiler/front_end/constraints.py + compiler/front_end/expression_bounds.py + compiler/front_end/type_check.py + compiler/util/ir_util.py + +Additionally, for `$to_int()`, minor tweaks would need to be made to +`tokenizer.py` (add the new function name) and `module_ir.py` (register the new +name as a function in the syntax). + +For `$from_int()`, the list is essentially the same, except that `module_ir.py` +would need much larger updates to allow whichever new syntax, and some of the +other changes would be more complex in order to verify that the type parameter +was actually an `enum`.