Option to always encode proto3 fields, even if they have their default value

[arbrauns]

Would it be possible to add a field/message/file option that forces fields to always be encoded, even if they have their default value and would thus normally be skipped in proto3?

My use case is the following: I have a device configuration on an embedded device that is serialized to EEPROM using nanopb. When upgrading to newer firmware with new config options, these new options should be set to specific values (as specified in the firmware, not in protobuf), while all old options should be used as-is. This mostly works fine by first filling the config struct with the "default" values, then deserializing into it using pb_decode_ex(..., PB_DECODE_NOINIT);.

However, since fields with a zero-ish value are not serialized at all, they always look like "new fields" on the next deserialization and are set to their "default" value.

From a spec perspective, is it legal to always encode fields, even if they have their default value? If so, would a PR adding such an option to nanopb be accepted?

[PetteriAimonen]

I think the best solution would be to use optional fields which were added to proto3 a few years ago. This generates a bool has_field that indicates whether each field is present and should be encoded.

It is also possible to set nanopb-specific (nanopb).proto3 = false option on a single field, which results in the same. But the official optional keyword should be preferred nowadays.

Encoding zero values should be acceptable for the format. But if there is no separate has_field generated, I wonder how you would distinguish between present and not-present fields? If you always encode every field, it would overwrite your old options also.

[arbrauns]

I always want every field to be encoded, exactly because during decoding, I assume default values for any field that is not present (through PB_DECODE_NOINIT). A field not being encoded should mean "the previous firmware version didn't have this field, fall back to a sane default value".

I do not want to have to check a has_* value for every field (there are ~100 of them) if nanopb can already almost do this all for me.

[PetteriAimonen]

Hmm, ok. Seems like a bit niche situation, but that behavior would make sense for your use case.
I'm not sure if it is worth including in the main library, unless there is some particularly straightforward way to implement this (I couldn't come up with any yet).

Have you considered using required fields with proto2 syntax? .. Edit: err, that would result in decode error .. have to think about this.

[PetteriAimonen]

https://github.com/nanopb/nanopb/blob/master/pb_encode.c#L270-L278

This piece of code may be relevant, the situation in #558 was somewhat similar except using proto2 syntax and specifying proto3 = true to remove the has_fields.

[arbrauns]

Maybe it could just be added as a flag to pb_encode_ex() instead of a protobuf option?

[PetteriAimonen]

Seems quite difficult to pass that flag down to where the decision is made.

[arbrauns]

Unfortunate, would a PR be accepted? Otherwise we'll just have to keep the hack in our private fork, not a big deal either.

[PetteriAimonen]

I'm not sure that there is a way to implement this that I would be comfortable merging. It seems it would need quite large changes to the rest of the code for a fairly niche use-case.

But if you already have a working hack, it could be interesting to see it. You can e.g. attach a patch file here.

[arbrauns]

Well right now I'm just commenting out the call to pb_check_proto3_default_value() in encode_field(), but that's obviously a global compile-time setting rather than a per-encode setting.

[PetteriAimonen]

Hmm yeah. I guess that would have minimum impact even though it is not as useful as a per-field option.

This is probably worth reconsidering.