Serialize/Deserialize required fields as default (null or false)

[kachaps4u]

Hello!
We have an API that need to merge the request with the backend data. So potentially the request can have null fields that are required.

eg:
Person(name: String, dob: LocalDate, isApplicant:Boolean)

Backend data:

{ "name": "Name", "dob": "2017-06-30", "isApplicant": true }

New request:

{ "name": "NewName" }

By new request, we want to only update "name" field in the backend but keep the rest of the fields as backend value. This works with Json4s as null fields can be configured to null and we have a merge function to merge correctly. Using jsoniter-scala, per the documentation:

• Serialization of null values is prohibited by throwing of NullPointerException errors
• Parsing of null values allowed only for optional or collection types (that means the None value or an empty collection accordingly) and for fields which have defined non-null default values

Is there anyway we can get around this in jsoniter-scala other than changing all data models to make all optional fields?

[plokhotnyuk]

You can write own custom codecs if your project have a small number of such models that do not evolve a lot.

But for generic case a better approach is to have classes for JSON messages that match requests as much as possible and keep them separated from your model classes:

case class Person(name: String, dob: LocalDate, isApplicant: Boolean) // data model

case class PersonUpdate(name: Option[String], dob: Option[LocalDate], isApplicant: Option[Boolean]) // request footprint

Then using such kind of libraries like chimney you can easily apply optional updates.

Keeping those request classes and data model classes separately will allow to evolve them independently that will simplify development a lot. As an example, you can add the id field to the data model without risk of updating it, etc.

[kachaps4u]

Our initial plan was to keep a separate class for request as you give example as. The issue was the model was so huge that we were wasting time on mapping and error prone. To reduce time, we ended up writing a JsonMerger that handle "null", "nil" etc. Nobody researched on chimney that time, may be because of the time crunch, looks excellent, may be I will do a POC on it and propose some ideas to the team. Thanks a lot for the reply and the library is amazing to work with!

I am working around a temporary solution, we will keep the merger, probably use Json4s for parsing the request but use jsoniter-scala for database model.

[kachaps4u]

You can write own custom codecs if your project have a small number of such models that do not evolve a lot.

But for generic case a better approach is to have classes for JSON messages that match requests as much as possible and keep them separated from your model classes:

case class Person(name: String, dob: LocalDate, isApplicant: Boolean) // data model

case class PersonUpdate(name: Option[String], dob: Option[LocalDate], isApplicant: Option[Boolean]) // request footprint

Then using such kind of libraries like chimney you can easily apply optional updates.

Keeping those request classes and data model classes separately will allow to evolve them independently that will simplify development a lot. As an example, you can add the id field to the data model without risk of updating it, etc.

We needed to differentiate between the request nulling a field (an optional field) vs not providing. When not provided, we want to keep the backend value which is doable. But I think nulling also would keep the backend value when we must clear the backend field. Anyway we can override the null parsing behavior for optional fields? Using Json4s parsing, we get JNull and we use that for the merge to work correctly.

[plokhotnyuk]

It could be modeled clearly by the following sum type (instead of Option):

sealed trait Action[T+]
case object Hold extends Action[Nothing]
case object Clear extends Action[Nothing]
case class Update(v: T) extends Action[T]

With a custom codec that will parse Clear (for JSON null values) and Update (for non-null values) only, while Hold should be used as default value of class fields that model update requests:

case class PersonUpdate(name: Action[String] = Hold, dob: Action[LocalDate] = Hold, isApplicant: Action[Boolean] = Hold)

I can add a runnable example to tests of jsoniter-scala-macros module if you interested to see that approach in details, but for that modification there are no patching macros for transformation yet.

[kachaps4u]

This is great and yes I would like to see the custom codec on handling the null. So if you can give more details, would help!

[plokhotnyuk]

@kachaps4u Before preparing a detailed example for Action, I want to mention a couple of other possible implementations:

1. Using Option[Option[_]] for modeling of null value and missing key-value pair together with withSkipNestedOptionValues(true) compile-time configuration for codecs, like here
2. Using a custom codec that wraps existing instances and apply update immediately during parsing, to minimize CPU and memory usage.

[kachaps4u]

@kachaps4u Before preparing a detailed example for Action, I want to mention a couple of other possible implementations:

1. Using Option[Option[_]] for modeling of null value and missing key-value pair together with withSkipNestedOptionValues(true) compile-time configuration for codecs, like here
2. Using a custom codec that wraps existing instances and apply update immediately during parsing, to minimize CPU and memory usage.

• 1 sounds good and I think the best plan is to use a different model for the API vs backend as Option[Option[ sounds to me like we are trying too much so may be for the API model. I think I understood the Action idea, its similar to Nullable in the same spec I believe?
• Are you saying to write custom code for all case classes with custom parsing of each fields? is that correct?

[plokhotnyuk]

@kachaps4u Before preparing a detailed example for Action, I want to mention a couple of other possible implementations:

1. Using Option[Option[_]] for modeling of null value and missing key-value pair together with withSkipNestedOptionValues(true) compile-time configuration for codecs, like here
2. Using a custom codec that wraps existing instances and apply update immediately during parsing, to minimize CPU and memory usage.

* 1 sounds good and I think the best plan is to use a different model for the API vs backend as Option[Option[ sounds to me like we are trying too much so may be for the API model. I think I understood the Action idea, its similar to Nullable in the same spec I believe?

The difference between Option[Option[_]] and Action[_] that 1st option is already supported by macros for auto-derivation of codecs of nested data structures, no need to derive codecs for all types that could be used in your data structures on place of _.

* Are you saying to write custom code for all case classes with custom parsing of each fields? is that correct?

Yes, but no need to write them manually we can generate them using simple macros that works with mirrors for sum/ product types and required types of collections. BTW, how do you merge collections with updates, could you please give some examples?

[kachaps4u]

got it, I like the Option[Option for sure, less code.

About the collection, that's something I wanted to bring up actually.

For Seq/List => if field not provided, keep backend else if field provided and null, then clear else update
For Map => if field not provided, keep backend else if field provided with key and value as null value, then clear that entry in backend model else replace the value for the key provided

case class Address( streetName: Option[String] = None, city: Option[String] = None, state: Option[String] = None )

case class Income(amt: Option[BigDecimal] = None)

case class Person( name: Option[String] = None, address: Seq[Address] = Seq.empty, incomes: Map[String, Income] = Map.empty )

So below is possible:
Given backend:
{"name":"name","address":[{"streetName":"street","city":"city","state":"state"}],"incomes":{"1":{"amt":20}, "2":{"amt":21}}}

With a request:
{"incomes":{"1":null,"2":{"amt":19}}}

[plokhotnyuk]

got it, I like the Option[Option for sure, less code.

About the collection, that's something I wanted to bring up actually.

For Seq/List => if field not provided, keep backend else if field provided and null, then clear else update For Map => if field not provided, keep backend else if field provided with key and value as null value, then clear that entry in backend model else replace the value for the key provided

case class Address( streetName: Option[String] = None, city: Option[String] = None, state: Option[String] = None )

case class Income(amt: Option[BigDecimal] = None)

case class Person( name: Option[String] = None, address: Seq[Address] = Seq.empty, incomes: Map[String, Income] = Map.empty )

So below is possible: Given backend: {"name":"name","address":[{"streetName":"street","city":"city","state":"state"}],"incomes":{"1":{"amt":20}, "2":{"amt":21}}}

With a request: {"incomes":{"1":null,"2":{"amt":19}}}

So Seq/List instances can be reset or updates as whole collections only, but Map instances can be updated as whole collections or per key-value pair (entry) basis?

[kachaps4u]

That's correct