Syncing Unchecked Shared field checkbox is not performing the movement of data from SharedFields to VersionFields tables

[pnittala]

Do you want to request a feature or report a bug?
Report a bug to determine a way forward for this issue with Unicorn

What is the current behavior?
Currently, when we sync a unicorn template whose field changed from Shared field to non-Shared field, the syncing process is not triggering the sitecore job that moves the data from SharedFields table to VersionedFields table.

If the current behavior is a bug, please provide the steps to reproduce.
Step 1: Create a unicorn configuration to serialize template.
Step 2: Create a template with Title field, which is Single-line text field. This should serialize the template item to disk

---

ID: "5ee7adfb-2e0e-4d8a-a326-d279329fc734"
Parent: "711c13bb-e970-4dda-b986-91d129c0fdfe"
Template: "455a3e98-a627-4b40-8035-e683a0331ac7"
Path: /sitecore/templates/Feature/Accordions/TestSharedField/Section/Title
DB: master
SharedFields:

• ID: "ab162cc0-dc80-4abf-8871-998ee5d7ba32"
  Hint: Type
  Value: "Single-Line Text"
• ID: "ba3f86a2-4a1c-4d78-b63d-91c2779c1b5e"
  Hint: __Sortorder
  Value: 100
• ID: "be351a73-fcb0-4213-93fa-c302d8ab4f51"
  Hint: Shared
  Type: Checkbox
  Value: 1

Step3: Now add few content items based on this template and fill in Title field. This content is not in unicorn serialization. Just for info, If you uncheck the Shared checkbox in sitecore, it would run the background job to move the field data from SharedFields table to VersionedFields table.
Step4: Now to simulate syncing of this template in upper environments, we make the modification to the serialized file on disk and change Shared checkbox to 0 as shown below

---

ID: "5ee7adfb-2e0e-4d8a-a326-d279329fc734"
Parent: "711c13bb-e970-4dda-b986-91d129c0fdfe"
Template: "455a3e98-a627-4b40-8035-e683a0331ac7"
Path: /sitecore/templates/Feature/Accordions/TestSharedField/Section/Title
DB: master
SharedFields:

• ID: "ab162cc0-dc80-4abf-8871-998ee5d7ba32"
  Hint: Type
  Value: "Single-Line Text"
• ID: "ba3f86a2-4a1c-4d78-b63d-91c2779c1b5e"
  Hint: __Sortorder
  Value: 100
• ID: "be351a73-fcb0-4213-93fa-c302d8ab4f51"
  Hint: Shared
  Type: Checkbox
  Value: 0

Step5: Now synchronize this item back into Sitecore. It synchronizes but does not run the background job to move the content field data. It syncs the item from disk as shown

The fields still show up in the SharedFields table

if you rerun the sync, then it says "detected field storage corruption in Sitecore database. See sitecore log for details". When we look at the logs

33000 16:39:55 ERROR [Dilithium] Data corruption in master://{d7f1118a-7003-497a-b588-1d16220197ad}! Field 5ee7adfb-2e0e-4d8a-a326-d279329fc734 (versioned) had a value in the shared fields table. The field value will be ignored.
7548 16:39:55 INFO [Unicorn]: Feature.Accordions is being synced with Dilithium SQL + Serialized enabled.
7548 16:39:57 INFO [Unicorn]: [U] master:/sitecore/templates/Feature/Accordions/TestSharedField/__Standard Values (d7f1118a-7003-497a-b588-1d16220197ad)
7548 16:39:57 INFO [Unicorn]: > Field Title - Serialized '$name', Source ''
7548 16:39:57 INFO [Unicorn]: Feature.Accordions sync complete: 29 items evaluated, 1 item modified (0 added, 1 updated, 0 recycled) in 2413ms (~83.22ms/item).
7548 16:39:57 INFO [Unicorn]:
7548 16:39:57 INFO [Unicorn] Dilithium SQL context has been released.
7548 16:39:57 INFO [Unicorn] Dilithium Rainbow context has been released.

What is the expected behavior?
The Sharedfield data should be moved from SharedFields to VersionedFields tables and the synchronization should not show data corruption issue.

Please mention your Sitecore version and Unicorn version.
Sitecore.NET 9.2.0 (rev. 002893) and Unicorn version 4.0.8

Thanks for any help on this issue.

[markgibbons25]

I have noticed the same in TDS too, so I have a suspicion that the UI triggers a separate pipeline for this. </shower thoughts>

[cassidydotdk]

Yes.. This data migration job is indeed started as a background process when you make the template change in the UI. The problem is as old as time itself, more or less, it also happens if you install a package with the same change.

I am unaware of anything we can do from an API level to address this. Is anyone aware if SCS has addressed and fixed this issue? I'll try and ask around.

[pnittala]

How do others handle this kind of change promotion to upper environments when using unicorn or packages considering there is content already in upper environments? Any suggestions?

[cassidydotdk]

I just checked; this problem also exists in SCS.

For "developer content" under source control, you generally do a full reserialize when changing field sharing status (shared, versioned, etc). This is because of the yaml format (which SCS and TDS also shares) where field values exist under "shared", "versioned", "unversioned" sections as well.

For templates that your authors use, and items not under direct serialization control, I would either

1. Introduce an entirely new field and, if necessary, use SPE to script the content across to this new field
2. Make the field sharing change manually on the target CM (thus triggering the background data migration job)

Or a combination of the above.

[pnittala]

Thanks a lot for the quick replies on this issue. As this is an issue that happens natively also, I think we can close this issue.

[cassidydotdk]

Well I don't mind keeping it here for the record. I think we may have another issue logged somewhere like it however.

That said, I'm in communication with someone inside Sitecore to see if a solution can be found. I will say this however, even if we DO find a working solution, I'm not convinced we can get the Dilithium engine to play along with this. But I will set aside some time as soon as I can to re-visit this problem and see if I can come up with some sort of solution.

[pnittala]

Thanks Mark!! Please let me know here if you do find any solution to this problem.

[markgibbons25]

Another nail in the coffin for dilithium 🗡️

[cassidydotdk]

Another nail in the coffin for dilithium 🗡️

Well it's not Dilithium's fault that Sitecore data has decided to move out of SQL 😂

[cassidydotdk]

It does beg the question however. What happens if you deploy your templates as protobuf? As in; what happens if you deploy first, and then deploy an updated protobuf file where the field sharing status has changed? I'll bet you right now, the exact same thing...