Add IANA-to-BCP47 and reverse mappings

[sffc]

Fixes #2909

Depends on #4021
Depends on #4022
Depends on #4023

Replaces #3499

[sffc]

Design Question:

The data store I'm using for IANA-to-BCP47 is

pub struct IanaToBcp47MapV1<'data> {
    /// A map from IANA time zone identifiers to indexes of BCP-47 time zone identifiers.
    #[cfg_attr(feature = "serde", serde(borrow))]
    pub map: ZeroTrie<ZeroVec<'data, u8>>,
    /// A sorted list of BCP-47 time zone identifiers.
    #[cfg_attr(feature = "serde", serde(borrow))]
    // Note: this is 9739B as ZeroVec<TinyStr8> and 9335B as VarZeroVec<str>
    pub bcp47_ids: ZeroVec<'data, TimeZoneBcp47Id>,
}

For BCP47-to-IANA, I could simply do ZeroMap<TimeZoneBcp47Id, String>. However, I could save a bit of space (probably on the order of 1 kB) by instead storing a VarZeroVec<String> where the indices in that map correspond to the indices of bcp47_ids in the other data key.

We have examples of multi-key dependencies, but I don't think they are the type that have invariants between them. In this case, the invariant would be that the lists correspond to each other (essentially the keys of the map are in one DataKey and the values of the map are in a different DataKey).

Thoughts? @robertbastian @Manishearth

[robertbastian]

Seems brittle.

[Manishearth]

Not a huge fan of interkey dependencies where a broken dependency is not detectable (in this case 99% of the time a mismatch in versions would lead to buggy data in an undetectable way)

Now, there is a way to solve that problem: we can store a hash of the index value with the map, and if they mismatch we throw an error. If we want to be really fancy we can even have the map store an Option<Indices> and you can configure datagen to emit a full index map when you know there is going to be a discrepancy in the two data keys.

This is in line with previous decisions that datagen configurability should not be used to add or omit data in a user-visible way but can be used for optimizations.

[sffc]

we can store a hash of the index value with the map

Yeah, or how about this: we take the hash of the whole indices zerovec and store just that in both keys, like this:

struct Foo<'data> {
    bcp47_ids: ZeroVec<'data, TinyStr8>,
    bcp47_ids_hash: u32,
    // ... other data ...
}

struct Bar<'data> {
    bcp47_ids: Option<ZeroVec<'data, TinyStr8>>,
    bcp47_ids_hash: u32,
    // ... other data ...
}

The behavior would be:

1. If the bcp47_ids_hash are the same in both data structs, use Foo::bcp47_ids
2. If the bcp47_ids_hash are not the same, use Bar::bcp47_ids if it is Some
3. If the hashes are not the same and Bar::bcp47_ids is None, return an error from the constructor

Is this a great idea to save a kilobyte or two, or is it overengineering?

[dpulls]

🎉 All dependencies have been resolved !

[sffc]

Okay, here is what I ended up implementing.

I made a checksum for the ZeroVec that was duplicated between the two keys, but I did not wire it up to be able to be duplicated in the second key. It is just always absent, and if the checksum is inconsistent between the two keys, the constructor fails. I didn't want to fiddle with datagen options for a situation that is unlikely to happen in the real world. If we ever do encounter this, we can add a new key or a V2 of the current key.

The ZeroVec in question is about 3.5 KB which seems about the size where this type of thing could be justifiable.

The data in postcard is 9749B (primary direction) and 7569B (reverse direction) which I'm quite happy with. For comparison, the non-ZeroTrie version was 14475B (primary direction) and 11249B (reverse with the deduplicated ZeroVec which ZeroTrie enabled).

[sffc]

Are all the following statements true?

1. The only blocking issue for this PR is the hash choice
2. The only hash crates still in contention are twox_hash and crc32fast
3. We consider both twox_hash and crc32fast to be robust, widely used crates that contain algorithms suitable for a checksum hash

[robertbastian]

1. The only blocking issue for this PR is the hash choice

✅

2. The only hash crates still in contention are twox_hash and crc32fast

I also still think SipHasher is a possible way forward, as we do not need any cryptographic properties. The worst that can happen is GIGO.

3. We consider both twox_hash and crc32fast to be robust, widely used crates that contain algorithms suitable for a checksum hash

✅

[robertbastian]

uhm this basically tests that the implementation of as_bytes is the same as all_bytes for each entry. While this is useful to test, it should be a test in zeroslice, not here.

This should test things like:

• If the order changes the checksums are different
• The checksum for "abc", "def" is different from the one of "abcd", "ef"
• The checksum for "abc", "def", "" is different from the one for "abc", "def", ""
• The checksum for the hardcoded list is equal to a hardcoded checksum (stability)

[sffc]

done

[robertbastian]

Discussion:

• @zbraniecki and @robertbastian advocate for SipHasher
• @sffc agrees if SipHasher is platform independent, will investigate

[sffc]

CC @Manishearth to weigh in on the hash choice (current decision is to use std::hash::SipHasher with an #[allow(deprecated)])

[justingrant]

Note that the next CLDR release will include a new iana attribute that, if present, overrides the alias order. See unicode-org/cldr#3105.

Does this PR handle that attribute?

[sffc]

No, CLDR 44 Alpha is not in scope for the 1.3 release, but this will be a priority for the 1.4 release. Filed #4044

[sffc]

@Manishearth said in #4030 (comment):

(we should not use SipHash, it's deprecated and nobody on the libs team wants people to use it for stuff like this. Since it is deprecated it's really not something that I would consider "maintained", even if it is in the stdlib.)

[sffc]

Based on @Manishearth's comment, I pre-emptively pushed another commit reverting SipHash back to XxHash, pending additional feedback from @robertbastian or @zbraniecki

[sffc]

I think we should rule out crc32fast because it only generates a 32-bit hash, and it mainly exists because of compiler intrinsics. I think we should choose between XxHash and SipHash.

[sffc]

I'm merging this and leaving the normalization follow-up for #4031.