Add string decoding benchmarks #352
Conversation
|
|
||
| // LLM-generated poems in east asian languages. | ||
| // Hopefully a lot of surrogate pairs in here. | ||
| var str = ` |
There was a problem hiding this comment.
As someone who doesn't speak these languages, I don't know what the content is beyond this comment; and I'm hoping the LLM didn't mistranslate.
Is there anything we can reference that maybe is public domain (and a known quantity) instead?
There was a problem hiding this comment.
I was originally looking at the Thousand Character Classic -- an ancient chinese text for teaching written chinese, but I decided it wasn't long enough (but we could just .repeat it I suppose).
I did plug all of this into google translate. It's some pretty generic and generally positive/optimistic poetry. Nothing weird that I could see.
There was a problem hiding this comment.
If you're really worried, I have a friend literate in Japanese and several friends literate in Chinese. I could have them double check it. The Korean part will be harder. I don't know anyone literate in Korean who can read beyond a 2nd grade level.
There was a problem hiding this comment.
I think I'll combine a few things. I'm finding some stuff on wikisource. The Art of War has a ton of text we can use. I'll find something for other languages too.
No description provided.