Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

关于中文编码 #75

Open
Leputa opened this issue Aug 20, 2020 · 0 comments
Open

关于中文编码 #75

Leputa opened this issue Aug 20, 2020 · 0 comments

Comments

@Leputa
Copy link

Leputa commented Aug 20, 2020

关于中文编码问题
bert应该都是能正常显示中文的,这个应该怎么解决

INFO:tensorflow:Writing example 0 of 3272863

2 |   | INFO:tensorflow:*** Example ***
3 |   | INFO:tensorflow:guid: train-0
4 |   | INFO:tensorflow:tokens: [CLS] \u6211 \u7684 \u4e16 \u754c \u963f \u9633 \u548c \u5c0f \u6708 \u5320 \u9b42 \u8054 \u673a \u751f [SEP] \u963f \u9633 \u5c0f \u6708 \u6211 \u7684 \u4e16 \u754c \u8054 \u673a \u751f \u5b58 84 [SEP]
5 |   | INFO:tensorflow:input_ids: 101 2769 4638 686 4518 7350 7345 1469 2207 3299 1269 7789 5468 3322 4495 102 7350 7345 2207 3299 2769 4638 686 4518 5468 3322 4495 2100 8479 102
6 |   | INFO:tensorflow:input_mask: 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1
7 |   | INFO:tensorflow:segment_ids: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 1 1 1 1 1 1 1
8 |   | INFO:tensorflow:label: 1 (id = 1)
9 |   | INFO:tensorflow:*** Example ***
10 |   | INFO:tensorflow:guid: train-1
11 |   | INFO:tensorflow:tokens: [CLS] \u548c \u5e73 \u7cbe \u82f1 oppo ##a5 \u624b \u673a \u7075 \u654f \u5ea6 [SEP] \u548c \u5e73 \u7cbe \u82f1 \u534e \u4e3a \u624b \u673a \u4e09 \u6307 \u538b \u67aa \u6700 \u7a33 \u7684 \u7075 [SEP]
12 |   | INFO:tensorflow:input_ids: 101 1469 2398 5125 5739 8806 12540 2797 3322 4130 3130 2428 102 1469 2398 5125 5739 1290 711 2797 3322 676 2900 1327 3366 3297 4937 4638 4130 102
13 |   | INFO:tensorflow:input_mask: 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1
14 |   | INFO:tensorflow:segment_ids: 0 0 0 0 0 0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant