Problem with Senders' names null in some chats only

[marcello-pietrobon]

Hello, thank you for your lifesaver software :)

Using Android on Google Pixel 2 XL

I've been able to extract all my wa databases with your extractor software. Also the exporter works without any problem.

Nonetheless some exported html chats have all their senders's name with the name of the chat, and by debugging I could see that msg.sender is null.

I can't figure out why this happens with only some chats

Those chats, and only those, have a filename number-name where number is starting by 120363

The groups that are working all have their filename without the number part in their exported html files.

In the 'messages' function in ..../Whatsapp_Chat/WhatsApp-Chat-Exporter/venv/lib/python3.11/site-packages/Whatsapp_Chat_Exporter/extract.py the c.execute() call happens in the sqlite3.OperationalError block.

I wander if there's a mistake in the code being that wa reports all the senders correctly

I would love to solve this (it's urgent because wa is clogging my cell completely)
I'm able to understand the 'jid jid_group' part in the line 263 of extract.py.

'LEFT JOIN jid jid_group
                            ON jid_group._id = message.sender_jid_row_id'

only intuitively. I guess is accepting the first of the two matching field that is not null. I couldn't find documentation for that syntax with two field names.

Also the senders' avatars are not present, I would love to have them, but that's of course much less important, and probably for a separate question.

Thanks for any help

M

[KnugiHK]

Hi.

In the 'messages' function in ..../Whatsapp_Chat/WhatsApp-Chat-Exporter/venv/lib/python3.11/site-packages/Whatsapp_Chat_Exporter/extract.py the c.execute() call happens in the sqlite3.OperationalError block.

Can you pinpoint the line where c.execute() was called?

Those chats, and only those, have a filename number-name where number is starting by 120363

Are "Those chats" PMs or group chats? sender will only be populated in group chats.

it's urgent because wa is clogging my cell completely

No worries. You won't lose any data as long as you have a backup of the database, the encryption key, and the media files. You could remove WhatsApp from your phone and re-run the exporter with the backup in the future.

I'm able to understand the 'jid jid_group' part in the line 263 of extract.py only intuitively. I guess is accepting the first of the two matching field that is not null. I couldn't find documentation for that syntax with two field names.

Do you mean joining the same table jid twice?

Also the senders' avatars are not present, I would love to have them, but that's of course much less important, and probably for a separate question.

It is available for iOS. Android support will need to be added.

[marcello-pietrobon]

Can you pinpoint the line where c.execute() was called?

With the very latest dev version of today, the line is main.__main__ line 312

    if not args.exported and not args.import_json:
        if os.path.isfile(msg_db):
            with sqlite3.connect(msg_db) as db:
                db.row_factory = sqlite3.Row
                messages(db, data, args.media)                        # line 312

Are "Those chats" PMs or group chats? sender will only be populated in group chats.

The ones having this problem are all group chats, but not all group chats show this problem. In the first post I specify the characteristics of their filenames.

Do you mean joining the same table jid twice?

No.
extract.py line 263 says

                        LEFT JOIN jid jid_group

which is an accepted sentence though I've not found documentation of a LEFT JOIN with two tables specified (jid and jid_group in this case). It was making me wander what it means, but I guess is correct. I rarely work with databases so I may be reading incorrectly and I incorrectly wrote fields rather than tables.

[KnugiHK]

In the 'messages' function in ..../Whatsapp_Chat/WhatsApp-Chat-Exporter/venv/lib/python3.11/site-packages/Whatsapp_Chat_Exporter/extract.py the c.execute() call happens in the sqlite3.OperationalError block.

This is because your WhatsApp database is using the new schema. c.execute() happens inside the except block of sqlite3.OperationalError to account for that.

The ones having this problem are all group chats, but not all group chats show this problem. In the first post I specify the characteristics of their filenames.

I suspected that the jid of the group does not contain a dash, in which group chats are assumed to have one, and therefore sender is not populated.

which is an accepted sentence though I've not found documentation of a LEFT JOIN with two tables specified (jid and jid_group in this case). It was making me wander what it means, but I guess is correct. I rarely work with databases so I may be reading incorrectly and I incorrectly wrote fields rather than tables.

It is not joining two tables. jid_group is an alias for jid.

[marcello-pietrobon]

I suspected that the jid of the group does not contain a dash, in which group chats are assumed to have one, and therefore sender is not populated

That's great. I looked at the code and I see this pattern recurring in many places, so I guess I'll wait for your fix.
The only thing maybe is to know if you'll be able to fix it this month or not, otherwise I may consider to fix it myself and open a PR for that.

It is available for iOS. Android support will need to be added.

I looked into the folder Media/WhatsApp Profile Photos/ and it appears empty... maybe the files are hidden or placed somewhere else? I wander if a simple hint could lead me to a quick fix. But again, this is way less important for me. (or I can open a separate thread for it, svp let me know).

Thank you so much for your prompt replies, very very helpful for me.

Best,
Marcello

[KnugiHK]

That's great. I looked at the code and I see this pattern recurring in many places, so I guess I'll wait for your fix.
The only thing maybe is to know if you'll be able to fix it this month or not, otherwise I may consider to fix it myself and open a PR for that.

It is possible to fix it within this month, as long as we discover another concrete way to determine if the chat is a group chat or a PM. Any ideas are welcome.

I looked into the folder Media/WhatsApp Profile Photos/ and it appears empty... maybe the files are hidden or placed somewhere else? I wander if a simple hint could lead me to a quick fix. But again, this is way less important for me. (or I can open a separate thread for it, svp let me know).

Android's WhatsApp avatars are stored in app's internal storage, which you can extract it with the WhatsApp-Key-DB-Extractor. It's been a long time since I implemented the support for retrieving avatars. Therefore, it may not work now. If it is the case, please open an issue at that repo.

Also, opening another issue for requesting the support of avatars in Android is preferred. It provides better tracking on the matter.

[marcello-pietrobon]

Thank you.

It is possible to fix it within this month, as long as we discover another concrete way to determine if the chat is a group chat or a PM. Any ideas are welcome.

What is a PM chat? Maybe I'm able to answer understanding better your reply because all one-to-one work well, only some group chats don't work, the others do but I cannot figure out with clarity where's their difference. I'm definitely available for help.

Also, opening another issue for requesting the support of avatars in Android is preferred. It provides better tracking on the matter.

I agree. I'll look into it and open a separate issue.

[KnugiHK]

What is a PM chat?

Private Message. Same meaning of one-to-one chat.

[marcello-pietrobon]

Definitely all PM chats are working right. :)

[Lehnemann]

The same is happening with me... on Android 13.

[marcello-pietrobon]

The same is happening with me... on Android 13.

I hope this coming week to be able to find the time to fix it, or at least to test KnugiHK's explanation.

[KnugiHK]

Does anyone know whether or not a group name is a must for WhatsApp groups? I am looking for the possibility of distinguishing groups/PMs via the subject column in the chat table.

EDIT: Group name is optional.

[KnugiHK]

@txanetxarra Could you try the dev branch and see if the problem is fixed?

[KnugiHK]

Hi everyone. Commit 7da71e8 should fixed the problem. Feel free to try it in dev branch.

[txanetxarra]

Hello @KnugiHK,

I've run again the scripts and the messages are now correctly exported.

Thank you so much for the dev branch!

One last question, In case the export is made including the wa.db file, Is it possible to have a contacts export including the name an phone number of all contacts in a single html file?

Best regards.

[KnugiHK]

Is it possible to have a contacts export including the name an phone number of all contacts in a single html file?

@txanetxarra Is feature no.2 mentioned in #69 similar to what you are asking for?

[txanetxarra]

Is it possible to have a contacts export including the name an phone number of all contacts in a single html file?

@txanetxarra Is feature no.3 mentioned in #69 similar to what you are asking for?

@KnugiHK,

Ok, glad to see that my suggestions are already being considered.

I'll be willing to try with the release.

Best regards 😉👍.

[Lehnemann]

@KnugiHK Thanks again for your effort... Your software is great! 😀🎆

[marcello-pietrobon]

Thank you so much. I'll try the fix as soon as I can, next week and let you know.

[txanetxarra]

Hello @marcello-pietrobon,

The main problem of " Problem with Senders' names null in some chats only #67" has been already adressed by @KnugiHK in commit 7da71e8 and available in the Dev branch.

I've tested commit 7da71e8 and it's working fine.

Just letting you know in case you were planning to check this issue.

Best regards.

[KnugiHK]

Released in 0.9.7.