events.NewMessage event is not triggered for certain channels

[M4jx]

Code that causes the issue

@client.on(events.NewMessage(chats = list(channels_hashmap.keys())))
async def main(event):
     print(now.strftime("%d/%m/%Y %H:%M:%S") + f" Received from channel: {event.chat_id}")

Expected behavior

When a new message is sent in a channel passed to chats, the events.NewMessage should be triggered.

Actual behavior

When a new message is sent, the events.NewMessage is not being triggered for certain channels.

I have tested to create a channel in an account A and monitor it in account B (where telethon is running) and the events.NewMessage was triggered instantly. However, for certain public channels that I do not own, the event is not being triggered.

Tested on a newly created channel with 2 subscribers and it worked.
Tested on a public channel with 500K subscribers and it did not work.

Traceback

No crashes are happening.

Telethon version

1.34.0

Python version

3.10.12

Operating system (including distribution name and version)

Ubuntu 22.04.4 LTS

Other details

No response

Checklist

• The error is in the library's code, and not in my own.
• I have searched for this issue before posting it and there isn't an open duplicate.
• I ran pip install -U https://github.com/LonamiWebs/Telethon/archive/v1.zip and triggered the bug in the latest version.

[Lonami]

Please react with the thumbs-up emoji instead of posting "me too" comments.

[M4jx]

Update:
While Telethon is running, if a message is sent in the monitored channel and the events.NewMessage event is not triggered. It can be triggered manually if you open the channel and read the message in the Telegram app (from the same user account telethon is running on).
Once the channel is opened, and the message is read, telethon triggers the event instantly.

[khoben]

Update:
While Telethon is running, if a message is sent in the monitored channel and the events.NewMessage event is not triggered. It can be triggered manually if you open the channel and read the message in the Telegram app (from the same user account telethon is running on).
Once the channel is opened, and the message is read, telethon triggers the event instantly.

Telegram doesn't always send all updates to the client. Some channels can only be updated after manually calling for updates. You can try restarting your app, creating a new session, changing the IP or the account itself.

[iSte94]

Same problem for me, and it's not the library. Same problem with Pyrogram

[bosslaiv]

I noticed the same ,,, few months back i was runing multiple clients via mobile IP , and received 100% all updates ,,, few months later , after a pause , i started a single client over VPN , and not all updates where being registered .

[Lonami]

Some people have reported that some channels must be polled manually to receive updates as of recently.

Telethon does not have any built-in way to do this yet (but of course can be done via raw API; I just haven't checked so I don't know exactly how).

I don't know if this will ever be "fixed" to work automatically, because if the need for manual polling is true, Telethon would have no way to know what it needs to poll manually.

[Tidalikk]

I noticed the same ,,, few months back i was runing multiple clients via mobile IP , and received 100% all updates ,,, few months later , after a pause , i started a single client over VPN , and not all updates where being registered .

Since when did you start noticing this issue?

[francestu96]

Update: While Telethon is running, if a message is sent in the monitored channel and the events.NewMessage event is not triggered. It can be triggered manually if you open the channel and read the message in the Telegram app (from the same user account telethon is running on). Once the channel is opened, and the message is read, telethon triggers the event instantly.

Actually, in my case, even the read of the message from the Telegram app does not trigger it. Is it only a problem of mine?

[sushilronghe99]

#4361

In my case the whole app just blackouts for randomly 1-2 minutes and then it receives the new messages updates properly. I have 10 channels subscribed and for 2 minutes app did not receive any update from any channels between 9:27-9:29 AM and 10:02-10:04 AM. Afterward, for the rest of the day, all updates were received properly.

So as suggested here NewMessage event is not reliable and we should poll those channels manually. I will try that and let you know how it goes.

[arvindavoudi]

I've encountered a peculiar situation with a message delivery order, and I'd like to explain it in hopes of shedding light on the issue.
Assume we have three messages (A, B, and C) sent in the following order with some minutes delay in between:

• A
• B
• C

In my case, the following occurred:

• Message (A) was received instantly.
• Message (C) was received 4 minutes after it was sent.
• Message (B) was not received until I manually opened the chat after receiving the update for message (C). This behavior aligns with the previous observation made by @M4jx.

To better understand and debug this issue, I have the following questions:

• Why do we sometimes encounter situations where new messages are received instantly, even in problematic channels (although rare)?
• How can a message be ignored and queued for later delivery, while the next message is received (albeit with a delay)?
• Is it possible that the Telegram server attempts to transmit the message (B) first, but upon failing for several minutes, it proceeds with the next available message (C), causing the delay in receiving (C)?
• What could be blocking the transmission of message (B)?
• Could this issue be related to an update to one of Telegram's layers? Since Pyrogram is no longer maintained and exhibits the same issue, I can't help but suspect a problem with an unfixed layer.
• Many users are encountering this issue, so it is a common problem corrupting people's bots, as well as mine. So how fast can we find a workaround to give Telethon back its correct functionality? Due to its critical consequences, is contacting Telegram Support considerable if we have no way to make it work as intended?

Checking the behavior of other packages written in other languages can also help us to confirm where the issue is from and why.

[Lonami]

Out-of-order messages shouldn't occur because the library checks that the incoming pts come in order:

Telethon/telethon/_updates/messagebox.py

Line 561 in 4767368

elif local_pts + pts.pts_count < pts.pts:

Make sure the message isn't being deleted and getting resent.

I have observed official clients displaying messages that had been deleted multiple times, and not too long ago.

If you really think Telethon is processing updates out of order, I either need someone with that problem to debug it themselves, or a way in which I can reliably reproduce it myself. Although I've already spent a significant amount of time going over the update handling code myself, so I doubt I would uncover anything.

As far as I know https://core.telegram.org/api/updates doesn't say anything Telethon isn't doing already (at the time of writing).

---

As for missing messages, or the library fetching them after a while. Yes, there is a default timeout after which the library will attempt to fetch updates:

Telethon/telethon/_updates/messagebox.py

Line 46 in 4767368

NO_UPDATES_TIMEOUT = 15 * 60

This is in line with Telegram's documentation above:

Manually obtaining updates is also required in the following situations:

• Long period without updates: no updates for 15 minutes or longer.

[arvindavoudi]

Dear @Lonami, I'm sure that I received message (C) before (B) but I'm not sure if it was deleted and sent again or not. If it was deleted, I expected to receive that exact same message again after a while, right? but as you know, sometimes messages do not get delayed, they simply get ignored and never get transmitted. Even after opening that channel on my phone, message (B) was not received "twice", just only once and I also didn't receive message (C) again (meaning that message (C) was not deleted and got resent). As far as I consider these messy updates complicated and not quite trackable, I cannot really guarantee that this unordered updates issue is valid but It was my observation that I decided to share with you because my whole project is now unusable after a few months of not being on the server. It's not working and I'm very concerned. :)
With the all experience you have, can you consider submitting a bug to https://bugs.telegram.org/ as Telethon owner? I think it would help if you're sure there is no issue with Telethon itself since Pyrogram also does not work as expected and a working solution is needed widely in the industry.

P.S.: If you haven't any channel suffering from this updating issue (not the ordering one), I know one that I can send only privately to you.

[Lonami]

I do not have interest in submitting bug reports to Telegram. You can certainly message me privately (or include the channel in a GitHub comment between <!-- HTML comments --> so that only maintainers can read it), and I may give it a go… eventually.

Perhaps trying this out would be a good excuse to make sure update handling works correctly in the (not ready for use) v2 version.

[Alex11381]

Has anyone found a solution to this problem?

[alan-insam]

Is there any workaround yet?

[sushilronghe99]

Hi There,

There is no solution to make the update function work the way we want. I also tried another library but no luck. It is already established its an issue with Telegram API and not with libraries.

Here is the workaround that might help your case. (It requires heavy usage of Asyncio)

Run a task every 30 sec (or less) to pull the messages from Telegram using iter_messages. Use offset_id instead of offset_date to fetch messages. The function should effectively use the message ID to track the current state of every channel you listen to.

Few important points

1. The logic will get complicated if you have multiple clients running on the same server use dictionaries or persistent storage to keep track of each channel state.
2. You can also use the New Message Trigger - In my experience this has a success rate of 20%. If we are keeping both then we need to make sure the message ID is used effectively to identify duplicate messages.
3. My Bot is running straight for 10hrs daily with an interval of 30 sec - so far there are no complaints from telegram or telethon. If you are going to run this for a lesser interval and longer time please run proper tests before going to production.

This works for me like a charm - not even a single message missed in the last 30 days for almost 5 clients running on one server. Not sure if this is the right solution but at least for my use case it's working.

Hope this helps.

Thanks

[galaSYT1]

I noticed this issue over 6 months ago. If Telegram hasn't fixed anything until now I don't see them fixing it at all.
Afaik there is no easy fix for it.

As already mentioned, if the App is open NewMessage works correctly.

[fabianski7]

for me this problem only happens on channels and only on those with a lot of subscribers. everything else works normally

this channel with over 100,000 subscribers is working https://t.me/apostasepalpites
this one with over 100,000 too doesn't work https://t.me/promocoesecuponsglobais

[sabitm]

I wonder if anybody has done reading the source code of official Android/iOS telegram app and find out what (hack) they use for new message event instead? It seems that the official mobile app didn't suffer from this problem at all

[khoben]

I wonder if anybody has done reading the source code of official Android/iOS telegram app and find out what (hack) they use for new message event instead? It seems that the official mobile app didn't suffer from this problem at all

Actually, with official clients exactly the same problem, some channels are updated only when you open them.

[sabitm]

Actually, with official clients exactly the same problem, some channels are updated only when you open them.

Oh yeah, totally missed that. I opened my Telegram when watching them. Now, what do they use instead? Manual fetch?

[khoben]

Actually, with official clients exactly the same problem, some channels are updated only when you open them.

Oh yeah, totally missed that. I opened my Telegram when watching them. Now, what do they use instead? Manual fetch?

I assume this is done for resource optimization purposes. Some channels get real time updates, some only manually.
Sometimes with new sessions or a different client IP, the channel starts to get updates.

Official clients also listen for updates just like telethon, but it's not as important for the user to get necessarily all the updates from 500+ channels. Further, when the channel is opened, the updates will be pulled up.

[delobanov]

As for missing messages, or the library fetching them after a while. Yes, there is a default timeout after which the library will attempt to fetch updates

Hello! Is there any way to fetch updates for client manually from my script?

[delobanov]

Hi, @Lonami! Can you help with my previous message please?

[Lonami]

client.catch_up is the only method the library offers to force-fetch updates. If that doesn't work, you'll need to getDifference yourself.

[FlutterGenerator]

Plz update

Traceback (most recent call last):
File "", line 198, in _run_module_as_main
File "", line 88, in _run_code File "/data/data/com.termux/files/home/friendly-telegram/friendly-telegram/main.py", line 33, in
from . import main File "/data/data/com.termux/files/home/friendly-telegram/friendly-telegram/main.py", line 32, in
from telethon import TelegramClient, events File "/data/data/com.termux/files/home/.local/lib/python3.12/site-packages/telethon/init.py", line 1, in
from .client.telegramclient import TelegramClient
File "/data/data/com.termux/files/home/.local/lib/python3.12/site-packages/telethon/client/init.py", line 21, in
from .downloads import DownloadMethods
File "/data/data/com.termux/files/home/.local/lib/python3.12/site-packages/telethon/client/downloads.py", line 16, in
import aiohttp File "/data/data/com.termux/files/usr/lib/python3.12/site-packages/aiohttp/init.py", line 6, in from .client import (
File "/data/data/com.termux/files/usr/lib/python3.12/site-packages/aiohttp/client.py", line 39, in
from . import hdrs, http, payload
File "/data/data/com.termux/files/usr/lib/python3.12/site-packages/aiohttp/http.py", line 7, in
from .http_parser import (
File "/data/data/com.termux/files/usr/lib/python3.12/site-packages/aiohttp/http_parser.py", line 49, in
from .http_writer import HttpVersion, HttpVersion10
File "/data/data/com.termux/files/usr/lib/python3.12/site-packages/aiohttp/http_writer.py", line 230, in
_c_serialize_headers = _http_writer._serialize_headers
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AttributeError: module 'aiohttp._http_writer' has no attribute '_serialize_headers'

[syntaxsurge]

client.catch_up is the only method the library offers to force-fetch updates. If that doesn't work, you'll need to getDifference yourself.

Thanks, I tried this solution and it works now:
await tg_client.catch_up()