Builtin TURN Service

[sando38]

Hi jitsi team,

here is a proposal/draft for a builtin TURN service. It can be build for linux/amd64 and linux/arm64.

The TURN service is based on eturnal.

I did not test all possible jitsi configurations, but I was able to establish a call while using the TURN service. More tests should be performed to ensure it works properly with different configurations.

Open items:

• TURNS (TURN via TLS).

Currently eturnal does not have any (valid) TLS certificates included. As a workaround, they could be mounted of course into the container. It would be nicer, however, to use the LE acme.sh cronjob to also update the certs for eturnal.

EDIT (2022-09-07): Startup script now copys the certs generated by jitsi web, when mounted into the container, to use TLS. A renewal cronjob or similar is not yet implemented. So the setup works also with TURNS via port 443 now.

EDIT (2022-09-11): Renewal cronjob is implemented together with a s6-overlay init.

2022-09-07 13:06:31.053469+00:00 [info] Listening on [::]:3478 (auto) (STUN/TURN)
2022-09-07 13:06:45.715878+00:00 [info] Accepting long-term STUN/TURN authentication [TLS, session 6aasdenljjsx, user 1662625689, client 192.0.2.44:63674]
2022-09-07 13:06:45.716419+00:00 [notice] Creating TURN allocation (lifetime: 600 seconds) [TLS, session 6aasdenljjsx, user 1662625689, client 192.0.2.44:63674, relay 203.0.113.4:50172]
2022-09-07 13:06:45.716588+00:00 [info] Accepting long-term STUN/TURN authentication [TLS, session oavtk6dp17ex, user 1662625689, client 192.0.2.44:56119]
2022-09-07 13:06:45.717062+00:00 [notice] Creating TURN allocation (lifetime: 600 seconds) [TLS, session oavtk6dp17ex, user 1662625689, client 192.0.2.44:56119, relay 203.0.113.4:50232]

What one could also think about is to terminate SSL already at the (new) nginx layer4 load balancer for web as well, herewith eturnal would not need valid certs anymore. I am not sure however, if this would be a behaviour you desire.
...

• offer TURN (plain TCP) also via port 443

To offer TURN (plain TCP) also via port 443 next to TLS, I am not sure if nginx could handle this. I know that traefik can do this. Here one could think if traefik would take over the layer4 load balancer role entirely and listens on port 443 instead of jitsi web (including LE cert management, at least for the TURN part).
...

• making TURN service ports and host domains configurable with environment variables

currently they are basically hard-coded.
...

• allow to define an external TURN service

the stream template must be adjusted for that.

Let me know what you think.

Cheers, sando

P.S.

example logs from eturnal TURN service:

# docker logs docker-jitsi-meet-eturnal-1
No cookie is set or found. This limits the scripts functionality, installing, upgrading, rpc and getting a list of versions will not work.
Exec: /opt/eturnal/erts-13.0.3/bin/erlexec -noinput +Bd -boot /opt/eturnal/releases/1.10.1/start -mode embedded -boot_var SYSTEM_LIB_DIR /opt/eturnal/lib -config /opt/eturnal/releases/1.10.1/sys.config -args_file /opt/eturnal/releases/1.10.1/vm.args -erl_epmd_port 3470 -start_epmd false -- foreground
Root: /opt/eturnal
/opt/eturnal
2022-09-06 09:01:09.137859+00:00 [notice] Starting eturnal 1.10.1 on Erlang/OTP 25 (ERTS 13.0.3)
2022-09-06 09:01:09.138549+00:00 [info] Got no NOTIFY_SOCKET, notifications disabled
2022-09-06 09:01:09.138638+00:00 [info] Relay IPv4 address: 203.0.113.4 (port range: 50000-50500)
2022-09-06 09:01:09.138682+00:00 [info] Relay IPv6 address not configured
2022-09-06 09:01:09.138736+00:00 [info] Listening on 127.0.0.1:3470 (tcp) (Erlang protocol version 5)
2022-09-06 09:01:09.139819+00:00 [warning] TLS enabled without 'tls_crt_file', creating self-signed certificate
2022-09-06 09:01:09.264446+00:00 [info] Started mod_log_stun
2022-09-06 09:01:09.264980+00:00 [info] Listening on [::]:3478 (udp) (STUN/TURN)
2022-09-06 09:01:09.265406+00:00 [info] Listening on [::]:3478 (auto) (STUN/TURN)
2022-09-06 09:02:11.273038+00:00 [info] Accepting proxied connection: 192.0.2.44:12622 -> 172.21.0.3:3478 [TCP|TLS, session gilvdfveu03s]
2022-09-06 09:02:11.305957+00:00 [info] Accepting long-term STUN/TURN authentication [TCP, session gilvdfveu03s, user 1662541324, client 192.0.2.44:12622]
2022-09-06 09:02:11.306981+00:00 [notice] Creating TURN allocation (lifetime: 599 seconds) [TCP, session gilvdfveu03s, user 1662541324, client 192.0.2.44:12622, relay 203.0.113.4:50083]
2022-09-06 09:02:11.326639+00:00 [info] Accepting proxied connection: 192.0.2.44:29915 -> 172.21.0.3:3478 [TCP|TLS, session 2hn1290aaqzd]
2022-09-06 09:02:11.339387+00:00 [info] Accepting long-term STUN/TURN authentication [TCP, session gilvdfveu03s, user 1662541324, client 192.0.2.44:12622]
2022-09-06 09:02:11.339487+00:00 [info] Creating TURN permission for 172.21.0.5 [TCP, session gilvdfveu03s, user 1662541324, client 192.0.2.44:12622, relay 203.0.113.4:50083]
2022-09-06 09:02:11.360997+00:00 [info] Accepting long-term STUN/TURN authentication [TCP, session 2hn1290aaqzd, user 1662541297, client 192.0.2.44:29915]
2022-09-06 09:02:11.361402+00:00 [notice] Creating TURN allocation (lifetime: 599 seconds) [TCP, session 2hn1290aaqzd, user 1662541297, client 192.0.2.44:29915, relay 203.0.113.4:50070]
2022-09-06 09:02:11.374313+00:00 [info] Accepting long-term STUN/TURN authentication [TCP, session gilvdfveu03s, user 1662541324, client 192.0.2.44:12622]
2022-09-06 09:02:11.374462+00:00 [info] Creating TURN permission for 203.0.113.4 [TCP, session gilvdfveu03s, user 1662541324, client 192.0.2.44:12622, relay 203.0.113.4:50083]
2022-09-06 09:02:11.393806+00:00 [info] Accepting long-term STUN/TURN authentication [TCP, session 2hn1290aaqzd, user 1662541297, client 192.0.2.44:29915]
2022-09-06 09:02:11.393997+00:00 [info] Creating TURN permission for 172.21.0.5 [TCP, session 2hn1290aaqzd, user 1662541297, client 192.0.2.44:29915, relay 203.0.113.4:50070]
2022-09-06 09:02:11.419107+00:00 [info] Accepting long-term STUN/TURN authentication [TCP, session 2hn1290aaqzd, user 1662541297, client 192.0.2.44:29915]
2022-09-06 09:02:11.419280+00:00 [info] Creating TURN permission for 203.0.113.4 [TCP, session 2hn1290aaqzd, user 1662541297, client 192.0.2.44:29915, relay 203.0.113.4:50070]

[damencho]

For the debian packages we use by default coturn. I'm not sure we want to have a different one for the docker, but let's also hear @saghul voice here.
Anyway on a first glance your config is missing the blacklist of the internal networks, using this configuration you open your internal network to the world. Look at the denied peers list in coturn config https://github.com/jitsi/jitsi-meet/blob/master/doc/debian/jitsi-meet-turn/turnserver.conf

[sando38]

Thanks for your answer.

Anyway on a first glance your config is missing the blacklist of the internal networks

This could easily adopted by uncommenting this line.
https://github.com/jitsi/docker-jitsi-meet/pull/1398/files#diff-eddd101bd2c5696608ed9e7b05361e1d646ce37c52c6afab7324102428f193d8R21

This defaults to the list here:
https://rtcsec.com/article/cve-2020-26262-bypass-of-coturns-access-control-protection/#further-concerns-what-else

I could adapt the PR accordingly.

[damencho]

This defaults to the list here:
https://rtcsec.com/article/cve-2020-26262-bypass-of-coturns-access-control-protection/#further-concerns-what-else

This is exactly the list we use for coturn. But you need to test it, as your current configuration maybe expecting to communicate with the bridge over the internal network.
For coturn for example if you set the external address of the turnserver it uses the internal network for the bridge which is closed by the blacklist. Skipping the setting for the external address though makes the communication between the turnserver and the bridge using the bridge public address and port 10000 udp.

Also, you only enable udp turn, which doesn't make much sense as if you have already udp connectivity you will be using the bridge anyway. The point of having the turnserver is to have the TCP connection so you can fallback to it for networks with restricted udp usage. But for the TLS connection, you need valid certificates as you mentioned above.

[sando38]

thanks for your feedback.

But you need to test it

yes, you are right, this needs to be tested. Thats why I did not put it as default immediately. Even with enabling the recommended blacklist option in the configuration, TURN works, just checked it.

Also, you only enable udp turn, which doesn't make much sense as if you have already udp connectivity you will be using the bridge anyway. The point of having the turnserver is to have the TCP connection so you can fallback to it for networks with restricted udp usage.

yes, of course, therefore there is the auto multiplex-part in the listening section, which listens for TCP/TLS. You can see that the TURN service reacted to TCP, not UDP also in the logs I provided in the first post.

Dropping the UDP listener in the configuration was on the list as well for final cleanup. BTW: https://github.com/jitsi/jitsi-meet/blob/master/doc/debian/jitsi-meet-turn/turnserver.conf does also include UDP, right? ;)

[damencho]

Dropping the UDP listener in the configuration was on the list as well for final cleanup. BTW: https://github.com/jitsi/jitsi-meet/blob/master/doc/debian/jitsi-meet-turn/turnserver.conf does also include UDP, right? ;)

Yep, it does. It just disabled the clear TCP turn in favor of turns.

[saghul]

Hey @sando38 ! Thanks for putting this PR together!

Generally speaking, this setup is a close replica of the stuff we run in production and have experience with. We use coturn as our TURN server currently, so if we are to ship a builtin server, it ought to be that.

Users that use this setup will inevitably run into issues and ask for help. We can only help them with what we know, and the reality is we don't know eturnal.

This is the one and only reason why we cannot have this PR in its current shape.

Would you be open to switching to coturn? If so, read on :-)

---

For starters, the image should be based on either the official image, but it's likely better if we have one using our base image, so all images have a familiar setup. This is important.

Next, since no browser sets ALPN properly for TURNS, we are left with using a separate domain and SNI.

The straightforward path would be the Let's Encrypt case. Just request a separate domain, and send the connection over to the TURN container if it matches the TURN domain (for TURNS, so we can share the web port). I think the tricky part here might be having the LE certs in both the web and coturn containers, since we don't want to do TLS offloading in nginx, I think.

If custom certs are used, we'd need them in both containers too.

Generating self-signed certs would probably be a bad idea here.

Sidenote:

Also, you only enable udp turn, which doesn't make much sense as if you have already udp connectivity you will be using the bridge anyway. The point of having the turnserver is to have the TCP connection so you can fallback to it for networks with restricted udp usage. But for the TLS connection, you need valid certificates as you mentioned above.

Wouldn't enabling UDP on port 443 help, ever so slightly, on networks which block all ports other than 80/443, but not the specific protocol?

[sando38]

hello @saghul

thanks for your reply.

Generally speaking, this setup is a close replica of the stuff we run in production and have experience with. We use coturn as our TURN server currently, so if we are to ship a builtin server, it ought to be that.

I am glad it generally matches your design. And I can follow your point with having coturn and keeping it due to your experiences.

I did not yet use coturn, so I cannot help very much with that. However, since it was only a minor adjustment, I did update this PR to address your design considerations for a builtin TURN service:

For starters, the image should be based on either the official image, but it's likely better if we have one using our base image, so all images have a familiar setup. This is important.

Yes, we maintain eturnal TURN server solution on github. I adjusted the Dockerfile in this PR to also have an s6 overlay (also needed for the cronjob cert renewal), so it behaves very similar to your base image (debian-based, glibc, s6-overlay). I adjusted the eturnal eturnal/rootfs directory in this PR accordingly.

Next, since no browser sets ALPN properly for TURNS, we are left with using a separate domain and SNI.

yes, exactly. This is the adjustment I made in the web image rootfs/default/turn and rootfs/default/default part, including the cert generation for the additional domain.

The straightforward path would be the Let's Encrypt case. Just request a separate domain, and send the connection over to the TURN container if it matches the TURN domain (for TURNS, so we can share the web port). I think the tricky part here might be having the LE certs in both the web and coturn containers, since we don't want to do TLS offloading in nginx, I think.

see answer above and...

I included an update mechanism into this PR which includes the LE certs into the eturnal container and reloads the service (see eturnal/rootfs/etc/service.d/cron/run). The reloading action does not interrupt/close any existing/running TURN connections of users. That's basically one reason why I choose eturnal instead of coturn for this setup.

If custom certs are used, we'd need them in both containers too.

this could be integrated the same way, as above.

Wouldn't enabling UDP on port 443 help, ever so slightly, on networks which block all ports other than 80/443, but not the specific protocol?

that might help. In this case, the easiest way would be to map -p 443:3478/udp to the turn service. Currently, the PR does not include it, as I removed the UDP listener as discussed earlier. If you want, I can include it again in the PR.

EDIT (2022-09-11): Included the UDP listener again.

In general:

• TURN services have some challenges in docker environment with port ranges (you probably know)
  • this could be overcome with setting network_mode: host which needs some changes in the web container setup, so the traffic is forwarded correctly to the TURN service (it is no rocket science though) :)

Some considerations for eturnal:

• Certificates can be renewed without interrupting/closing any connections
• Logging is clearer. Additionally, the command eturnalctl sessions provides infos for existing TURN sessions. Both could be very helpful also in debugging scenarios. Here is an example log.
• the included Prometheus exporter (not activated by default - commented out) makes a better job.
• may not be relevant, but eturnal can also be hot upgraded with the upcoming release.

So, I invite you to test eturnal. We can also assist you herewith.

In any other cases, feel free to use this PR as a basis for your builtin TURN service. It is good, that coturn is continued to be maintained again. I any cases, if I can help with any other this, just drop in.

Cheers, sando

[saghul]

Sorry I missed the last reply @sando38 .

Thanks for making improvements! I'm afraid my original point still stands, we cannot offer a core container like a TURN server when we don't use it ourselves.

I'm not sure this will change in the future, but it's not currently under consideration.

You are welcome to maintain this image independently, I could add you to jitsi-contrib if you want to host it there even, and the handbook could mention how to use it.

[sando38]

hola @saghul,

no worries for the late reply.

You are welcome to maintain this image independently, I could add you to jitsi-contrib if you want to host it there even, and the handbook could mention how to use it.

Let me have a check next week regarding jitsi contributation, so I can see, what I potentially commit myself to ;) Sounds like a good option though.

Wishing you a great day,
sando

[sando38]

hi @saghul,

I think it is a good way to contribute the setup in the jitsi contributions.

I just wanted to quickly double check some general considerations with you:

• Offering TURNS via port 443

If we would like to offer it via port 443, we should think of a PR for the jitsi web container to include a solution similar to the files default and turn in this PR ... actually all of the web container adjustments from this PR.

This way paired with e.g. an environment variable BUILTIN_TURN=yes or similar we could integrate a switch, so that the web container can forward the traffic to the "integrated" TURN service. Default value could be no which implies a solution not proxied by the web container (in fact it is the status quo). We could think about better wording, but I hope you got my point.

• ACME certs

With a value yes like in the example above the web container would also request the respective domain cert turn.example.com to make SNI based routing, if ACME service is enabled. This could be an environment variable as well of course.

herewith, eturnal but also a potential coturn solution could just mount those certs in the container.

Please let me know what you think.

Have a great day!
sando