git over ssh via reverse proxy

[the-moog]

Hi, trying to get TCP+TLS mode in Taxy to work with Gitea

I can login to Gitea via ssh directly.

I know it is possible to use https's port (443) through some reverse proxies but in this case I am using a dedicated port, 222.
On trying ssh via TCP+TLS I get this reply.

kex_exchange_identification: banner line contains invalid characters
banner exchange: Connection to ###.###.###.### port 222: invalid format

And via the existing forward on port 443

kex_exchange_identification: Connection closed by remote host
Connection closed by ###.###.###.### port 443

Are these methods something that should work?

[the-moog]

OK, so a little more info: Comparing the initial detailed debug log from ssh between direct and via Taxy.
Using ssh -vvvv -l git git.server -i <priv key file> To connect to gitea via Taxy I get:

OpenSSH_9.2p1 Debian-2+deb12u3, OpenSSL 3.0.14 4 Jun 2024
debug1: Reading configuration data /home/moog/.ssh/config
debug1: /home/moog/.ssh/config line 4: Applying options for git.server
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 19: include /etc/ssh/ssh_config.d/*.conf matched no files
debug1: /etc/ssh/ssh_config line 21: Applying options for *
debug3: expanded UserKnownHostsFile '~/.ssh/known_hosts' -> '/home/moog/.ssh/known_hosts'
debug3: expanded UserKnownHostsFile '~/.ssh/known_hosts2' -> '/home/moog/.ssh/known_hosts2'
debug2: resolving "git.server" port 22
debug3: resolve_host: lookup git.server:22
debug3: ssh_connect_direct: entering
debug1: Connecting to git.server [XXX.XXX.XXX.XXX] port 22.
debug3: set_sock_tos: set socket 3 IP_TOS 0x10
debug1: Connection established.
debug1: identity file /home/user/.ssh/[email protected] type 0
debug1: identity file /home/user/.ssh/[email protected] type -1
debug1: identity file /home/user/.ssh/[email protected] type 0
debug1: identity file /home/user/.ssh/[email protected] type -1
debug1: Local version string SSH-2.0-OpenSSH_9.2p1 Debian-2+deb12u3
kex_exchange_identification: banner line contains invalid characters
banner exchange: Connection to XXX.XXX.XXX.XXX port 22: invalid format
.... Quits to shell after this

Where as doing the same via Taxy using ssh -vvvv -l git real-git.server -i <priv key file>

OpenSSH_9.2p1 Debian-2+deb12u3, OpenSSL 3.0.14 4 Jun 2024
debug1: Reading configuration data /home/moog/.ssh/config
debug1: /home/moog/.ssh/config line 29: Applying options for nas6
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 19: include /etc/ssh/ssh_config.d/*.conf matched no files
debug1: /etc/ssh/ssh_config line 21: Applying options for *
debug3: expanded UserKnownHostsFile '~/.ssh/known_hosts' -> '/home/moog/.ssh/known_hosts'
debug3: expanded UserKnownHostsFile '~/.ssh/known_hosts2' -> '/home/moog/.ssh/known_hosts2'
debug2: resolving "real-git.server" port 222
debug3: resolve_host: lookup real-git.server:222
debug3: ssh_connect_direct: entering
debug1: Connecting to nas6 [YYY.YYY.YYY.YYY] port 222.
debug3: set_sock_tos: set socket 3 IP_TOS 0x10
debug1: Connection established.
debug1: identity file /home/moog/.ssh/[email protected] type 0
debug1: identity file /home/moog/.ssh/[email protected] type -1
debug1: identity file /home/moog/.ssh/[email protected] type 0
debug1: identity file /home/moog/.ssh/[email protected] type -1
debug1: Local version string SSH-2.0-OpenSSH_9.2p1 Debian-2+deb12u3
debug1: Remote protocol version 2.0, remote software version OpenSSH_9.7
debug1: compat_banner: match: OpenSSH_9.7 pat OpenSSH* compat 0x04000000
.... Carries on fine after this

So it seems something thinks that
SSH-2.0-OpenSSH_9.2p1 Debian-2+deb12u3
is not the same as
SSH-2.0-OpenSSH_9.2p1 Debian-2+deb12u3

Which, as you can see, as far as the terminal is concerned (or the debug print function in ssh) these ARE the same. This provides a hint. Could this be an accidental included NULL terminator, CRLF or string length buffer overflow in Taxy's SSH banner response or the forwarded banner response (depending on how this works)?

[the-moog]

Hi. It's been a while and I've still not managed to get to the bottom of this. For now I've just put in a fixed IP to remove the need for a proxy on that app and a couple others. There are other services I need to bring up and at some point I will need one that is production ready.

Just slightly worried in that I've not noticed much activity here recently.
Are the devs taking a sabbatical?
I hope everything is well with them and that this project is able to continue.

Am kind of considering putting myself forward as a maintainer, though worried that I am a complete rust noob, so that's a bit daunting.
I like what I hear about the language and it's concepts and for some time have thought it worth studying and have been on the lookout for an opportunity to work with it on something non-mission critical (i.e. my job). Sadly the grey cells are not as capable as they once were and my employer (combined with increasing slowness) takes most of my time up. I would struggle to get my employer to allow me to use up yet more time on a some pet rust project when nobody else I work with is using rust - yet.

So helping out here could be a well constrained opportunity to learn rust and at the same time help grow a young open source project. Though I need the confidence that the time and effort is worth it, and that I would not be biting off more than I can handle or going down some rabbit hole that I should have left well alone?

BTW:
I agree with the author of #87 that is is a really nice project compared to the other two popular solutions. Yes they more capable and better resourced, but I think so complicated to setup and maintain compared to Taxy which I think already does 90% or more of what most people want out of a reverse proxy, without the complexity and bloat, plus it is much easier to install and maintain. Less complicated means less opportunity for vulns due to misconfiguration. Kind of critical to a program that has a direct internet connection.

Any thoughts on this?

If you like this and think it's worth it, are there any pointers on how to debug and where to start would be appreciated and maybe some thoughts as to the possible cause. Docs or references to study. I'm from a low level C/C++ background. Does GDB even work with rust? I understand internally it does not work in the same way as other languages does that go as far as it's abi or are there other tools I should get to know?

I mostly work in VSCode or VSCodium

Thoughts as to what can be done to the code or the setup or both to start debugging this, assuming this is worth the effort.

Cheers,
Jay.

[picoHz]

Hi, Jay @the-moog . Thank you for your enthusiasm for my software!

Just slightly worried in that I've not noticed much activity here recently.

To give you a quick update, the main blocker right now is the migration to hyper v1.0, the HTTP library that taxy depends on. Since this version introduces many breaking changes, I’m holding off on making large changes until the migration is complete. Additionally, taxy relies on warp, which still uses hyper 0.14, so I’m unable to proceed until warp supports hyper v1.0.

Any thoughts on this?

You’re absolutely right - ease of setup and maintenance is the main selling point of my project. However, due to this focus, taxy isn’t particularly optimized for performance and may not be ideal for high-traffic sites.

If you like this and think it's worth it, are there any pointers on how to debug and where to start would be appreciated and maybe some thoughts as to the possible cause.

Please allow me some time to reproduce the issue and analyze it.

[picoHz]

@the-moog
Do you use TCP over TLS mode in the port settings?
In this mode, taxy attempts to connect to the SSH server via TLS, which causes the error.
In this mode, taxy only accepts tls connections, which causes it to reject ssh clients.
You should use TCP mode to proxy the SSH connection.