Connecting with SSH timed out #1031

raine-works · 2024-04-26T17:48:56Z

What happened?
Attempting to connect to provider remote host via SSH times out with error: "Waiting for Devpod agent to come up..." Cannot connect via SSH with VSCode or the terminal. The Devpod CLI SSH command works as intended.

What did you expect to happen instead?
Devcontainer to be accessible via SSH.

How can we reproduce the bug? (as minimally and precisely as possible)

My devcontainer.json:

{
	"name": "Homelab",
	"build": {
		"dockerfile": "Dockerfile",
		"context": ".",
		"args": {
			"USERNAME": "${localEnv:USER}"
		}
	},
	"remoteUser": "${localEnv:USER}",
	"features": {
		"ghcr.io/devcontainers/features/github-cli:1": {}
	},
	"forwardPorts": [3000],
	"otherPortsAttributes": {
		"onAutoForward": "ignore"
	},
	"mounts": [
		"source=${env:HOME}${env:USERPROFILE}/.config/gh,target=/home/${localEnv:USER}/.config/gh,type=bind",
		"source=${env:HOME}${env:USERPROFILE}/.ssh,target=/home/${localEnv:USER}/.ssh,type=bind"
	],
	"postAttachCommand": "git config --global diff.tool ...",
	"postStartCommand": "pnpm config set store-dir ~/.pnpm-store && pnpm setup",
	"customizations": {
		"vscode": {
			"extensions": [
				"esbenp.prettier-vscode",
				"yoavbls.pretty-ts-errors",
				"dbaeumer.vscode-eslint",
				"bradlc.vscode-tailwindcss",
				"eamodio.gitlens",
				"ms-azuretools.vscode-docker"
			]
		}
	},
	"runArgs": ["--network=host"]
}

My Dockerfile:

FROM mcr.microsoft.com/devcontainers/typescript-node:20 AS base

ARG USERNAME
ENV SHELL="/bin/bash"

RUN apt update
RUN apt upgrade -y
RUN usermod -l ${USERNAME} -d /home/${USERNAME} -m node
RUN echo ${USERNAME} ALL=\(root\) NOPASSWD:ALL > /etc/sudoers.d/${USERNAME}

My SSH config:

# DevPod Start homelab.devpod
Host homelab.devpod
  ForwardAgent yes
  LogLevel error
  StrictHostKeyChecking no
  UserKnownHostsFile /dev/null
  HostKeyAlgorithms rsa-sha2-256,rsa-sha2-512,ssh-rsa
  ProxyCommand "C:\Program Files\DevPod\devpod-cli.exe" ssh --stdio --context default --user raine homelab
  User raine
# DevPod End homelab.devpod

Host virgo.lab.raineworks.com
  HostName virgo.lab.raineworks.com
  User raine

Local Environment:

devpod 0.5.5
Operating System: windows
ARCH of the OS: AMD64

DevPod Provider:

Remote provider: ssh

Anything else we need to know?
I am able to connect to the remote host via SSH directly.

Docker engine on remote host has the following custom remote access config:

[Service]
ExecStart=
ExecStart=/usr/bin/dockerd -H fd:// -H tcp://virgo.lab.raineworks.com:2375

The text was updated successfully, but these errors were encountered:

pascalbreuninger · 2024-04-29T17:23:36Z

@raine-works are you experiencing the same problem with the latest v0.5.6 release?

raine-works · 2024-04-29T19:46:50Z

I updated to v0.5.6 and I am still seeing the same error.

PS C:\Users\Raine> ssh homelab.devpod
13:40:39 info Waiting for devpod agent to come up...
13:41:02 info Waiting for devpod agent to come up...

I tried deleting that workspace and creating a new one after updating. I am seeing the same results. The devpod cli ssh command works but the standard ssh command does not.

89luca89 · 2024-04-30T15:54:34Z

Hi @raine-works can you please to a test for me?

Can you open the ssh config file (It should be $HOME\.ssh\config also on windows) and add a --debug to the ProxyCommand of devpod?

Like:

...
...
  ProxyCommand devpod ssh --debug --stdio --context default --user vscode distrobox
  User vscode

So we can get some more useful logs from your command?

raine-works · 2024-04-30T18:23:45Z

Here is my SSH config.

# DevPod Start hukt.devpod
Host hukt.devpod
  ForwardAgent yes
  LogLevel error
  StrictHostKeyChecking no
  UserKnownHostsFile /dev/null
  HostKeyAlgorithms rsa-sha2-256,rsa-sha2-512,ssh-rsa
  ProxyCommand "C:\Program Files\DevPod\devpod-cli.exe" ssh --debug --stdio --context default --user raine hukt
  User raine
# DevPod End hukt.devpod

Host virgo.lab.raineworks.com
  HostName virgo.lab.raineworks.com
  User raine

Here are the logs I am seeing after adding the --debug flag.

PS C:\Users\Raine> ssh hukt.devpod
12:21:05 debug Adding ssh keys to agent, disable via 'devpod context set-options -o SSH_ADD_PRIVATE_KEYS=false'
12:21:05 debug Run ssh-add C:\Users\Raine\.ssh\id_rsa
12:21:05 debug Error adding key C:\Users\Raine\.ssh\id_rsa to agent: Error connecting to agent: No such file or directory

exit status 2
12:21:05 debug Acquire workspace lock...
12:21:05 debug Acquired workspace lock...
12:21:05 debug execute inject script
12:21:05 debug download agent from https://github.com/loft-sh/devpod/releases/download/v0.5.6
12:21:05 debug Run command provider command: ${SSH_PROVIDER} command
12:21:25 debug done inject
12:21:25 debug done injecting
12:21:25 info Waiting for devpod agent to come up...

The "inject script" portion repeats endlessly.

89luca89 · 2024-05-02T12:16:11Z

Thanks @raine-works
can you update the ssh provider to the latest version? we implemented some fixes for how it behaves and those could potentially help for this issue

raine-works · 2024-05-02T16:19:48Z

I assume this can be done by deleting the provider and recreating it? I did that and Its still not working. I am seeing the same error message.

89luca89 · 2024-05-02T16:35:31Z

Can you post it?

raine-works · 2024-05-04T15:55:07Z

PS C:\Users\Raine> ssh hukt.devpod
11:53:18 debug Adding ssh keys to agent, disable via 'devpod context set-options -o SSH_ADD_PRIVATE_KEYS=false'
11:53:18 debug Run ssh-add C:\Users\Raine\.ssh\id_rsa
11:53:18 debug Error adding key C:\Users\Raine\.ssh\id_rsa to agent: Error connecting to agent: No such file or directory

exit status 2
11:53:18 debug Acquire workspace lock...
11:53:18 debug Acquired workspace lock...
11:53:18 debug execute inject script
11:53:18 debug download agent from https://github.com/loft-sh/devpod/releases/download/v0.5.6
11:53:18 debug Run command provider command: ${SSH_PROVIDER} command
11:53:38 debug done inject
11:53:38 debug done injecting
11:53:38 info Waiting for devpod agent to come up...
11:53:38 debug Inject Error: context deadline exceeded
11:53:38 debug done exec
11:53:41 debug execute inject script
11:53:41 debug download agent from https://github.com/loft-sh/devpod/releases/download/v0.5.6
11:53:41 debug Run command provider command: ${SSH_PROVIDER} command

Host hukt.devpod
  ForwardAgent yes
  LogLevel error
  StrictHostKeyChecking no
  UserKnownHostsFile /dev/null
  HostKeyAlgorithms rsa-sha2-256,rsa-sha2-512,ssh-rsa
  ProxyCommand "C:\Program Files\DevPod\devpod-cli.exe" ssh --debug --stdio --context default --user raine hukt
  User raine
# DevPod End hukt.devpod

Host virgo.lab.raineworks.com
  HostName virgo.lab.raineworks.com
  User raine

shanman190 · 2024-05-06T22:58:10Z

@raine-works, what OpenSSH version are you using? ssh -V

The default OpenSSH version from Windows is VERY old (8.1), so downloading the latest release from PowerShell/Win32-OpenSSH is probably necessary. Alternatively, if you don't make use of the SSH agent, you can disable the Windows service to fix it as well.

raine-works · 2024-05-07T06:32:42Z

OpenSSH_for_Windows_8.6p1, LibreSSL 3.4.3

shanman190 · 2024-05-07T14:48:13Z

Would you be open to upgrading by following this guide from Microsoft?
https://github.com/PowerShell/Win32-OpenSSH/wiki/Install-Win32-OpenSSH

It's most likely that the source (8.6) is incompatible with the target (likely 9.x).

raine-works · 2024-05-07T16:53:20Z

I updated to OpenSSH version 9.5.

PS C:\Users\Raine> ssh -V
OpenSSH_for_Windows_9.5p1, LibreSSL 3.8.2

I am still seeing a similar error.

PS C:\Users\Raine> ssh hukt.devpod
10:49:24 debug Adding ssh keys to agent, disable via 'devpod context set-options -o SSH_ADD_PRIVATE_KEYS=false'
10:49:24 debug Run ssh-add C:\Users\Raine\.ssh\id_rsa
10:49:24 debug Acquire workspace lock...
10:49:24 debug Acquired workspace lock...
10:49:24 debug execute inject script
10:49:24 debug download agent from https://github.com/loft-sh/devpod/releases/download/v0.5.6
10:49:24 debug Run command provider command: ${SSH_PROVIDER} command
10:49:44 debug done inject
10:49:44 debug done injecting
10:49:44 info Waiting for devpod agent to come up...
10:49:44 debug Inject Error: context deadline exceeded
10:49:44 debug done exec
10:49:47 debug execute inject script
10:49:47 debug download agent from https://github.com/loft-sh/devpod/releases/download/v0.5.6
10:49:47 debug Run command provider command: ${SSH_PROVIDER} command

shanman190 · 2024-05-07T16:57:06Z

So that "error" might happen a couple of times while the agent process is getting started. Then it normally goes away once the agent is ready to accept commands. Does it just continue to spin like that indefinitely?

raine-works · 2024-05-07T17:00:47Z

The longest I've let it run is for about 5 minutes. This is the log output after 3 minutes.

PS C:\Users\Raine> ssh hukt.devpod
10:57:09 debug Adding ssh keys to agent, disable via 'devpod context set-options -o SSH_ADD_PRIVATE_KEYS=false'
10:57:09 debug Run ssh-add C:\Users\Raine\.ssh\id_rsa
10:57:09 debug Acquire workspace lock...
10:57:09 debug Acquired workspace lock...
10:57:09 debug execute inject script
10:57:09 debug download agent from https://github.com/loft-sh/devpod/releases/download/v0.5.6
10:57:09 debug Run command provider command: ${SSH_PROVIDER} command
10:57:29 debug done inject
10:57:29 debug done injecting
10:57:29 info Waiting for devpod agent to come up...
10:57:29 debug Inject Error: context deadline exceeded
10:57:29 debug done exec
10:57:32 debug execute inject script
10:57:32 debug download agent from https://github.com/loft-sh/devpod/releases/download/v0.5.6
10:57:32 debug Run command provider command: ${SSH_PROVIDER} command
10:57:52 debug done inject
10:57:52 debug done injecting
10:57:52 info Waiting for devpod agent to come up...
10:57:52 debug Inject Error: context deadline exceeded
10:57:52 debug done exec
10:57:55 debug execute inject script
10:57:55 debug download agent from https://github.com/loft-sh/devpod/releases/download/v0.5.6
10:57:55 debug Run command provider command: ${SSH_PROVIDER} command
10:58:15 debug done inject
10:58:15 debug done injecting
10:58:15 info Waiting for devpod agent to come up...
10:58:15 debug Inject Error: context deadline exceeded
10:58:15 debug done exec
10:58:18 debug execute inject script
10:58:18 debug download agent from https://github.com/loft-sh/devpod/releases/download/v0.5.6
10:58:18 debug Run command provider command: ${SSH_PROVIDER} command
10:58:38 debug done inject
10:58:38 debug done injecting
10:58:38 info Waiting for devpod agent to come up...
10:58:38 debug Inject Error: context deadline exceeded
10:58:38 debug done exec
10:58:41 debug execute inject script
10:58:41 debug download agent from https://github.com/loft-sh/devpod/releases/download/v0.5.6
10:58:41 debug Run command provider command: ${SSH_PROVIDER} command
10:59:01 debug done inject
10:59:01 debug done injecting
10:59:01 info Waiting for devpod agent to come up...
10:59:01 debug Inject Error: context deadline exceeded
10:59:01 debug done exec
10:59:04 debug execute inject script
10:59:04 debug download agent from https://github.com/loft-sh/devpod/releases/download/v0.5.6
10:59:04 debug Run command provider command: ${SSH_PROVIDER} command
10:59:24 debug done inject
10:59:24 debug done injecting
10:59:24 info Waiting for devpod agent to come up...
10:59:24 debug Inject Error: context deadline exceeded
10:59:24 debug done exec
10:59:27 debug execute inject script
10:59:27 debug download agent from https://github.com/loft-sh/devpod/releases/download/v0.5.6
10:59:27 debug Run command provider command: ${SSH_PROVIDER} command
10:59:47 debug done inject
10:59:47 debug done injecting
10:59:47 info Waiting for devpod agent to come up...
10:59:47 debug Inject Error: context deadline exceeded
10:59:47 debug done exec
10:59:50 debug execute inject script
10:59:50 debug download agent from https://github.com/loft-sh/devpod/releases/download/v0.5.6
10:59:50 debug Run command provider command: ${SSH_PROVIDER} command
11:00:10 debug done inject
11:00:10 debug done injecting
11:00:10 info Waiting for devpod agent to come up...
11:00:10 debug Inject Error: context deadline exceeded
11:00:10 debug done exec

Sword-Breaker · 2024-05-08T14:15:05Z

I think there is a typo in 'https://github.com/loft-sh/devpod/releases/download/v0.5.6', the URL is 404.

raine-works · 2024-05-09T17:07:29Z

The default agent URL under general in the settings is https://github.com/loft-sh/devpod/releases/latest/download/ which redirects to https://github.com/loft-sh/devpod/releases/download/v0.5.6. What should the override agent URL be?

shanman190 · 2024-05-09T18:04:59Z

So that download URL is just the first fragment. The file name gets appended to it based upon the target systems OS and architecture.

It was a little misleading when I was debugging that area as well, but that part is just fine from what I've seen.

raine-works · 2024-05-12T20:16:37Z

I removed the agent from the remote machine and tried re-creating the workspace. The agent was successfully re-installed on the remote machine, but still hangs when trying to open a connection.

raine-works · 2024-05-16T22:14:48Z

I find it strange that the devpod cli ssh command can connect to the remote devcontainer but open-ssh in powershell can't.

PowerShell 7.4.2

Raine  ~  ♥ 16:04  ssh node-test.devpod
16:04:29 info Waiting for devpod agent to come up...
16:04:52 info Waiting for devpod agent to come up...
16:05:15 info Waiting for devpod agent to come up...
16:05:38 info Waiting for devpod agent to come up...
16:06:01 info Waiting for devpod agent to come up...
16:06:24 info Waiting for devpod agent to come up...
16:06:47 info Waiting for devpod agent to come up...
16:07:10 info Waiting for devpod agent to come up...
16:07:33 info Waiting for devpod agent to come up...
16:07:56 info Waiting for devpod agent to come up...
16:08:19 info Waiting for devpod agent to come up...
16:08:42 info Waiting for devpod agent to come up...
16:09:05 info Waiting for devpod agent to come up...
16:09:28 fatal connect to server: timeout waiting for instance connection: context deadline exceeded

Raine  ~  ♥ 16:09  devpod node-test
16:10:35 error Try using the --debug flag to see a more verbose output
16:10:35 fatal unknown command "node-test" for "devpod"

Raine  ~  ♥ 16:10  devpod ssh node-test
node ➜ ~ $ ls
node ➜ ~ $ ls -a
.   .bash_logout  .cache   .docker     .npm        .profile        .zprofile
..  .bashrc       .config  .gitconfig  .oh-my-zsh  .vscode-server  .zshrc
node ➜ ~ $ cd /workspaces/
node ➜ /workspaces $ ls
node-test
node ➜ /workspaces $ cd node-test/
node ➜ /workspaces/node-test (main) $ ls
CODE_OF_CONDUCT.md  LICENSE  README.md  SECURITY.md  node_modules  package.json  server.js  yarn.lock
node ➜ /workspaces/node-test (main) $ npm run start

> [email protected] start
> node server.js

Running on http://0.0.0.0:3000

raine-works · 2024-05-16T22:27:18Z

@shanman190

I copied the SSH config from the Windows file system and pasted it in the SSH config in WSL. I adjusted the proxy command to use /mnt/c.

# DevPod Start node-test.devpod
Host node-test.devpod
  ForwardAgent yes
  LogLevel error
  StrictHostKeyChecking no
  UserKnownHostsFile /dev/null
  HostKeyAlgorithms rsa-sha2-256,rsa-sha2-512,ssh-rsa
  ProxyCommand "/mnt/c/Program Files/DevPod/devpod-cli.exe" ssh --stdio --context default --user node node-test
  User node
# DevPod End node-test.devpod

In WSL it worked immediately.

OpenSSH_for_Windows_9.5p1, LibreSSL 3.8.2

OpenSSH_8.9p1 Ubuntu-3ubuntu0.7, OpenSSL 3.0.2 15 Mar 2022

My issue must be related to the open-ssh version like you suggested before. Although updating to 9.x didn't solve the problem.

raine-works added the kind/bug label Apr 26, 2024

89luca89 added the windows label May 2, 2024

89luca89 added the wait for user label May 2, 2024

deniseschannon added the wait-on-user label May 8, 2024 — with Linear

deniseschannon removed the wait for user label May 8, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Connecting with SSH timed out #1031

Connecting with SSH timed out #1031

raine-works commented Apr 26, 2024 •

edited

pascalbreuninger commented Apr 29, 2024

raine-works commented Apr 29, 2024 •

edited

89luca89 commented Apr 30, 2024

raine-works commented Apr 30, 2024 •

edited

89luca89 commented May 2, 2024

raine-works commented May 2, 2024 •

edited

89luca89 commented May 2, 2024

raine-works commented May 4, 2024

shanman190 commented May 6, 2024 •

edited

raine-works commented May 7, 2024

shanman190 commented May 7, 2024

raine-works commented May 7, 2024 •

edited

shanman190 commented May 7, 2024

raine-works commented May 7, 2024 •

edited

Sword-Breaker commented May 8, 2024

raine-works commented May 9, 2024 •

edited

shanman190 commented May 9, 2024

raine-works commented May 12, 2024 •

edited

raine-works commented May 16, 2024

raine-works commented May 16, 2024 •

edited

Connecting with SSH timed out #1031

Connecting with SSH timed out #1031

Comments

raine-works commented Apr 26, 2024 • edited

pascalbreuninger commented Apr 29, 2024

raine-works commented Apr 29, 2024 • edited

89luca89 commented Apr 30, 2024

raine-works commented Apr 30, 2024 • edited

89luca89 commented May 2, 2024

raine-works commented May 2, 2024 • edited

89luca89 commented May 2, 2024

raine-works commented May 4, 2024

shanman190 commented May 6, 2024 • edited

raine-works commented May 7, 2024

shanman190 commented May 7, 2024

raine-works commented May 7, 2024 • edited

shanman190 commented May 7, 2024

raine-works commented May 7, 2024 • edited

Sword-Breaker commented May 8, 2024

raine-works commented May 9, 2024 • edited

shanman190 commented May 9, 2024

raine-works commented May 12, 2024 • edited

raine-works commented May 16, 2024

raine-works commented May 16, 2024 • edited

raine-works commented Apr 26, 2024 •

edited

raine-works commented Apr 29, 2024 •

edited

raine-works commented Apr 30, 2024 •

edited

raine-works commented May 2, 2024 •

edited

shanman190 commented May 6, 2024 •

edited

raine-works commented May 7, 2024 •

edited

raine-works commented May 7, 2024 •

edited

raine-works commented May 9, 2024 •

edited

raine-works commented May 12, 2024 •

edited

raine-works commented May 16, 2024 •

edited