Occasional Timeout waiting for SSH #79

sanieldalib · 2023-06-29T20:56:14Z

Hey there! Thanks for all your great work on this!

I sometimes get the following error when building an image from IPSW. My ssh_timeout is set to 180s, and I'm killing the tart process before beginning the build to ensure its not a result of too many VMs running (>2).

==> tart-cli.base: Waiting for SSH to become available...
==> tart-cli.base: Timeout waiting for SSH.

Here is what my source block looks like, its essentially identical to the templates provided in https://github.com/cirruslabs/macos-image-templates/blob/master/templates/vanilla-ventura.pkr.hcl

source "tart-cli" "base" {
  # You can find macOS IPSW URLs on various websites like https://ipsw.me/
  # and https://www.theiphonewiki.com/wiki/Beta_Firmware/Mac/13.x
  from_ipsw    = "${var.home}/macOS/UniversalMac_13.3_22E252_Restore.ipsw"
  vm_name      = var.vm_name
  cpu_count    = 4
  memory_gb    = 8
  disk_size_gb = 60
  ssh_username = "admin"
  ssh_password = "admin"
  ssh_timeout  = "180s"
  boot_command = [
    # hello, hola, bonjour, etc.
    "<wait60s><spacebar>",
    # Language
    "<wait30s>english<enter>",
    # Select Your Country and Region
    "<wait30s>united states<leftShiftOn><tab><leftShiftOff><spacebar>",
    # Written and Spoken Languages
    "<wait10s><leftShiftOn><tab><leftShiftOff><spacebar>",
    # Accessibility
    "<wait10s><leftShiftOn><tab><leftShiftOff><spacebar>",
    # Data & Privacy
    "<wait10s><leftShiftOn><tab><leftShiftOff><spacebar>",
    # Migration Assistant
    "<wait10s><tab><tab><tab><spacebar>",
    # Sign In with Your Apple ID
    "<wait10s><leftShiftOn><tab><leftShiftOff><leftShiftOn><tab><leftShiftOff><spacebar>",
    # Are you sure you want to skip signing in with an Apple ID?
    "<wait10s><tab><spacebar>",
    # Terms and Conditions
    "<wait10s><leftShiftOn><tab><leftShiftOff><spacebar>",
    # I have read and agree to the macOS Software License Agreement
    "<wait10s><tab><spacebar>",
    # Create a Computer Account
    "<wait10s>admin<tab><tab>admin<tab>admin<tab><tab><tab><spacebar>",
    # Enable Location Services
    "<wait10s><leftShiftOn><tab><leftShiftOff><spacebar>",
    # Are you sure you don't want to use Location Services?
    "<wait10s><tab><spacebar>",
    # Select Your Time Zone
    "<wait10s><tab>UTC<enter><leftShiftOn><tab><leftShiftOff><spacebar>",
    # Analytics
    "<wait10s><leftShiftOn><tab><leftShiftOff><spacebar>",
    # Screen Time
    "<wait10s><tab><spacebar>",
    # Siri
    "<wait10s><tab><spacebar><leftShiftOn><tab><leftShiftOff><spacebar>",
    # Choose Your Look
    "<wait10s><leftShiftOn><tab><leftShiftOff><spacebar>",
    # Enable Voice Over
    "<wait10s><leftAltOn><f5><leftAltOff><wait5s>v",
    # Now that the installation is done, open "System Settings"
    "<wait10s><leftAltOn><spacebar><leftAltOff>System Settings<enter>",
    # Navigate to "Sharing"
    "<wait10s><leftAltOn>f<leftAltOff>sharing<enter>",
    # Navigate to "Screen Sharing" and enable it
    "<wait10s><tab><down><spacebar>",
    # Navigate to "Remote Login" and enable it
    "<wait10s><tab><tab><tab><tab><tab><tab><spacebar>",
    # Open "Remote Login" details
    "<wait10s><tab><spacebar>",
    # Enable "Full Disk Access"
    "<wait10s><tab><spacebar>",
    # Click "Done"
    "<wait10s><leftShiftOn><tab><leftShiftOff><leftShiftOn><tab><leftShiftOff><spacebar>",
    # Disable Voice Over
    "<leftAltOn><f5><leftAltOff>",
  ]

  // A (hopefully) temporary workaround for Virtualization.Framework's
  // installation process not fully finishing in a timely manner
  create_grace_time = "30s"

The text was updated successfully, but these errors were encountered:

fkorotkov · 2023-06-30T15:57:44Z

Do you observe any failures during the installation process of the macOS? This might be related to boot_command not successfully doing the installation so your VM ends up without a user configured to be SSHed in via.

sridhar-vadlamani-ck · 2024-07-01T16:41:34Z

My vanilla script seems to be timing out as well.

    sonoma-vanilla.tart-cli.tart: 85%
    sonoma-vanilla.tart-cli.tart: 86%
    sonoma-vanilla.tart-cli.tart: 87%
    sonoma-vanilla.tart-cli.tart: 90%
    sonoma-vanilla.tart-cli.tart: 100%
==> sonoma-vanilla.tart-cli.tart: Waiting 30s to let the Virtualization.Framework's installation process to finish correctly...
==> sonoma-vanilla.tart-cli.tart: Updating virtual machine resources...
==> sonoma-vanilla.tart-cli.tart: Inspecting machine disk image...
==> sonoma-vanilla.tart-cli.tart: Getting partition table...
==> sonoma-vanilla.tart-cli.tart: Found a recovery partition at GPT entry 3, let's remove it to save space and allow for resizing the main partition...
==> sonoma-vanilla.tart-cli.tart: Successfully updated partitions!
==> sonoma-vanilla.tart-cli.tart: Starting the virtual machine...
==> sonoma-vanilla.tart-cli.tart: Waiting for the VNC server credentials from Tart...
==> sonoma-vanilla.tart-cli.tart: Retrieved VNC credentials, connecting...
    sonoma-vanilla.tart-cli.tart: If you want to view the screen of the VM, connect via VNC with the password "echo-family-amount-copy" to
    sonoma-vanilla.tart-cli.tart: vnc://127.0.0.1:62266
==> sonoma-vanilla.tart-cli.tart: Connected to the VNC!
==> sonoma-vanilla.tart-cli.tart: Typing the commands over VNC...
==> sonoma-vanilla.tart-cli.tart: Successfully started the virtual machine...
==> sonoma-vanilla.tart-cli.tart: Using SSH communicator to connect: 192.168.64.2
==> sonoma-vanilla.tart-cli.tart: Waiting for SSH to become available...
==> sonoma-vanilla.tart-cli.tart: Timeout waiting for SSH.
==> sonoma-vanilla.tart-cli.tart: Waiting for the tart process to exit..

No errors that I can see.

edigaryev · 2024-07-01T16:44:30Z

@sridhar-vadlamani-ck how are you invoking the Packer, from CI, or from a user-interactive Terminal?

Also this might be related to the DHCP shortage, please check out https://tart.run/faq/#changing-the-default-dhcp-lease-time.

sridhar-vadlamani-ck · 2024-07-01T18:58:40Z

It's being invoked in CI, from a non-interactive terminal. $TERM is set to xterm-256color but I haven't passed in any additional flags to the packer script. Are there any additional settings that would be needed to streamline building vanilla vms in CI?

This is my source:


source "tart-cli" "tart" {
  from_ipsw    = "${var.ipsw}"
  vm_name      = "vanilla"
  cpu_count    = 4
  memory_gb    = 8
  disk_size_gb = 40
  ssh_password = "admin"
  ssh_username = "admin"
  ssh_timeout  = "120s"
  boot_command = [
    # hello, hola, bonjour, etc.
    "<wait60s><spacebar>",
    # Language: most of the times we have a list of "English"[1], "English (UK)", etc. with
    # "English" language already selected. If we type "english", it'll cause us to switch
    # to the "English (UK)", which is not what we want. To solve this, we switch to some other
    # language first, e.g. "Italiano" and then switch back to "English". We'll then jump to the
    # first entry in a list of "english"-prefixed items, which will be "English".
    #
    # [1]: should be named "English (US)", but oh well 🤷
    "<wait30s>italiano<esc>english<enter>",
    # Select Your Country and Region
    "<wait30s>united states<leftShiftOn><tab><leftShiftOff><spacebar>",
    # Written and Spoken Languages
    "<wait10s><leftShiftOn><tab><leftShiftOff><spacebar>",
    # Accessibility
    "<wait10s><leftShiftOn><tab><leftShiftOff><spacebar>",
    # Data & Privacy
    "<wait10s><leftShiftOn><tab><leftShiftOff><spacebar>",
    # Migration Assistant
    "<wait10s><tab><tab><tab><spacebar>",
    # Sign In with Your Apple ID
    "<wait10s><leftShiftOn><tab><leftShiftOff><leftShiftOn><tab><leftShiftOff><spacebar>",
    # Are you sure you want to skip signing in with an Apple ID?
    "<wait10s><tab><spacebar>",
    # Terms and Conditions
    "<wait10s><leftShiftOn><tab><leftShiftOff><spacebar>",
    # I have read and agree to the macOS Software License Agreement
    "<wait10s><tab><spacebar>",
    # Create a Computer Account
    "<wait10s>admin<tab><tab>admin<tab>admin<tab><tab><tab><spacebar>",
    # Enable Location Services
    "<wait30s><leftShiftOn><tab><leftShiftOff><spacebar>",
    # Are you sure you don't want to use Location Services?
    "<wait10s><tab><spacebar>",
    # Select Your Time Zone
    "<wait10s><tab>UTC<enter><leftShiftOn><tab><leftShiftOff><spacebar>",
    # Analytics
    "<wait10s><leftShiftOn><tab><leftShiftOff><spacebar>",
    # Screen Time
    "<wait10s><tab><spacebar>",
    # Siri
    "<wait10s><tab><spacebar><leftShiftOn><tab><leftShiftOff><spacebar>",
    # Choose Your Look
    "<wait10s><leftShiftOn><tab><leftShiftOff><spacebar>",
    # Enable Voice Over
    "<wait10s><leftAltOn><f5><leftAltOff><wait5s>v",
    # Now that the installation is done, open "System Settings"
    "<wait10s><leftAltOn><spacebar><leftAltOff>System Settings<enter>",
    # Navigate to "Sharing"
    "<wait10s><leftAltOn>f<leftAltOff>sharing<enter>",
    # Navigate to "Screen Sharing" and enable it
    "<wait10s><tab><tab><tab><tab><tab><spacebar>",
    # Navigate to "Remote Login" and enable it
    "<wait10s><tab><tab><tab><tab><tab><tab><tab><tab><tab><tab><tab><tab><spacebar>",
    # Disable Voice Over
    "<leftAltOn><f5><leftAltOff>",
  ]

  // A (hopefully) temporary workaround for Virtualization.Framework's
  // installation process not fully finishing in a timely manner
  create_grace_time = "30s"
}

Also, it's not the DHCP shortage. Timeout is set to 10mins and the vanilla vm build was basically the first one on these machines. I'm still building it out.

fkorotkov · 2024-07-01T19:37:44Z

@sridhar-vadlamani-ck are you trying macOS Sequia Beta 2 host by any chance? We've see this issue on Sequia hosts.

sridhar-vadlamani-ck · 2024-07-01T19:59:16Z

@fkorotkov I'm building from a sonoma ipsw

fkorotkov · 2024-07-01T20:03:36Z

But what about the host? Just checking this. But please also check @edigaryev's tip about DHCP.

sridhar-vadlamani-ck · 2024-07-01T20:43:29Z

The host is also a sonoma instance.

% sw_vers
ProductName:            macOS
ProductVersion:         14.5
BuildVersion:           23F79

These are freshly launched instances with the DHCP lease time set to 10mins per the FAQ already.

sridhar-vadlamani-ck · 2024-07-01T20:44:42Z

I re-ran it a bunch of times, and of about 12 runs, one has succeeded. Everything else seems to fail with the SSH connection issue.

sridhar-vadlamani-ck · 2024-07-03T08:11:43Z

Is there anything else you'd like me to rule out?

edigaryev · 2024-07-03T08:20:47Z

Is there anything else you'd like me to rule out?

Can you connect to the VM using the ssh command after Packer emits the Using SSH communicator to connect: <IP> message?

sridhar-vadlamani-ck · 2024-07-04T08:26:52Z

% ssh -v [email protected]
OpenSSH_9.6p1, LibreSSL 3.3.6
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 21: include /etc/ssh/ssh_config.d/* matched no files
debug1: /etc/ssh/ssh_config line 54: Applying options for *
debug1: Authenticator provider $SSH_SK_PROVIDER did not resolve; disabling
debug1: Connecting to 192.168.64.12 [192.168.64.12] port 22.
debug1: connect to address 192.168.64.12 port 22: Connection refused
ssh: connect to host 192.168.64.12 port 22: Connection refused

==> sonoma-vanilla.tart-cli.ipsw: Connected to the VNC!
==> sonoma-vanilla.tart-cli.ipsw: Typing the commands over VNC...
==> sonoma-vanilla.tart-cli.ipsw: Successfully started the virtual machine...
==> sonoma-vanilla.tart-cli.ipsw: Using SSH communicator to connect: 192.168.64.12
==> sonoma-vanilla.tart-cli.ipsw: Waiting for SSH to become available...

sridhar-vadlamani-ck · 2024-07-05T18:36:25Z

The SSH issue is only happening when I build the vanilla image. If I build from another image directly, there are no issues.

edigaryev · 2024-07-07T08:38:47Z

The SSH issue is only happening when I build the vanilla image. If I build from another image directly, there are no issues.

This most likely has to do with Remote Login not being enabled in System Settings → Sharing from some reason in the guest.

Make sure that your boot_command sequence is correct for the IPSW you're using by observing how it interacts with the installer/macOS UI.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Occasional Timeout waiting for SSH #79

Occasional Timeout waiting for SSH #79

sanieldalib commented Jun 29, 2023 •

edited

Loading

fkorotkov commented Jun 30, 2023

sridhar-vadlamani-ck commented Jul 1, 2024

edigaryev commented Jul 1, 2024

sridhar-vadlamani-ck commented Jul 1, 2024 •

edited

Loading

fkorotkov commented Jul 1, 2024

sridhar-vadlamani-ck commented Jul 1, 2024 •

edited

Loading

fkorotkov commented Jul 1, 2024

sridhar-vadlamani-ck commented Jul 1, 2024

sridhar-vadlamani-ck commented Jul 1, 2024

sridhar-vadlamani-ck commented Jul 3, 2024

edigaryev commented Jul 3, 2024

sridhar-vadlamani-ck commented Jul 4, 2024

sridhar-vadlamani-ck commented Jul 5, 2024

edigaryev commented Jul 7, 2024 •

edited

Loading

Occasional Timeout waiting for SSH #79

Occasional Timeout waiting for SSH #79

Comments

sanieldalib commented Jun 29, 2023 • edited Loading

fkorotkov commented Jun 30, 2023

sridhar-vadlamani-ck commented Jul 1, 2024

edigaryev commented Jul 1, 2024

sridhar-vadlamani-ck commented Jul 1, 2024 • edited Loading

fkorotkov commented Jul 1, 2024

sridhar-vadlamani-ck commented Jul 1, 2024 • edited Loading

fkorotkov commented Jul 1, 2024

sridhar-vadlamani-ck commented Jul 1, 2024

sridhar-vadlamani-ck commented Jul 1, 2024

sridhar-vadlamani-ck commented Jul 3, 2024

edigaryev commented Jul 3, 2024

sridhar-vadlamani-ck commented Jul 4, 2024

sridhar-vadlamani-ck commented Jul 5, 2024

edigaryev commented Jul 7, 2024 • edited Loading

sanieldalib commented Jun 29, 2023 •

edited

Loading

sridhar-vadlamani-ck commented Jul 1, 2024 •

edited

Loading

sridhar-vadlamani-ck commented Jul 1, 2024 •

edited

Loading

edigaryev commented Jul 7, 2024 •

edited

Loading