Azure image-id from marketplace with :latest fails

[cg505]

Referencing a "latest" VM image from the Azure marketplace like Canonical:ubuntu-24_04-lts:server:latest does not work:

Traceback (most recent call last):
  File "/home/cooperc/.local/bin/sky", line 8, in <module>
    sys.exit(cli())
             ^^^^^
  File "/home/cooperc/a/sky-venv/lib/python3.11/site-packages/click/core.py", line 1157, in __call__
    return self.main(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/cooperc/a/sky-venv/lib/python3.11/site-packages/click/core.py", line 1078, in main
    rv = self.invoke(ctx)
         ^^^^^^^^^^^^^^^^
  File "/home/cooperc/a/skypilot/sky/utils/common_utils.py", line 366, in _record
    return f(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^
  File "/home/cooperc/a/skypilot/sky/cli.py", line 838, in invoke
    return super().invoke(ctx)
           ^^^^^^^^^^^^^^^^^^^
  File "/home/cooperc/a/sky-venv/lib/python3.11/site-packages/click/core.py", line 1688, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/cooperc/a/sky-venv/lib/python3.11/site-packages/click/core.py", line 1434, in invoke
    return ctx.invoke(self.callback, **ctx.params)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/cooperc/a/sky-venv/lib/python3.11/site-packages/click/core.py", line 783, in invoke
    return __callback(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/cooperc/a/skypilot/sky/utils/common_utils.py", line 386, in _record
    return f(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^
  File "/home/cooperc/a/skypilot/sky/cli.py", line 1118, in launch
    task_or_dag = _make_task_or_dag_from_entrypoint_with_overrides(
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/cooperc/a/skypilot/sky/cli.py", line 818, in _make_task_or_dag_from_entrypoint_with_overrides
    task.set_resources_override(override_params)
  File "/home/cooperc/a/skypilot/sky/task.py", line 664, in set_resources_override
    new_resources = res.copy(**override_params)
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/cooperc/a/skypilot/sky/resources.py", line 1261, in copy
    resources = Resources(
                ^^^^^^^^^^
  File "/home/cooperc/a/skypilot/sky/resources.py", line 249, in __init__
    self._try_validate_image_id()
  File "/home/cooperc/a/skypilot/sky/resources.py", line 929, in _try_validate_image_id
    image_size = self.cloud.get_image_size(image_id, region)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/cooperc/a/skypilot/sky/clouds/azure.py", line 202, in get_image_size
    image = compute_client.virtual_machine_images.get(
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/cooperc/a/sky-venv/lib/python3.11/site-packages/azure/core/tracing/decorator.py", line 105, in wrapper_use_tracer
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/home/cooperc/a/sky-venv/lib/python3.11/site-packages/azure/mgmt/compute/v2024_07_01/operations/_operations.py", line 18308, in get
    raise HttpResponseError(response=response, error_format=ARMErrorFormat)
azure.core.exceptions.HttpResponseError: (InvalidParameter) The value of parameter version is invalid.
Code: InvalidParameter
Message: The value of parameter version is invalid.
Target: version

It seems like virtual_machine_images.get cannot cope with the latest version for some reason. Using a specific version (e.g. Canonical:ubuntu-24_04-lts:server:24.04.202411030) works.

Version & Commit info:

• sky -v: skypilot, version 1.0.0-dev0
• sky -c: skypilot, commit 6c9acac-dirty

[cg505]

@yika-luo any idea about this?

[yika-luo]

I don't think we currently support using "latest" as version as Azure SDK doesn't have an out of the box method for that. We can list the images to get the latest one

[cg505]

Ah... that's confusing to me since az vm image list and sites like https://documentation.ubuntu.com/azure/en/latest/azure-how-to/instances/find-ubuntu-images/ use latest as the version and it's quite hard to find actual version numbers.

[Yisaer]

Hi, @yika-luo

I'm interested in solving this issue. Could anyone tell me the expected behavior when recived latest tag in this case?

[cg505]

@Yisaer It should be equivalent to specifying latest in the Azure console or other Azure tools. I'm not exactly sure what that does, probably finds the most recently created image, but maybe something more complex. You may need to search for documentation from Azure on how it works.

[Yisaer]

Hi @cg505

I tried Azure cli and found it supports use Canonical:ubuntu-24_04-lts:server:latest to create an vm or spot machine, so I think it's reasonable to expect latest supported in the image-id in this case. I will try to find out how to support it later.