Not showing upgraded packages with the package module

[cooperspencer]

What is the issue ?

We have a playbook to update our systems and we use the package module. After running it, I checked Ara and found this in the output:

Is it a bug or did I misconfigure it somewhere along the way?

What should be happening ?

I expected the updated packages.

[dmsimard]

Hi and thanks for the issue.

Could you provide a playbook to reproduce the issue ?
What does the console output look like ?

Also, please provide:

• version of ansible(-core)
• version of python
• version of operating system

[cooperspencer]

Ansible should be 6.5

ansible [core 2.13.8]
  python version = 3.9.18 (main, Nov 16 2023, 02:13:35) [GCC 8.5.0 20210514 (Red Hat 8.5.0-20.0.1)]
  jinja version = 3.1.3
  libyaml = True

The host I ran the playbook on is a Oracle Linux 8.

Playbook

- name: Update all packages on OL
  when: ansible_os_family == 'RedHat'
  block:
  - name: Update packages on OL 6
    when: ansible_distribution_major_version < '7'
    shell: yum -y update
    become: yes

  - name: Update packages on newer OL versions
    when: ansible_distribution_major_version >= '7'
    package:
      name: "*"
      state: latest
      update_cache: yes
    become: yes

  - name: run ksplice
    command: uptrack-upgrade -y
    ignore_errors: yes
    become: yes

  - name: Check if a reboot is required
    command: needs-restarting -r
    register: reboot_check
    changed_when: False
    ignore_errors: yes

  - name: Is a reboot required?
    ignore_errors: yes
    debug:
      msg: "Server requires a reboot."
    when: reboot_check.rc == 1

and this is the output:

PLAY [group_A:group_B:group_C:group_D:group_E:group_F] **********************************************************************************************************************************************************************************************************************************************************************************************************************

TASK [Gathering Facts] ******************************************************************************************************************************************************************************************************************************************************************************************************************************************************
ok: [xxx]

TASK [update : Get current time] ********************************************************************************************************************************************************************************************************************************************************************************************************************************************
ok: [xxx]

TASK [update : Update packages on OL 6] *************************************************************************************************************************************************************************************************************************************************************************************************************************************
skipping: [xxx]

TASK [update : Update packages on newer OL versions] ************************************************************************************************************************************************************************************************************************************************************************************************************************
ok: [xxx]

TASK [update : run ksplice] *************************************************************************************************************************************************************************************************************************************************************************************************************************************************
changed: [xxx]

TASK [update : Check if a reboot is required] *******************************************************************************************************************************************************************************************************************************************************************************************************************************
ok: [xxx]

TASK [update : Is a reboot required?] ***************************************************************************************************************************************************************************************************************************************************************************************************************************************
skipping: [xxx]

PLAY RECAP ******************************************************************************************************************************************************************************************************************************************************************************************************************************************************************
tootst12                   : ok=4    changed=1    unreachable=0    failed=0    skipped=2    rescued=0    ignored=0

[dmsimard]

There is nothing that jumps at me at first glance and I haven't encountered a similar issue before.

I would say ansible 6/ansible-core 2.13 is starting to be a little bit old (they're up to 9 and 2.16 respectively) but it should still work. If we can't figure it out, I'd be curious to know if the issue reproduces with more recent versions.

Otherwise, I wonder whether any data was recorded at all and it's a display issue or if there is missing data.

When browsing the result with missing details, could you click on the API link at the top? It should open the API response for that particular result.

Do you see any results there ? What does it look like ?

[cooperspencer]

I just clicked on API and there I get all the updated packages.

{
    "id": 166902,
    "content": {
        "changed": true,
        "invocation": {
            "module_args": {
                "allow_downgrade": false,
                "allowerasing": false,
                "autoremove": false,
                "bugfix": false,
                "cacheonly": false,
                "conf_file": null,
                "disable_excludes": null,
                "disable_gpg_check": false,
                "disable_plugin": [],
                "disablerepo": [],
                "download_dir": null,
                "download_only": false,
                "enable_plugin": [],
                "enablerepo": [],
                "exclude": [],
                "install_repoquery": true,
                "install_weak_deps": true,
                "installroot": "/",
                "list": null,
                "lock_timeout": 30,
                "name": [
                    "*"
                ],
                "nobest": false,
                "releasever": null,
                "security": false,
                "skip_broken": false,
                "sslverify": true,
                "state": "latest",
                "update_cache": true,
                "update_only": false,
                "validate_certs": true
            }
        },
        "msg": "",
        "rc": 0,
        "results": [
            "Installed: kernel-devel-4.18.0-513.18.1.el8_9.x86_64",
            "Installed: kernel-headers-4.18.0-513.18.1.el8_9.x86_64",
            "Installed: mdadm-4.2-8.0.3.el8.x86_64",
            "Installed: kernel-tools-4.18.0-513.18.1.el8_9.x86_64",
            "Installed: kernel-tools-libs-4.18.0-513.18.1.el8_9.x86_64",
            "Installed: systemd-239-78.0.4.el8.x86_64",
            "Installed: python3-perf-4.18.0-513.18.1.el8_9.x86_64",
            "Installed: systemd-libs-239-78.0.4.el8.i686",
            "Installed: systemd-libs-239-78.0.4.el8.x86_64",
            "Installed: systemd-pam-239-78.0.4.el8.x86_64",
            "Installed: systemd-udev-239-78.0.4.el8.x86_64",
            "Installed: btrfs-progs-5.15.1-1.el8.x86_64",
            "Installed: device-mapper-8:1.02.181-13.0.3.el8_9.x86_64",
            "Installed: mcelog-3:197-1.el8.x86_64",
            "Installed: device-mapper-event-8:1.02.181-13.0.3.el8_9.x86_64",
            "Installed: device-mapper-event-libs-8:1.02.181-13.0.3.el8_9.x86_64",
            "Installed: device-mapper-libs-8:1.02.181-13.0.3.el8_9.x86_64",
            "Installed: lvm2-8:2.03.14-13.0.3.el8_9.x86_64",
            "Installed: lvm2-libs-8:2.03.14-13.0.3.el8_9.x86_64",
            "Installed: glibc-2.28-236.0.1.el8_9.12.x86_64",
            "Installed: bpftool-4.18.0-513.18.1.el8_9.x86_64",
            "Installed: glibc-2.28-236.0.1.el8_9.12.i686",
            "Installed: glibc-common-2.28-236.0.1.el8_9.12.x86_64",
            "Installed: glibc-devel-2.28-236.0.1.el8_9.12.x86_64",
            "Installed: glibc-gconv-extra-2.28-236.0.1.el8_9.12.x86_64",
            "Installed: glibc-gconv-extra-2.28-236.0.1.el8_9.12.i686",
            "Installed: glibc-headers-2.28-236.0.1.el8_9.12.x86_64",
            "Installed: selinux-policy-3.14.3-128.0.4.el8_9.1.noarch",
            "Installed: selinux-policy-targeted-3.14.3-128.0.4.el8_9.1.noarch",
            "Installed: libsss_autofs-2.9.1-4.0.1.el8_9.5.x86_64",
            "Installed: libsss_certmap-2.9.1-4.0.1.el8_9.5.x86_64",
            "Installed: libsss_idmap-2.9.1-4.0.1.el8_9.5.x86_64",
            "Installed: libsss_nss_idmap-2.9.1-4.0.1.el8_9.5.x86_64",
            "Installed: kernel-uek-5.4.17-2136.329.3.1.el8uek.x86_64",
            "Installed: python36-oci-sdk-2.124.2-1.el8.x86_64",
            "Installed: kernel-uek-devel-5.4.17-2136.329.3.1.el8uek.x86_64",
            "Installed: libsss_sudo-2.9.1-4.0.1.el8_9.5.x86_64",
            "Installed: python3-cryptography-3.2.1-7.0.1.el8_9.x86_64",
            "Installed: python3-unbound-1.16.2-5.el8_9.2.x86_64",
            "Installed: sssd-client-2.9.1-4.0.1.el8_9.5.x86_64",
            "Installed: sssd-common-2.9.1-4.0.1.el8_9.5.x86_64",
            "Installed: glibc-langpack-en-2.28-236.0.1.el8_9.12.x86_64",
            "Installed: sssd-kcm-2.9.1-4.0.1.el8_9.5.x86_64",
            "Installed: unbound-libs-1.16.2-5.el8_9.2.x86_64",
            "Installed: binutils-devel-2.30-123.0.2.el8.x86_64",
            "Installed: sssd-nfs-idmap-2.9.1-4.0.1.el8_9.5.x86_64",
            "Installed: dracut-049-228.git20230802.0.3.el8.x86_64",
            "Installed: dracut-config-rescue-049-228.git20230802.0.3.el8.x86_64",
            "Installed: dracut-network-049-228.git20230802.0.3.el8.x86_64",
            "Installed: dracut-squash-049-228.git20230802.0.3.el8.x86_64",
            "Installed: sos-4.6.1-1.0.2.el8.noarch",
            "Installed: bcache-tools-1.0.8-3.101.0.3.el8.x86_64",
            "Installed: binutils-2.30-123.0.2.el8.x86_64",
            "Removed: python36-oci-sdk-2.120.0-1.el8.x86_64",
            "Removed: lvm2-8:2.03.14-13.0.2.el8_9.x86_64",
            "Removed: lvm2-libs-8:2.03.14-13.0.2.el8_9.x86_64",
            "Removed: selinux-policy-3.14.3-128.0.2.el8_9.1.noarch",
            "Removed: selinux-policy-targeted-3.14.3-128.0.2.el8_9.1.noarch",
            "Removed: bcache-tools-1.0.8-3.101.0.2.el8.x86_64",
            "Removed: mcelog-3:194-1.0.1.el8.x86_64",
            "Removed: mdadm-4.2-8.0.2.el8.x86_64",
            "Removed: binutils-2.30-123.0.1.el8.x86_64",
            "Removed: python3-cryptography-3.2.1-6.0.1.el8.x86_64",
            "Removed: binutils-devel-2.30-123.0.1.el8.x86_64",
            "Removed: sos-4.6.1-1.0.1.el8.noarch",
            "Removed: kernel-devel-4.18.0-477.13.1.el8_8.x86_64",
            "Removed: bpftool-4.18.0-513.11.0.1.el8_9.x86_64",
            "Removed: sssd-client-2.9.1-4.0.1.el8_9.x86_64",
            "Removed: sssd-common-2.9.1-4.0.1.el8_9.x86_64",
            "Removed: sssd-kcm-2.9.1-4.0.1.el8_9.x86_64",
            "Removed: sssd-nfs-idmap-2.9.1-4.0.1.el8_9.x86_64",
            "Removed: kernel-headers-4.18.0-513.11.0.1.el8_9.x86_64",
            "Removed: kernel-tools-4.18.0-513.11.0.1.el8_9.x86_64",
            "Removed: kernel-tools-libs-4.18.0-513.11.0.1.el8_9.x86_64",
            "Removed: btrfs-progs-5.15.1-0.el8.x86_64",
            "Removed: systemd-239-78.0.3.el8.x86_64",
            "Removed: systemd-libs-239-78.0.3.el8.x86_64",
            "Removed: systemd-libs-239-78.0.3.el8.i686",
            "Removed: systemd-pam-239-78.0.3.el8.x86_64",
            "Removed: systemd-udev-239-78.0.3.el8.x86_64",
            "Removed: kernel-uek-5.4.17-2136.321.4.el8uek.x86_64",
            "Removed: kernel-uek-devel-5.4.17-2136.321.4.el8uek.x86_64",
            "Removed: glibc-2.28-236.0.1.el8.7.x86_64",
            "Removed: glibc-2.28-236.0.1.el8.7.i686",
            "Removed: glibc-common-2.28-236.0.1.el8.7.x86_64",
            "Removed: glibc-devel-2.28-236.0.1.el8.7.x86_64",
            "Removed: glibc-gconv-extra-2.28-236.0.1.el8.7.x86_64",
            "Removed: glibc-gconv-extra-2.28-236.0.1.el8.7.i686",
            "Removed: glibc-headers-2.28-236.0.1.el8.7.x86_64",
            "Removed: glibc-langpack-en-2.28-236.0.1.el8.7.x86_64",
            "Removed: python3-perf-4.18.0-513.11.0.1.el8_9.x86_64",
            "Removed: libsss_autofs-2.9.1-4.0.1.el8_9.x86_64",
            "Removed: libsss_certmap-2.9.1-4.0.1.el8_9.x86_64",
            "Removed: libsss_idmap-2.9.1-4.0.1.el8_9.x86_64",
            "Removed: libsss_nss_idmap-2.9.1-4.0.1.el8_9.x86_64",
            "Removed: libsss_sudo-2.9.1-4.0.1.el8_9.x86_64",
            "Removed: unbound-libs-1.16.2-5.el8.x86_64",
            "Removed: device-mapper-8:1.02.181-13.0.2.el8_9.x86_64",
            "Removed: device-mapper-event-8:1.02.181-13.0.2.el8_9.x86_64",
            "Removed: device-mapper-event-libs-8:1.02.181-13.0.2.el8_9.x86_64",
            "Removed: device-mapper-libs-8:1.02.181-13.0.2.el8_9.x86_64",
            "Removed: python3-unbound-1.16.2-5.el8.x86_64",
            "Removed: dracut-049-228.git20230802.0.2.el8.x86_64",
            "Removed: dracut-config-rescue-049-228.git20230802.0.2.el8.x86_64",
            "Removed: dracut-network-049-228.git20230802.0.2.el8.x86_64",
            "Removed: dracut-squash-049-228.git20230802.0.2.el8.x86_64"
        ]
    },
    "delegated_to": [],
    "created": "2024-03-21T08:23:38.086825Z",
    "updated": "2024-03-21T08:23:38.086841Z",
    "started": "2024-03-21T08:17:54.265553Z",
    "ended": "2024-03-21T08:23:38.044480Z",
    "duration": "00:05:43.778927",
    "status": "ok",
    "changed": true,
    "ignore_errors": false,
    "host": 1612,
    "task": 88382,
    "play": 442,
    "playbook": 433
}

[cooperspencer]

And we are slowly upgrading our Ansible version, but it takes some time...

[dmsimard]

It feels like this could be an odd behaviour with using * as the package name and how the module returns the results.

I don't have an answer for now but it is worth looking into. I would expect that it would work fine with a proper list of packages.

[flowerysong]

This is an unfortunate choice of return structure on the part of the module; results normally means that this is a nested set of results from a loop (https://docs.ansible.com/ansible/latest/playbook_guide/playbooks_loops.html#registering-variables-with-a-loop) but here it's just a list of arbitrary strings from the module's processing, which is why ara is failing to display it correctly.

[flowerysong]

A quick hack:

diff --git a/ara/ui/templates/result.html b/ara/ui/templates/result.html
index c2eac09..8bca8f5 100644
--- a/ara/ui/templates/result.html
+++ b/ara/ui/templates/result.html
@@ -103,7 +103,7 @@
               <td>
                 {% if field == "diff" %}
                   {{ value | diff_result | format_data | safe }}
-                {% elif field == "results" %}
+                {% elif field == "results" and value|length > 0 and "changed" in value|first %}
                   {% include "partials/nested_results.html" with results=value %}
                 {% else %}
                   {{ value | format_data | safe }}

[dmsimard]

This is an unfortunate choice of return structure on the part of the module; results normally means that this is a nested set of results from a loop (https://docs.ansible.com/ansible/latest/playbook_guide/playbooks_loops.html#registering-variables-with-a-loop) but here it's just a list of arbitrary strings from the module's processing, which is why ara is failing to display it correctly.

Indeed, I had not gone that far yet but I believed the issue to be something like that. Thanks :)

The name of the variables and the format of the data they contain is really up to each individual modules.
Recently there was a mention of a prepared key that some modules seem to have a convention around: #540

Thinking out loud here: we could try to do a best effort guess in the template like suggested.

Other options might be:

• Guess in the view so the template gets values in the expected format:

  ara/ara/ui/views.py

  Lines 338 to 343 in c750fcd

  	def get(self, request, *args, **kwargs):
  	# Results can contain a wide array of non-ascii or binary characters, escape them
  	codecs.register_error("strict", codecs.lookup_error("surrogateescape"))
  	result = self.get_object()
  	serializer = serializers.DetailedResultSerializer(result)
  	return Response({"result": serializer.data, "static_generation": False, "page": "result"})

• Guess in the callback: such that the data gets sent to the API in the expected format (this doesn't help playbooks that have already been recorded)

I don't think we can get away from how the variable might end up being a string, a list of strings or a list of dicts.

I am OK handling it in the template for now {# with a link to reference to this issue #}.
@flowerysong: feel free to open a PR if you would like, otherwise I will eventually get to it.

[flowerysong]

Anything we do is going to be hacky, because (unlike prepared) this is not an ansible-core convention, but a collision between engine choices (loop results are put in results, results is not a reserved key) and every module's ability to return arbitrary data.

Like, you suggest guessing in the view or callback so that the template gets the expected format, but what would that actually entail? The template as it is now expects results to contain a list of task results, but the results key from a module execution contains arbitrary data (string, dict, or list) so it's not something that can or should be transformed into a list of task results. AFAICS the only transformation you could do at those points is rename the key, and that will just confuse people.

EDIT: I don't know enough about Django templating to be confident that my change won't break things, e.g. I don't know how it would behave if handed a dictionary.