IMA and Keylime in the presence of package updates

[mruffin]

Environment

• OS/version: any
• Processor architecture: any
• TPM Manufacturer: any
• Keylime version: any

Scenario

• We use a package repository (e.g., a Debian repo) as the source for generating Keylime IMA allowlists.
• At time t=t0, we generate allowlist "L0" based on the current state of the repository at that time.
• L0 is used as the IMA policy for a keylime instance at a time t > t0.
• At time t=t1 the repository maintainer updates the packages in the repository.
  • New packages become available.
  • Some packages are replaced with newer versions.
  • Some packages disappear.
  • CVEs may be recorded on certain obsolete package versions
• Consequently a new keylime allow list, L1, is generated from the repository.
• At time t=t2 > t1 the IMA allow list for our keylime instance is updated with a union of L0 and L1 (L0 + L1)
• Next the target systems are updated according to the new repository. All obsolete packages are replaced or removed, but the system is not rebooted.
• Finally, the keylime instance is updated again with the new policy L1.

Expected behavior

• Specificity (no false positives): the keylime system does not record a failure during a correctly applied upgrade process as described above.
• Sensitivity (no false negatives): the keylime system finds any package that was not correctly updated during the upgrade process.

Actual behavior

False positives: a correctly upgraded keylime system fails attestation as soon as (a) the system is upgraded, (b) the keylime policy becomes L1, and (c) the keylime agent is restarted.

This happens because when the keylime agent is restarted, the entire IMA log is re-sent to the verifier and replayed. The IMA log contains hashes that were measured while the old packages were installed (those conforming to policy L0). Since the policy is now L1, and may not contain hashes of obsolete packages, errors are recorded even though the system is currently "in policy".

False negatives: the keylime system fails to find packages that have been erroneously left in place; this happens in a situation where the (a) system was upgraded in a faulty manner (some old packages were not upgraded) (b) the keylime policy has now become L1 and c) the keylime agent was not restarted, and no false positives were raised.

This happens because hashes of old measured binaries exist in the IMA hash tables in the Linux kernel, preventing the re-recording of old executed packages. Thus, any re-execution of obsolete binaries does not cause even an IMA recording event, leaving Keylime completely in the dark about the execution of proscribed binaries.

Steps to reproduce problem

Day One:

• On a freshly updated and rebooted machine use the latest generated Allowlist (Original) to enable Runtime Integrity Monitoring with IMA. The agent should begin attesting.

Day Two:

• Update the machine, but do not reboot it. Grab the generated Allowlist (Original) from the repo for that day. I try to add new ones daily.
• Update keylime with the new policy by restarting the agent with the tenant and new Allowlist. If attestation fails, check the verifier log for errors.
• Look for WARNING - Hashes for file path/to/file do not match and search for the file in the IMA log. It can be expected to find it there twice.

Figure 1: Current Version of package openssh-server before system update. The SHA256 hash can be found in the IMA log for this version. It is the only occurrence.

---

Figure 2: The openssh-server package has been updated on system to match the most recent version in repository.

---

Figure 3: The file /usr/sbin/sshd for version 1:8.9p1-3ubuntu0.4 is still in the IMA log after system update.

---

Figure 4: The file /usr/sbin/sshd for version 1:8.9p1-3ubuntu0.6 is also in the IMA log after the system update. Only the new hash is in the Allowlist L1.

---

Figure 5: Output from the Keylime Verifier Log. IMA log cannot match old hash for /usr/sbin/sshd.

[github-actions]

Thank you for reporting your first issue. If the issue relates to a change you intend to work on, please ask that someone assign it to you.

[stefanberger]

To pick one item from the list: For as long as you don't reboot the system and you restart the keylime agent or re-register the system using the tenant tool you have to have an allowlist with the old measurements of packages before the upgrade. If there was a list of hashes of files with CVEs in some sort of policy you would probably want to know about these offending hashes now so you have some indication to tell you to deal with these apps and/or reboot the system. If skipping over the old hashes was the solution then that would not let you know what happened before the upgrade. I think we would always want to look at the hashes on the system from the very beginning to see what has run there. Whether the offending applications are still running is a different problem.

[THS-on]

So here the idea from yesterdays meeting.
We allow the IMA log to be evaluated with different policies to a certain index. To prohibit rollback attacks. Old policies only apply if the TPM clock reboot and reset variables haven't changed.

User story would be:

1. Node gets added for IMA attestation with policy A
2. User wants to update Node
3. User adds via tenant new policy B that includes the new and old software packages. That now applies starting index x1.
4. User updates Node with new software packages
5. User adds via tenant new policy C that only includes new packages that now applies starting index x2.

Start -- A --> x1 -- B --> x2 -- C -->

• The main open question is how we handle deletion of policies etc. Do we delete old policies automatically or do we fully leverage the named policy feature and require manual interaction?
• Next do we allow setting the index manually or do we determine them dynamically via agent state?

Ideally we would also allow to update the agent without fully removing and re-adding it, but this is a mostly orthogonal problem.

[aplanas]

What is an index in this context?

Something to take into account is that some systems, like SLE Micro or MicroOS does support rollbacks. A rollback here is a generalization of a A/B installation (you are in A, you update the partition B, reboot to B and if something goes wrong go back to A), but with N possible installations (A/B/C/...)

Rollback is a very natural operation here. If the automatic health checker detect something fishy, the old boot entry will be set as default and the system will command a reboot to recover the last good known state.

This can influence how we want to keep the good list of IMA hashes and policies in Keylime

[THS-on]

What is an index in this context?

Index is just a position in the IMA log.

Rollback is indeed a use case that we should consider. Here would need to update the agent anyways on the verifier because of the reboot. Maybe we should solve the user story of scheduled reboot without false positives when we make this change.

[stefanberger]

If we start with indices in the log then I suppose we would have to make this a per-system index to be added on the -C add/update command line in some form. We'll also have to recognize kexec type of reboots versus warm reboots where in the former we need to keep the index and in the latter we could discard/reset the index -- something we didn't have to do so far.

[kkaarreell]

Do we need to verify every single line from the IMA log? Let's say there are 100 records in the IMA log and we want to update keylime policy for the measured system. At this point, keylime_tenant (or other tool) instructing the verifier to update the policy could also tell verifier to make a "record" of this policy update event, i.e. storing the current/latest IMA log index and the respective hash prior policy update. Since the system is still attested we can save this trusted state and in the future when replaying the IMA log verifier would know that there has been policy update at IMA log index 100. Therefore, if encountering mismatch at IMA log line 20 it would continue computing the running hash until line 100 and verify the hash against the former record. If these two matches, the verifier would continue computing the running hash until the end of the IMA log, matching everything passed line 100 against the new keylime policy and TPM.

[stefanberger]

Do we need to verify every single line from the IMA log?

A reason why one may wants to verify every single line may be that one wants to be able to detect whether applications with hashes in a blacklist were started on the system at some point even though they have been replaced with newer versions now. IMA won't tell you whether the apps are still running. We don't have support for a blacklist in the policy right now (afaict) but if we had one I think it would be important that at least the blacklist would still be 'applied' to the first n log entries. For now we would have to resort to 'missing' hashes in our list indicating untrusted applications.