feat: omegaconf inventory by @MatteoVoges #1173

ademariag · 2024-04-24T07:52:56Z

Proposed Changes

Implements OmegaConf based Inventory by @MatteoVoges (@kapicorp/nexenio-team)
Speeds up inventory loading and compilation (2x faster)

Instructions

Activate the new inventory

 global:
   compose-target-name: true
   inventory-backend: omegaconf

or use the command line flag --inventory-backend=omegaconf

Migrate your inventory to the new setup

Migrate your inventory
 ./kapitan compile --migrate

Docs and Tests

Tests added
Updated documentation

Incremental changes from OmegaConf work #1173

simu · 2024-05-27T07:27:02Z

kapitan/inventory/inventory.py

 from pydantic import BaseModel, Field
 from kapitan.errors import KapitanError
-from typing import Annotated, Dict, Any, Optional
+from typing import Dict


This import is deprecated in Python >= 3.9, cf. https://docs.python.org/3/library/typing.html#aliases-to-built-in-types

It's preferred to use dict as type annotation.

Suggested change

from typing import Dict

kapitan/inventory/inventory.py

simu · 2024-05-27T07:29:32Z

kapitan/inventory/inventory.py

@@ -100,10 +105,10 @@ def get_parameters(self, target_names: str | list[str], ignore_class_not_found:
 target = self.get_target(target_names, ignore_class_not_found)
 return target.parameters

- return {name: target.parameters for name, target in self.get_targets(target_names)}
+ return {name: {"parameters": Dict(target.parameters)} for name, target in self.get_targets(target_names)}


Suggested change

return {name: {"parameters": Dict(target.parameters)} for name, target in self.get_targets(target_names)}

return {name: {"parameters": dict(target.parameters)} for name, target in self.get_targets(target_names)}

simu · 2024-05-27T07:47:16Z

kapitan/targets.py


 logger = logging.getLogger(__name__)


 def compile_targets(
- inventory_path, search_paths, output_path, parallel, targets, labels, ref_controller, **kwargs
+ inventory_path, search_paths, output_path, parallelism, desidered_targets, labels, ref_controller, **kwargs


Is desidered_targets a typo of desired_targets?

simu · 2024-05-27T08:48:39Z

kapitan/cached.py


 inv = cache_dict["inv"]
+ global_inv = Dict(inv.inventory)


From some local testing it seems like using kadet.Dict() to construct the inventory dict here is quite slow. When replacing this with dict() I see a speedup of ~2s for each target compilation on one of our production inventories:

Suggested change

global_inv = Dict(inv.inventory)

global_inv = dict(inv.inventory)

without the change (parallelism 4):

Rendered inventory (1.98s): discovered 52 targets. Compiling 51/52 targets using 4 concurrent processes: (12 CPU detected) Compiled vshn-vcluster-dev (2.38s) Compiled cert-exporter (2.40s) Compiled openshift4-networking (2.17s) Compiled kyverno (4.25s) Compiled storageclass (2.13s) Compiled openshift-upgrade-controller (4.57s) Compiled cortex-tenant-ns-label (2.07s) Compiled openshift4-service-mesh (2.22s) Compiled openshift4-ingress (2.24s) Compiled openshift4-console (2.25s) Compiled openshift4-authentication (2.35s) Compiled networkpolicy (2.02s) Compiled openshift4-config (2.11s) TRACE: /home/simon/work/syn/commodore/vendor/argocd/component/common.libsonnet:15 Field `image` for selecting the container image `argocd` has been removed. Please use fields `registry` and `repository` instead TRACE: /home/simon/work/syn/commodore/vendor/argocd/component/common.libsonnet:15 Field `image` for selecting the container image `redis` has been removed. Please use fields `registry` and `repository` instead TRACE: /home/simon/work/syn/commodore/vendor/argocd/component/common.libsonnet:15 Field `image` for selecting the container image `argocd` has been removed. Please use fields `registry` and `repository` instead TRACE: /home/simon/work/syn/commodore/vendor/argocd/component/common.libsonnet:15 Field `image` for selecting the container image `kapitan` has been removed. Please use fields `registry` and `repository` instead TRACE: /home/simon/work/syn/commodore/vendor/argocd/component/common.libsonnet:15 Field `image` for selecting the container image `kapitan` has been removed. Please use fields `registry` and `repository` instead TRACE: /home/simon/work/syn/commodore/vendor/argocd/component/common.libsonnet:15 Field `image` for selecting the container image `vault_agent` has been removed. Please use fields `registry` and `repository` instead TRACE: /home/simon/work/syn/commodore/vendor/argocd/component/common.libsonnet:15 Field `image` for selecting the container image `vault_agent` has been removed. Please use fields `registry` and `repository` instead Compiled cert-manager (3.41s) Compiled argocd (4.64s) Compiled syn-openshift-distributed-tracing (2.48s) Compiled appuio-managed-openshift-reporting (2.26s) Compiled openshift4-dns (2.43s) Compiled topolvm (2.55s) Compiled vertical-pod-autoscaler (2.25s) Compiled rbac (2.24s) Compiled openshift4-registry (2.30s) Compiled syn-openshift-kiali (2.26s) Compiled cloudscale-cloud-controller-manager (2.19s) Compiled rook-ceph (4.44s) Compiled csi-cloudscale (2.33s) copying /home/simon/work/syn/commodore/vendor/espejo/manifests/v0.4.0/crds/espejo-crd.yaml -> /tmp/tmptabntxj6.kapitan/compiled/espejo/espejo/00_crds Compiled espejo (2.24s) Compiled vshn-keycloak-dev (2.42s) Compiled patch-operator (2.45s) Compiled appuio-cloud-reporting (2.40s) Compiled crossplane (2.59s) TRACE: /home/simon/work/syn/commodore/vendor/openshift4-api/component/monitoring.libsonnet:99 Monitoring disabled or component `prometheus` not present, not deploying ServiceMonitors Compiled openshift4-api (2.29s) copying /home/simon/work/syn/commodore/vendor/backup-k8up/crds/v2.7.2/02_k8up_crds.yaml -> /tmp/tmptabntxj6.kapitan/compiled/backup-k8up/backup-k8up Compiled backup-k8up (2.32s) Compiled steward (2.40s) Compiled openshift4-scheduling (2.35s) TRACE: /home/simon/work/syn/commodore/vendor/openshift4-logging/component/elasticsearch.libsonnet:130 Elasticsearch disabled, not deploying Elasticsearch stack Compiled openshift4-logging (3.15s) Compiled appcat (11.95s) TRACE: /home/simon/work/syn/commodore/dependencies/cilium/component/render-helm-values.jsonnet:69 Parameter `helm_values` is deprecated. Please move your configs to `cilium_helm_values`, `hubble_enterprise_helm_values` or `hubble_ui_helm_values`. Compiled cilium (3.25s) Compiled cluster-backup (2.41s) Compiled openshift4-slos (10.47s) Compiled emergency-credentials-controller (4.37s) Compiled prometheus-pushgateway (2.30s) Compiled openshift4-nodes (2.91s) Compiled appuio-cloud (4.93s) Compiled openshift4-terraform (2.59s) copying /home/simon/work/syn/commodore/inventory/classes/t-falling-frog-2211/manifests/c-appuio-lab-cloudscale-rma-0/appuio-cloud-reporting.yaml -> /tmp/tmptabntxj6.kapitan/compiled/adhoc-configurations/adhoc-configurations creating /tmp/tmptabntxj6.kapitan/compiled/adhoc-configurations/adhoc-configurations/billing copying /home/simon/work/syn/commodore/inventory/classes/t-falling-frog-2211/manifests/c-appuio-lab-cloudscale-rma-0/billing/cluster-role-bindings.yaml -> /tmp/tmptabntxj6.kapitan/compiled/adhoc-configurations/adhoc-configurations/billing copying /home/simon/work/syn/commodore/inventory/classes/t-falling-frog-2211/manifests/c-appuio-lab-cloudscale-rma-0/billing/cluster-roles.yaml -> /tmp/tmptabntxj6.kapitan/compiled/adhoc-configurations/adhoc-configurations/billing copying /home/simon/work/syn/commodore/inventory/classes/t-falling-frog-2211/manifests/c-appuio-lab-cloudscale-rma-0/billing/service-accounts.yaml -> /tmp/tmptabntxj6.kapitan/compiled/adhoc-configurations/adhoc-configurations/billing copying /home/simon/work/syn/commodore/inventory/classes/t-falling-frog-2211/manifests/c-appuio-lab-cloudscale-rma-0/cilium-bgp.yaml -> /tmp/tmptabntxj6.kapitan/compiled/adhoc-configurations/adhoc-configurations creating /tmp/tmptabntxj6.kapitan/compiled/adhoc-configurations/adhoc-configurations/common copying /home/simon/work/syn/commodore/inventory/classes/t-falling-frog-2211/manifests/c-appuio-lab-cloudscale-rma-0/common/tailscale-router-proxy-template.yaml -> /tmp/tmptabntxj6.kapitan/compiled/adhoc-configurations/adhoc-configurations/common copying /home/simon/work/syn/commodore/inventory/classes/t-falling-frog-2211/manifests/c-appuio-lab-cloudscale-rma-0/openshift-alertmanager-patch.yaml -> /tmp/tmptabntxj6.kapitan/compiled/adhoc-configurations/adhoc-configurations copying /home/simon/work/syn/commodore/inventory/classes/t-falling-frog-2211/manifests/c-appuio-lab-cloudscale-rma-0/selinux-any-scc.yaml -> /tmp/tmptabntxj6.kapitan/compiled/adhoc-configurations/adhoc-configurations creating /tmp/tmptabntxj6.kapitan/compiled/adhoc-configurations/adhoc-configurations/test-argosync-issue copying /home/simon/work/syn/commodore/inventory/classes/t-falling-frog-2211/manifests/c-appuio-lab-cloudscale-rma-0/test-argosync-issue/arisnet-argo.yaml -> /tmp/tmptabntxj6.kapitan/compiled/adhoc-configurations/adhoc-configurations/test-argosync-issue Compiled adhoc-configurations (2.53s) Compiled appuio-keycloak-dev (2.79s) Compiled secrets (2.24s) Compiled openshift4-monitoring (46.86s) Compiled alerts-exporter (3.23s) Compiled openshift-operators-redhat (2.26s) Compiled 51 targets in (53.61s)

with the change (parallelism 4):

Rendered inventory (2.00s): discovered 52 targets. Compiling 51/52 targets using 4 concurrent processes: (12 CPU detected) copying /home/simon/work/syn/commodore/vendor/espejo/manifests/v0.4.0/crds/espejo-crd.yaml -> /tmp/tmpdad28mil.kapitan/compiled/espejo/espejo/00_crds Compiled openshift4-service-mesh (0.30s) Compiled espejo (0.30s) Compiled openshift4-config (0.21s) Compiled syn-openshift-distributed-tracing (0.24s) Compiled networkpolicy (0.24s) Compiled appuio-cloud-reporting (0.45s) Compiled syn-openshift-kiali (0.54s) Compiled topolvm (0.69s) Compiled openshift-upgrade-controller (2.99s) Compiled cluster-backup (0.60s) Compiled rbac (0.28s) Compiled vshn-keycloak-dev (0.54s) Compiled vertical-pod-autoscaler (0.44s) Compiled openshift-operators-redhat (0.25s) Compiled patch-operator (0.72s) Compiled appuio-cloud (3.11s) TRACE: /home/simon/work/syn/commodore/vendor/openshift4-logging/component/elasticsearch.libsonnet:130 Elasticsearch disabled, not deploying Elasticsearch stack Compiled openshift4-logging (1.29s) Compiled alerts-exporter (1.34s) Compiled openshift4-dns (0.36s) Compiled prometheus-pushgateway (0.64s) Compiled openshift4-nodes (1.19s) Compiled openshift4-slos (9.57s) Compiled crossplane (0.61s) TRACE: /home/simon/work/syn/commodore/dependencies/cilium/component/render-helm-values.jsonnet:69 Parameter `helm_values` is deprecated. Please move your configs to `cilium_helm_values`, `hubble_enterprise_helm_values` or `hubble_ui_helm_values`. Compiled cilium (1.23s) Compiled openshift4-console (0.71s) Compiled appuio-keycloak-dev (0.67s) Compiled vshn-vcluster-dev (0.77s) Compiled storageclass (0.42s) TRACE: /home/simon/work/syn/commodore/vendor/argocd/component/common.libsonnet:15 Field `image` for selecting the container image `argocd` has been removed. Please use fields `registry` and `repository` instead TRACE: /home/simon/work/syn/commodore/vendor/argocd/component/common.libsonnet:15 Field `image` for selecting the container image `redis` has been removed. Please use fields `registry` and `repository` instead TRACE: /home/simon/work/syn/commodore/vendor/argocd/component/common.libsonnet:15 Field `image` for selecting the container image `argocd` has been removed. Please use fields `registry` and `repository` instead TRACE: /home/simon/work/syn/commodore/vendor/argocd/component/common.libsonnet:15 Field `image` for selecting the container image `kapitan` has been removed. Please use fields `registry` and `repository` instead Compiled openshift4-registry (0.51s) TRACE: /home/simon/work/syn/commodore/vendor/argocd/component/common.libsonnet:15 Field `image` for selecting the container image `kapitan` has been removed. Please use fields `registry` and `repository` instead TRACE: /home/simon/work/syn/commodore/vendor/argocd/component/common.libsonnet:15 Field `image` for selecting the container image `vault_agent` has been removed. Please use fields `registry` and `repository` instead TRACE: /home/simon/work/syn/commodore/vendor/argocd/component/common.libsonnet:15 Field `image` for selecting the container image `vault_agent` has been removed. Please use fields `registry` and `repository` instead Compiled secrets (0.42s) Compiled cloudscale-cloud-controller-manager (0.46s) Compiled cortex-tenant-ns-label (0.53s) Compiled argocd (3.46s) copying /home/simon/work/syn/commodore/vendor/backup-k8up/crds/v2.7.2/02_k8up_crds.yaml -> /tmp/tmpdad28mil.kapitan/compiled/backup-k8up/backup-k8up Compiled backup-k8up (0.58s) Compiled steward (0.49s) Compiled kyverno (2.57s) TRACE: /home/simon/work/syn/commodore/vendor/openshift4-api/component/monitoring.libsonnet:99 Monitoring disabled or component `prometheus` not present, not deploying ServiceMonitors Compiled openshift4-api (0.56s) Compiled csi-cloudscale (0.38s) Compiled openshift4-ingress (0.59s) Compiled openshift4-terraform (0.44s) Compiled appuio-managed-openshift-reporting (0.37s) Compiled appcat (10.05s) copying /home/simon/work/syn/commodore/inventory/classes/t-falling-frog-2211/manifests/c-appuio-lab-cloudscale-rma-0/appuio-cloud-reporting.yaml -> /tmp/tmpdad28mil.kapitan/compiled/adhoc-configurations/adhoc-configurations creating /tmp/tmpdad28mil.kapitan/compiled/adhoc-configurations/adhoc-configurations/billing copying /home/simon/work/syn/commodore/inventory/classes/t-falling-frog-2211/manifests/c-appuio-lab-cloudscale-rma-0/billing/cluster-role-bindings.yaml -> /tmp/tmpdad28mil.kapitan/compiled/adhoc-configurations/adhoc-configurations/billing copying /home/simon/work/syn/commodore/inventory/classes/t-falling-frog-2211/manifests/c-appuio-lab-cloudscale-rma-0/billing/cluster-roles.yaml -> /tmp/tmpdad28mil.kapitan/compiled/adhoc-configurations/adhoc-configurations/billing copying /home/simon/work/syn/commodore/inventory/classes/t-falling-frog-2211/manifests/c-appuio-lab-cloudscale-rma-0/billing/service-accounts.yaml -> /tmp/tmpdad28mil.kapitan/compiled/adhoc-configurations/adhoc-configurations/billing copying /home/simon/work/syn/commodore/inventory/classes/t-falling-frog-2211/manifests/c-appuio-lab-cloudscale-rma-0/cilium-bgp.yaml -> /tmp/tmpdad28mil.kapitan/compiled/adhoc-configurations/adhoc-configurations creating /tmp/tmpdad28mil.kapitan/compiled/adhoc-configurations/adhoc-configurations/common copying /home/simon/work/syn/commodore/inventory/classes/t-falling-frog-2211/manifests/c-appuio-lab-cloudscale-rma-0/common/tailscale-router-proxy-template.yaml -> /tmp/tmpdad28mil.kapitan/compiled/adhoc-configurations/adhoc-configurations/common copying /home/simon/work/syn/commodore/inventory/classes/t-falling-frog-2211/manifests/c-appuio-lab-cloudscale-rma-0/openshift-alertmanager-patch.yaml -> /tmp/tmpdad28mil.kapitan/compiled/adhoc-configurations/adhoc-configurations copying /home/simon/work/syn/commodore/inventory/classes/t-falling-frog-2211/manifests/c-appuio-lab-cloudscale-rma-0/selinux-any-scc.yaml -> /tmp/tmpdad28mil.kapitan/compiled/adhoc-configurations/adhoc-configurations creating /tmp/tmpdad28mil.kapitan/compiled/adhoc-configurations/adhoc-configurations/test-argosync-issue copying /home/simon/work/syn/commodore/inventory/classes/t-falling-frog-2211/manifests/c-appuio-lab-cloudscale-rma-0/test-argosync-issue/arisnet-argo.yaml -> /tmp/tmpdad28mil.kapitan/compiled/adhoc-configurations/adhoc-configurations/test-argosync-issue Compiled adhoc-configurations (0.52s) Compiled cert-manager (1.60s) Compiled emergency-credentials-controller (2.29s) Compiled openshift4-networking (0.36s) Compiled rook-ceph (3.06s) Compiled cert-exporter (0.66s) Compiled openshift4-scheduling (0.26s) Compiled openshift4-authentication (0.74s) Compiled openshift4-monitoring (40.73s) Compiled 51 targets in (45.81s)

kapitan/cached.py

simu · 2024-05-27T08:52:52Z

kapitan/cached.py


 inv = {}
+ global_inv = Dict({})


Suggested change

global_inv = Dict({})

global_inv = {}

simu · 2024-05-27T08:54:06Z

pyproject.toml

@@ -38,6 +38,7 @@ coverage = "^7.5.0"
 docker = "^7.0.0"
 pytest-md = "^0.2.0"
 pytest-emoji = "^0.2.0"
+regex = "^2024.5.10"


Should be a non-dev dependency, since the omegaconf migration implementation depends on the library

simu · 2024-05-28T14:52:44Z

kapitan/targets.py

+ shutil.rmtree(compile_path)
+ shutil.copytree(temp_compile_path, compile_path)
+ logger.debug("Copied %s into %s", temp_compile_path, compile_path)
+ logger.info(f"Compiled {len(targets)} targets in (%.2fs)", time.time() - compile_start)


I just noticed that this log message looks odd, you probably want either

Suggested change

logger.info(f"Compiled {len(targets)} targets in (%.2fs)", time.time() - compile_start)

logger.info(f"Compiled {len(targets)} targets (%.2fs)", time.time() - compile_start)

or

Suggested change

logger.info(f"Compiled {len(targets)} targets in (%.2fs)", time.time() - compile_start)

logger.info(f"Compiled {len(targets)} targets in %.2fs", time.time() - compile_start)

ademariag force-pushed the test/omegaconf branch from f9eaff9 to 0f982f9 Compare April 24, 2024 07:58

ademariag marked this pull request as ready for review April 24, 2024 07:59

ademariag changed the title ~~Test/omegaconf~~ feat: omegaconf inventory by @MatteoVoges Apr 24, 2024

This was referenced Apr 24, 2024

fix: implement compose_target_name #1138

Closed

feat: omegaconf inventory #1132

Closed

ademariag mentioned this pull request May 1, 2024

Incremental changes from OmegaConf work #1173 #1175

Merged

2 tasks

ademariag added a commit that referenced this pull request May 1, 2024

Merge pull request #1175 from kapicorp/move-to-pytest

0443a54

Incremental changes from OmegaConf work #1173

ademariag force-pushed the test/omegaconf branch 4 times, most recently from 249d61b to 69d5883 Compare May 13, 2024 23:43

feat: Implements omegaconf backend inventory

6668fdd

ademariag force-pushed the test/omegaconf branch from 69d5883 to 6668fdd Compare May 13, 2024 23:43

ademariag added 6 commits May 14, 2024 00:06

fix speed

dbd28eb

fix missing regexp module

4766cf7

fix migrate script path

2050718

fix migrate script path

ef52dbc

Improve timing with inventory load

ddce8a4

Improve timing with inventory load

41dc759

ademariag force-pushed the test/omegaconf branch from a439925 to 41dc759 Compare May 15, 2024 14:56

ademariag added 6 commits May 15, 2024 21:38

fix

e4bd659

adds to_yaml resolver

59bbebe

adds to_yaml resolver

5f5dd6d

adds to_yaml resolver

36ff756

fix: fix resolvers

9e08608

adds formatter

0604d42

gbamaxw approved these changes May 16, 2024

View reviewed changes

ademariag added 3 commits May 18, 2024 13:14

fix tests

cac1f7c

Remove remote inventory code

7b42cb3

Remove extra info statements

9070bc0

ademariag added 4 commits May 20, 2024 12:10

upgrade poetry

b68e8d8

Upgrade poetry

008d83e

Upgrade poetry

f2e5cb7

Fix resolver load

6baf45e

simu reviewed May 27, 2024

View reviewed changes

simu reviewed May 28, 2024

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: omegaconf inventory by @MatteoVoges #1173

feat: omegaconf inventory by @MatteoVoges #1173

ademariag commented Apr 24, 2024 •

edited

simu May 27, 2024

simu May 27, 2024

simu May 27, 2024

simu May 27, 2024

simu May 27, 2024

simu May 27, 2024

simu May 28, 2024

	return {name: {"parameters": Dict(target.parameters)} for name, target in self.get_targets(target_names)}
	return {name: {"parameters": dict(target.parameters)} for name, target in self.get_targets(target_names)}

	global_inv = Dict(inv.inventory)
	global_inv = dict(inv.inventory)

	logger.info(f"Compiled {len(targets)} targets in (%.2fs)", time.time() - compile_start)
	logger.info(f"Compiled {len(targets)} targets (%.2fs)", time.time() - compile_start)

feat: omegaconf inventory by @MatteoVoges #1173

Are you sure you want to change the base?

feat: omegaconf inventory by @MatteoVoges #1173

Conversation

ademariag commented Apr 24, 2024 • edited

Proposed Changes

Instructions

Activate the new inventory

Migrate your inventory to the new setup

Docs and Tests

simu May 27, 2024

Choose a reason for hiding this comment

simu May 27, 2024

Choose a reason for hiding this comment

simu May 27, 2024

Choose a reason for hiding this comment

simu May 27, 2024

Choose a reason for hiding this comment

simu May 27, 2024

Choose a reason for hiding this comment

simu May 27, 2024

Choose a reason for hiding this comment

simu May 28, 2024

Choose a reason for hiding this comment

ademariag commented Apr 24, 2024 •

edited