Merge pull request #397 from stephen-up/feature/create-indexes

Feature/create indexes
ClickHouse · Jan 6, 2025 · 82a405e · 82a405e
2 parents b9f9191 + 0a6cc4d
commit 82a405e
Show file tree

Hide file tree

Showing 3 changed files with 45 additions and 10 deletions.
diff --git a/CHANGELOG.md b/CHANGELOG.md
@@ -1,3 +1,18 @@
+### Next Version
+
+#### New Features
+* [ClickHouse indexes](https://clickhouse.com/docs/en/optimize/sparse-primary-indexes) are now fully supported for `table` materialization.
+The index config should be added to the model config. for instance: 
+  ```python
+  {{ config(
+         materialized='%s',
+         indexes=[{
+            'name': 'your_index_name',
+            'definition': 'your_column TYPE minmax GRANULARITY 2'
+         }]
+  ) }}
+  ```
+
 ### Release [1.8.7], 2025-01-05
 
 ### New Features

diff --git a/README.md b/README.md
@@ -117,6 +117,7 @@ your_profile_name:
 | settings               | A map/dictionary of "TABLE" settings to be used to DDL statements like 'CREATE TABLE' with this model                                                                                                                                                                                                                |                |
 | query_settings         | A map/dictionary of ClickHouse user level settings to be used with `INSERT` or `DELETE` statements in conjunction with this model                                                                                                                                                                                    |                |
 | ttl                    | A TTL expression to be used with the table.  The TTL expression is a string that can be used to specify the TTL for the table.                                                                                                                                                                                       |                |
+| indexes                | A list of indexes to create, available only for `table` materialization. For examples look at ([#397](https://github.com/ClickHouse/dbt-clickhouse/pull/397))                                                                                                                                                        |                |
 
 ## Column Configuration
 
@@ -373,25 +374,32 @@ refreshable config object):
 | depends_on_validation | Whether to validate the existence of the dependencies provided in `depends_on`. In case a dependency doesn't contain a schema, the validation occurs on schema `default` |          | False         |
 
 A config example for refreshable materialized view:
+
 ```python
 {{
     config(
         materialized='materialized_view',
         refreshable={
-                "interval": "EVERY 5 MINUTE",
-                "randomize": "1 MINUTE",
-                "append": True,
-                "depends_on": ['schema.depend_on_model'],
-                "depends_on_validation": True
+            "interval": "EVERY 5 MINUTE",
+            "randomize": "1 MINUTE",
+            "append": True,
+            "depends_on": ['schema.depend_on_model'],
+            "depends_on_validation": True
         }
-        )
+    )
 }}
 ```
 
 ### Limitations
-* When creating a refreshable materialized view (MV) in ClickHouse that has a dependency, ClickHouse does not throw an error if the specified dependency does not exist at the time of creation. Instead, the refreshable MV remains in an inactive state, waiting for the dependency to be satisfied before it starts processing updates or refreshing.
-This behavior is by design, but it may lead to delays in data availability if the required dependency is not addressed promptly. Users are advised to ensure all dependencies are correctly defined and exist before creating a refreshable materialized view.
-* As of today, there is no actual "dbt linkage" between the mv and its dependencies, therefore the creation order is not guaranteed.
+
+* When creating a refreshable materialized view (MV) in ClickHouse that has a dependency, ClickHouse does not throw an
+  error if the specified dependency does not exist at the time of creation. Instead, the refreshable MV remains in an
+  inactive state, waiting for the dependency to be satisfied before it starts processing updates or refreshing.
+  This behavior is by design, but it may lead to delays in data availability if the required dependency is not addressed
+  promptly. Users are advised to ensure all dependencies are correctly defined and exist before creating a refreshable
+  materialized view.
+* As of today, there is no actual "dbt linkage" between the mv and its dependencies, therefore the creation order is not
+  guaranteed.
 * The refreshable feature was not tested with multiple mvs directing to the same target model.
 
 # Dictionary materializations (experimental)

diff --git a/dbt/include/clickhouse/macros/materializations/table.sql b/dbt/include/clickhouse/macros/materializations/table.sql
@@ -150,7 +150,9 @@
          {% if config.get('projections')%}
                 {{ projection_statement(relation) }}
          {% endif %}
-
+          {% if config.get('indexes') %}
+                {{ indexes_statement(relation) }}
+         {% endif %}
 
         {{ clickhouse__insert_into(relation, sql, has_contract) }}
     {%- endif %}
@@ -169,6 +171,16 @@
     {%- endfor %}
 {%- endmacro %}
 
+{% macro indexes_statement(relation) %}
+    {%- set indexes = config.get('indexes', default=[]) -%}
+
+    {%- for index in indexes %}
+         {% call statement('add_indexes') %}
+                ALTER TABLE {{ relation }} ADD INDEX {{ index.get('name') }} {{ index.get('definition') }}
+            {%endcall  %}
+    {%- endfor %}
+{%- endmacro %}
+
 {% macro create_table_or_empty(temporary, relation, sql, has_contract) -%}
     {%- set sql_header = config.get('sql_header', none) -%}