diff --git a/data-explorer/customer-managed-keys.md b/data-explorer/customer-managed-keys.md index 690f07cf80..a7a1900b9e 100644 --- a/data-explorer/customer-managed-keys.md +++ b/data-explorer/customer-managed-keys.md @@ -19,7 +19,7 @@ This article shows you how to configure customer-managed keys. ## Configure Azure Key Vault -To configure customer-managed keys with Azure Data Explorer, you must [set two properties on the key vault](/azure/key-vault/key-vault-ovw-soft-delete): **Soft Delete** and **Do Not Purge**. These properties aren't enabled by default. To enable these properties, perform **Enabling soft-delete** and **Enabling Purge Protection** in [PowerShell](/azure/key-vault/key-vault-soft-delete-powershell) or [Azure CLI](/azure/key-vault/key-vault-soft-delete-cli) on a new or existing key vault. Only RSA keys of size 2048 are supported. For more information about keys, see [Key Vault keys](/azure/key-vault/about-keys-secrets-and-certificates#key-vault-keys). +To configure customer-managed keys with Azure Data Explorer, you must [set two properties on the key vault](/azure/key-vault/key-vault-ovw-soft-delete): **Soft Delete** and **Do Not Purge**. These properties aren't enabled by default. To enable these properties, perform **Enabling soft-delete** and **Enabling Purge Protection** in [PowerShell](/azure/key-vault/key-vault-soft-delete-powershell) or [Azure CLI](/azure/key-vault/key-vault-soft-delete-cli) on a new or existing key vault. RSA and RSA-HSM keys of size 2048, 3072, and 4096 are supported. To utilize RSA-HSM keys utilize C#, Azure CLI, PowerShell, or ARM Template methods below. For more information about keys, see [Key Vault keys](/azure/key-vault/about-keys-secrets-and-certificates#key-vault-keys). > [!NOTE] > For information about the limitations of using customer managed keys on leader and follower clusters, see [Limitations](follower.md#limitations). diff --git a/data-explorer/kusto/management/alter-merge-table-update-policy-command.md b/data-explorer/kusto/management/alter-merge-table-update-policy-command.md index 09b090cda0..d58c24f469 100644 --- a/data-explorer/kusto/management/alter-merge-table-update-policy-command.md +++ b/data-explorer/kusto/management/alter-merge-table-update-policy-command.md @@ -15,7 +15,6 @@ Changes the table's update policy. The [update policy](update-policy.md) simplif > > * The source table and the table for which the update policy is defined must be in the same database. > * The update policy function schema and the target table schema must match in their column names, types, and order. -> * If the policy already defined on the table, the PolicyObjects specified in the command are added to the array of PolicyObjects in the existing policy. ## Permissions @@ -31,7 +30,7 @@ You must have at least [Table Admin](../access-control/role-based-access-control |Name|Type|Required|Description| |--|--|--|--| -| *DatabaseName* | `string` | | The name of the database. When you run the command from the database context that contains the table to alter, *DatabaseName* is not required. | +| *DatabaseName* | `string` | | The name of the database. When you run the command from the database context that contains the table to alter, *DatabaseName* isn't required. | | *TableName* | `string` | :heavy_check_mark: | The name of the table. A wildcard, `*`, denotes all tables.| | *ArrayOfPolicyObjects* | `string` | :heavy_check_mark: | A serialized array of policy objects. For more information, see [update policy](update-policy.md).| @@ -39,6 +38,9 @@ You must have at least [Table Admin](../access-control/role-based-access-control Returns a JSON representation of the policy. +> [!WARNING] +> If an update policy is already defined on the table, the PolicyObjects specified in the command are added to the array of PolicyObjects in the existing policy. Entries aren't deduplicated, so if the exact same PolicyObject already exists in the current update policy, another entry will be added, potentially causing duplicates during ingestion time. + ## Example The following command changes the update policy for a table using [multi-line string literals](../query/scalar-data-types/string.md#multi-line-string-literals). diff --git a/data-explorer/kusto/management/alter-table-update-policy-command.md b/data-explorer/kusto/management/alter-table-update-policy-command.md index b8dc11eff7..c335254dfb 100644 --- a/data-explorer/kusto/management/alter-table-update-policy-command.md +++ b/data-explorer/kusto/management/alter-table-update-policy-command.md @@ -12,8 +12,9 @@ ms.date: 08/11/2024 Use this command to change the table update policy. The [update policy](update-policy.md) simplifies the process of syncing and updating data between two tables. When new data is inserted into the source table, a transformation query runs over this data to modify and insert the data into the target table. > [!NOTE] -> The source table and the table for which the update policy is defined must be in the same database. -> The update policy function schema and the target table schema must match in their column names, types, and order. +> +> * The source table and the table for which the update policy is defined must be in the same database. +> * The update policy function schema and the target table schema must match in their column names, types, and order. ## Permissions @@ -37,6 +38,11 @@ You must have at least [Table Admin](../access-control/role-based-access-control Returns a JSON representation of the policy. +> [!WARNING] +> The alter command *replaces* the current update policy with a new policy. If you would like to *add* a new entry to the array +> of policy objects, in addition to the current ones, use the +> [.alter-merge table policy update command](alter-merge-table-update-policy-command.md). + ## Example Change the update policy for a table (using [multi-line string literals](../query/scalar-data-types/string.md#multi-line-string-literals)): diff --git a/data-explorer/kusto/management/update-policy.md b/data-explorer/kusto/management/update-policy.md index ab066ed389..3cdad15a85 100644 --- a/data-explorer/kusto/management/update-policy.md +++ b/data-explorer/kusto/management/update-policy.md @@ -92,8 +92,9 @@ Each such object is represented as a JSON property bag, with the following prope |Property |Type | Description | |---------|---------|----------------| |IsEnabled |`bool` |States if update policy is *true* - enabled, or *false* - disabled| -|Source |`string` |Name of the table that triggers invocation of the update policy | -|Query |`string` |A query used to produce data for the update | +|Source |`string` |Name of the table that triggers invocation of the update policy. | +|SourceIsWildCard |`bool` |If *true*, the `Source` property can be a wildcard pattern. See [Update policy with source table wildcard pattern](#update-policy-with-source-table-wildcard-pattern) | +|Query |`string` |A query used to produce data for the update. | |IsTransactional |`bool` |States if the update policy is transactional or not, default is *false*. If the policy is transactional and the update policy fails, the source table isn't updated. | |PropagateIngestionProperties |`bool`|States if properties specified during ingestion to the source table, such as [extent tags](extent-tags.md) and creation time, apply to the target table. | |ManagedIdentity | `string` | The managed identity on behalf of which the update policy runs. The managed identity can be an object ID, or the `system` reserved word. The update policy must be configured with a managed identity when the query references tables in other databases or tables with an enabled [row level security policy](row-level-security-policy.md). For more information, see [Use a managed identity to run a update policy](update-policy-with-managed-identity.md). | @@ -105,6 +106,7 @@ Each such object is represented as a JSON property bag, with the following prope |---------|---------|----------------| |IsEnabled |`bool` |States if update policy is *true* - enabled, or *false* - disabled| |Source |`string` |Name of the table that triggers invocation of the update policy | +|SourceIsWildCard |`bool` |If *true*, the `Source` property can be a wildcard pattern. | |Query |`string` |A query used to produce data for the update | |IsTransactional |`bool` |States if the update policy is transactional or not, default is *false*. If the policy is transactional and the update policy fails, the source table isn't updated. | |PropagateIngestionProperties |`bool`|States if properties specified during ingestion to the source table, such as [extent tags](extent-tags.md) and creation time, apply to the target table. | @@ -144,6 +146,14 @@ Update policies take effect when data is ingested or moved to a source table, or > Data may be lost in all tables with an update policy relationship if the `replace` command is invoked. > Consider using `.set-or-append` instead. +## Update policy with source table wildcard pattern + +Update policy supports ingesting from multiple source tables that share the same pattern, while using the +same query as the update policy query. This is useful if you have several source tables, usually sharing the same schema (or a subset of columns that share a common schema), and you would like to trigger ingestion to a +single target table, when ingesting to either of those tables. In this case, instead of defining multiple +update policies, each for a single source table, you can define a single update policy with wildcard as `Source`.The `Query` of the update policy must comply with all source tables matching the pattern. +To reference the source table in the update policy query, you can use a special symbol named `$source_table`. See example in [Example of wild card update policy](#example-of-wild-card-update-policy). + ## Remove data from source table After ingesting data to the target table, you can optionally remove it from the source table. Set a soft-delete period of `0sec` (or `00:00:00`) in the source table's [retention policy](retention-policy.md), and the update policy as transactional. The following conditions apply: @@ -251,6 +261,86 @@ In this example, use an update policy with a simple function to perform ETL. Fir .alter-merge table MySourceTable policy retention softdelete = 0s ``` +## Example of wild card update policy + +The following example creates an update policy with a single entry on table `TargetTable`. The policy references all tables matching pattern `SourceTable*` as its source. +Any ingestion to a table which matches the pattern (in local database) will trigger the update policy, and ingest data to `TargetTable`, based on the update policy query. + +1. Create two source tables: + + ```kusto + .create table SourceTable1(Id:long, Value:string) + ``` + + ```kusto + .create table SourceTable2(Id:long, Value:string) + ``` + +1. Create the target table: + + ```kusto + .create table TargetTable(Id:long, Value:string, Source:string) + ``` + +1. Create a function which will serve as the `Query` of the update policy. The function uses the `$source_table` symbol to reference the `Source` of the update policy. Use `skipValidation=true` to skip validation during the create function, since `$source_table` is only known during update policy execution. The function is validated during the next step, when altering the update policy. + + ```kusto + .create function with(skipValidation=true) IngestToTarget() + { + $source_table + | parse Value with "I'm from table " Source + | project Id, Value, Source + } + ``` + +1. Create the update policy on `TargetTable`. The policy references all tables matching pattern `SourceTable*` as its source. + + ````kusto + .alter table TargetTable policy update + ```[{ + "IsEnabled": true, + "Source": "SourceTable*", + "SourceIsWildCard" : true, + "Query": "IngestToTarget()", + "IsTransactional": true, + "PropagateIngestionProperties": true + }]``` + + ```` + +1. Ingest to source tables. Both ingestions trigger the update policy: + + ```kusto + .set-or-append SourceTable1 <| + datatable (Id:long, Value:string) + [ + 1, "I'm from table SourceTable1", + 2, "I'm from table SourceTable1" + ] + ``` + + ```kusto + .set-or-append SourceTable2 <| + datatable (Id:long, Value:string) + [ + 3, "I'm from table SourceTable2", + 4, "I'm from table SourceTable2" + ] + ``` + +1. Query `TargetTable`: + + ```kusto + TargetTable + ``` + + |Id|Value|Source| + |---|---|---| + |1|I'm from table SourceTable1|SourceTable1| + |2|I'm from table SourceTable1|SourceTable1| + |3|I'm from table SourceTable2|SourceTable2| + |4|I'm from table SourceTable2|SourceTable2| + ## Related content * [Common scenarios for using table update policies](update-policy-common-scenarios.md)