diff --git a/pages/data-warehouse/how-to/connect-applications.mdx b/pages/data-warehouse/how-to/connect-applications.mdx index cde63f2949..31ab2ecd04 100644 --- a/pages/data-warehouse/how-to/connect-applications.mdx +++ b/pages/data-warehouse/how-to/connect-applications.mdx @@ -17,7 +17,6 @@ To connect your deployment with BI tools, refer to the [dedicated documentation] - A Scaleway account logged into the [console](https://console.scaleway.com) - [Owner](/iam/concepts/#owner) status or [IAM permissions](/iam/concepts/#permission) allowing you to perform actions in the intended Organization -- [Signed up to the private beta](https://www.scaleway.com/fr/betas/) and received a confirmation email. - Created a [Data Warehouse deployment](/data-warehouse/how-to/create-deployment/) 1. Click **ClickHouse®** under **Data & Analytics** on the side menu. The Data Warehouse deployment page displays. diff --git a/pages/data-warehouse/how-to/connect-bi-tools.mdx b/pages/data-warehouse/how-to/connect-bi-tools.mdx index cee52e3485..195c40b5ae 100644 --- a/pages/data-warehouse/how-to/connect-bi-tools.mdx +++ b/pages/data-warehouse/how-to/connect-bi-tools.mdx @@ -15,7 +15,6 @@ This page explains how to integrate your Data Warehouse for ClickHouse® deploym - A Scaleway account logged into the [console](https://console.scaleway.com) - [Owner](/iam/concepts/#owner) status or [IAM permissions](/iam/concepts/#permission) allowing you to perform actions in the intended Organization -- [Signed up to the private beta](https://www.scaleway.com/fr/betas/) and received a confirmation email. - Created a [Data Warehouse deployment](/data-warehouse/how-to/create-deployment/) ## Tableau diff --git a/pages/data-warehouse/how-to/create-deployment.mdx b/pages/data-warehouse/how-to/create-deployment.mdx index f921343806..381617df3f 100644 --- a/pages/data-warehouse/how-to/create-deployment.mdx +++ b/pages/data-warehouse/how-to/create-deployment.mdx @@ -15,7 +15,6 @@ A Data Warehouse is a centralized repository designed to store, manage, and anal - A Scaleway account logged into the [console](https://console.scaleway.com) - [Owner](/iam/concepts/#owner) status or [IAM permissions](/iam/concepts/#permission) allowing you to perform actions in the intended Organization -- [Signed up to the private beta](https://www.scaleway.com/fr/betas/) and received a confirmation email. ## Creating a Data Warehouse Deployment diff --git a/pages/data-warehouse/how-to/edit-autoscaling.mdx b/pages/data-warehouse/how-to/edit-autoscaling.mdx index 345bf583d8..2713590b17 100644 --- a/pages/data-warehouse/how-to/edit-autoscaling.mdx +++ b/pages/data-warehouse/how-to/edit-autoscaling.mdx @@ -15,7 +15,7 @@ This page explains how to edit the autoscaling configuration of your Data Wareho - A Scaleway account logged into the [console](https://console.scaleway.com) - [Owner](/iam/concepts/#owner) status or [IAM permissions](/iam/concepts/#permission) allowing you to perform actions in the intended Organization -- [Signed up to the private beta](https://www.scaleway.com/fr/betas/) and received a confirmation email. + - Created a [Data Warehouse deployment](/data-warehouse/how-to/create-deployment/) diff --git a/pages/data-warehouse/how-to/import-data.mdx b/pages/data-warehouse/how-to/import-data.mdx index bd154c0294..70ce718e58 100644 --- a/pages/data-warehouse/how-to/import-data.mdx +++ b/pages/data-warehouse/how-to/import-data.mdx @@ -17,7 +17,7 @@ Scaleway Data Warehouse for ClickHouse® allows you to quickly import any type o - A Scaleway account logged into the [console](https://console.scaleway.com) - [Owner](/iam/concepts/#owner) status or [IAM permissions](/iam/concepts/#permission) allowing you to perform actions in the intended Organization -- [Signed up to the private beta](https://www.scaleway.com/fr/betas/) and received a confirmation email. + - Created a [Data Warehouse deployment](/data-warehouse/how-to/create-deployment/) ## How to import data using the ClickHouse® CLI diff --git a/pages/data-warehouse/how-to/manage-databases.mdx b/pages/data-warehouse/how-to/manage-databases.mdx index 6d8e402582..73d429ec01 100644 --- a/pages/data-warehouse/how-to/manage-databases.mdx +++ b/pages/data-warehouse/how-to/manage-databases.mdx @@ -9,29 +9,57 @@ dates: import Requirements from '@macros/iam/requirements.mdx' -This page explains how to manage databases in your Data Warehouse for ClickHouse® deployment using the [Scaleway console](https://console.scaleway.com/). - - -During the private beta phase, database management can only be done with the **scwadmin** & via the ClickHouse® CLI, MySQL or HTTPS protocols. - +This page explains how to manage databases in your Data Warehouse for ClickHouse® deployment. - A Scaleway account logged into the [console](https://console.scaleway.com) - [Owner](/iam/concepts/#owner) status or [IAM permissions](/iam/concepts/#permission) allowing you to perform actions in the intended Organization -- [Signed up to the private beta](https://www.scaleway.com/fr/betas/) and received a confirmation email. - Created a [Data Warehouse deployment](/data-warehouse/how-to/create-deployment/) ## How to manage databases using the Scaleway console -During the private beta phase, the management of databases in your Data Warehouse for ClickHouse® deployment is limited to viewing databases in your deployment. +The Scaleway console allows you to create and delete databases for your Data Warehouse for ClickHouse® deployment. + +### How to create a database using the Scaleway console + +1. Click **ClickHouse®** under **Data & Analytics** on the side menu. The Data Warehouse deployment page displays. + +2. Click the name of the desired Data Warehouse deployment. The **Overview** tab of the deployment displays. + +3. Click the **Databases** tab. A list of your current databases displays. + +4. Click **+ Create database**. A pop-up displays. + +5. Enter a name for your new database. It can can only contain alphanumeric characters, underscores, and dashes. + +6. Click **Create database** to confirm. + +Your new database appears in the list. + +### How to delete a database using the Scaleway console + + +The `default` database cannot be deleted. + + +1. Click **ClickHouse®** under **Data & Analytics** on the side menu. The Data Warehouse deployment page displays. + +2. Click the name of the desired Data Warehouse deployment. The **Overview** tab of the deployment displays. + +3. Click next to the name of the database you want to delete. A confirmation pop-up displays. + +4. Enter `DELETE`, then click **Delete database** to confirm. + +The deleted database no longer appears in the list. ## How to manage databases using frameworks -[Connect to your deployment](/data-warehouse/how-to/connect-applications/) using your preferred framework. - - You can also connect using the ClickHouse HTTP console from the **Overview** tab of your deployment. Make sure to enter valid credentials in the top-right fields. - +[Connect to your deployment](/data-warehouse/how-to/connect-applications/) using your preferred framework. + + +You can also connect using the ClickHouse HTTP console from the **Overview** tab of your deployment. Make sure to enter valid credentials in the top-right fields. + Once connected, you can run SQL queries to `CREATE/ALTER/DROP` your `DATABASE/TABLE`. diff --git a/pages/data-warehouse/how-to/manage-delete-deployment.mdx b/pages/data-warehouse/how-to/manage-delete-deployment.mdx index ceb5ae1771..496d84d680 100644 --- a/pages/data-warehouse/how-to/manage-delete-deployment.mdx +++ b/pages/data-warehouse/how-to/manage-delete-deployment.mdx @@ -15,7 +15,6 @@ This page explains how to manage and delete your Data Warehouse deployment. - A Scaleway account logged into the [console](https://console.scaleway.com) - [Owner](/iam/concepts/#owner) status or [IAM permissions](/iam/concepts/#permission) allowing you to perform actions in the intended Organization -- [Signed up to the private beta](https://www.scaleway.com/fr/betas/) and received a confirmation email. - Created a [Data Warehouse deployment](/data-warehouse/how-to/create-deployment/) ## How to manage a Data Warehouse deployment diff --git a/pages/data-warehouse/how-to/manage-storage.mdx b/pages/data-warehouse/how-to/manage-storage.mdx new file mode 100644 index 0000000000..01b7afa0f6 --- /dev/null +++ b/pages/data-warehouse/how-to/manage-storage.mdx @@ -0,0 +1,141 @@ +--- +title: How to manage data storage in a Data Warehouse for ClickHouse® deployment +description: This page explains how to manage data storage in a Data Warehouse for ClickHouse® deployment using storage policies +tags: data storage policy strategy options object block +dates: + validation: 2025-10-16 + posted: 2025-10-16 +--- + +import Requirements from '@macros/iam/requirements.mdx' + + + +- A Scaleway account logged into the [console](https://console.scaleway.com) +- [Owner](/iam/concepts/#owner) status or [IAM permissions](/iam/concepts/#permission) allowing you to perform actions in the intended Organization + +## Overview + +Scaleway Data Warehouse for ClickHouse® uses ClickHouse®'s native storage policies to decide where table data lives: on Scaleway Block storage, and/or on Object Storage (Amazon S3). Policies are applied at table-level during table creation, using the settings presented below. You can choose between three policies for your deployment: + +- `tiered` (default in Scaleway Data Warehouse for ClickHouse®) +- `s3_cache` (cache mode) +- `default` (local-only mode) + +The table below provides an overview of each storage policy's behavior and use case. + +| Storage policy | Behavior | When to use | +|------|---------|------------| +| `tiered` | Writes on Block Storage first, and parts are automatically moved to Object Storage when disk is 90% full. | General purpose setting for fast local writes and transparent spillover to Object Storage as the dataset grows. | +| `s3_cache` | Data is stored on Object Storage. A local Block Storage cache layer keeps frequently-read parts to accelerate repeated reads. | Large datasets that mostly live in Object Storage, with repeated reads on smaller subsets that benefit from local caching. | +| `default` | Data is stored on Block storage only. | Small datasets where lowest latency for reads/writes on local disk is desired and capacity fits the Block Storage volume. | + +## Tiered storage policy + +### Applying the tiered policy + +**Tiered** is the deployment's default policy. Data is stored on the `hot` volume (Block Storage) until it reaches 90% of its capacity, then data is moved to the `cold` volume. This mechanism is controlled by `move_factor:0.1`. + +[Connect to you deployment](/data-warehouse/how-to/connect-applications/), then run the SQL query below to apply the `tiered` storage policy. + + + + + Storage policy is explicitly defined as `tiered`. + + ```sql + CREATE TABLE tiered_table + ( + id UInt64, + ts DateTime, + value Float32 + ) + ENGINE = MergeTree + PARTITION BY toYYYYMM(ts) + ORDER BY id + SETTINGS storage_policy = 'tiered'; + ``` + + + + + Storage policy is not defined, ClickHouse® applies the `tiered` storage policy by default. + + ```sql + CREATE TABLE dw.tiered_table_implicit + ( + id UInt64, + ts DateTime, + value Float32 + ) + ENGINE = MergeTree + PARTITION BY toYYYYMM(ts) + ORDER BY id; + ``` + + + + +If you omit the `storage_policy` setting, ClickHouse® will automatically apply the `tiered` policy. + + +### Moving data with the tiered policy + +When using the `tiered` storage policy, you can manually move partitions between `hot` (Block Storage) and `cold` (Object Storage) volumes using the SQL queries below. + +- Moving specific partitions to the `cold` volume (Object Storage): + + ```sql + ALTER TABLE tiered_table MOVE PARTITION '2024-08' TO VOLUME 'cold'; + ``` + +- Moving a non-partitioned table (`tuple()`) to the `cold` volume (Object Storage): + + ```sql + ALTER TABLE tiered_table_without_partitions MOVE PARTITION tuple() TO VOLUME 'cold'; + ``` + +- Moving a partition to the `hot` volume (Block Storage): + + ```sql + ALTER TABLE tiered_table MOVE PARTITION '2024-08' TO VOLUME 'hot'; + ``` +- Moving a non-partitioned table (`tuple()`) to the `hot` volume (Block Storage): + + ```sql + ALTER TABLE tiered_table_without_partitions MOVE PARTITION tuple() TO VOLUME 'hot'; + ``` + +## S3 cache policy + +[Connect to your deployment](/data-warehouse/how-to/connect-applications/), then run the SQL query below to create a new table with the `s3_cache` policy enabled. +```sql +CREATE TABLE cache_table +( + id UInt64, + ts DateTime, + value Float32 +) +ENGINE = MergeTree +PARTITION BY toYYYYMM(ts) +ORDER BY id +SETTINGS storage_policy = 's3_cache'; +``` + +## Local-only policy + +[Connect to your deployment](/data-warehouse/how-to/connect-applications/), then run the SQL query below to create a new table with the `default` (Block Storage only) policy enabled. + +```sql +CREATE TABLE local_table +( + id UInt64, + ts DateTime, + value Float32 +) +ENGINE = MergeTree +PARTITION BY toYYYYMM(ts) +ORDER BY id +SETTINGS storage_policy = 'default'; +``` + diff --git a/pages/data-warehouse/how-to/manage-users.mdx b/pages/data-warehouse/how-to/manage-users.mdx index 90586137cc..2cc7e64799 100644 --- a/pages/data-warehouse/how-to/manage-users.mdx +++ b/pages/data-warehouse/how-to/manage-users.mdx @@ -9,24 +9,63 @@ dates: import Requirements from '@macros/iam/requirements.mdx' -This page explains how to manage users in your Data Warehouse for ClickHouse® deployment using the [Scaleway console](https://console.scaleway.com/). - - -During the private beta phase, user management can only be done with the **scwadmin** user via the ClickHouse® CLI, MySQL or HTTPS protocols. - +This page explains how to manage users in your Data Warehouse for ClickHouse® deployment. - A Scaleway account logged into the [console](https://console.scaleway.com) - [Owner](/iam/concepts/#owner) status or [IAM permissions](/iam/concepts/#permission) allowing you to perform actions in the intended Organization -- [Signed up to the private beta](https://www.scaleway.com/fr/betas/) and received a confirmation email. - Created a [Data Warehouse deployment](/data-warehouse/how-to/create-deployment/) ## How to manage users using the Scaleway console -During the private beta phase, the management of users for your Data Warehouse for ClickHouse® deployment is limited to: -- Viewing users in your deployment -- Changing the password of an existing user +The Scaleway console allows you to create users for your Data Warehouse for ClickHouse® deployment, to grant them admin rights, to update their password, and to delete them. + +### How to create a user using the Scaleway console + +1. Click **ClickHouse®** under **Data & Analytics** on the side menu. The Data Warehouse deployment page displays. + +2. Click the name of the desired Data Warehouse deployment. The **Overview** tab of the deployment displays. + +3. Click the **Users** tab. A list of your current users displays. + +4. Click **+ Create user**. A pop-up displays. + +5. Enter a username. It must be unique, contain up to 63 characters, and must start with a letter. Only alphanumeric characters, underscores, and dashes are accepted. + +6. Enter a password, or generate one automatically. + +7. Enable the toggle if you want to grant admin rights to the user. + +8. Click **Create user** to proceed. + +The newly created user appears in the list of your deployment's users. + +### How to update a user's password and admin rights + +1. Click **ClickHouse®** under **Data & Analytics** on the side menu. The Data Warehouse deployment page displays. + +2. Click the name of the desired Data Warehouse deployment. The **Overview** tab of the deployment displays. + +3. Click the **Users** tab. A list of your current users displays. + +4. Click next to the name of the user you want to update, then click **Update user**. A pop-up displays. + +5. Update the user's password and/or admin rights, then click **Update user** to confirm. + +### How to delete a user using the Scaleway console + +1. Click **ClickHouse®** under **Data & Analytics** on the side menu. The Data Warehouse deployment page displays. + +2. Click the name of the desired Data Warehouse deployment. The **Overview** tab of the deployment displays. + +3. Click the **Users** tab. A list of your current users displays. + +4. Click next to the name of the user you want to delete. A confirmation pop-up displays. + +5. Enter `DELETE`, then click **Delete user** to confirm.`` + +The user is deleted and no longer appears in the list. ## How to manage users using frameworks @@ -37,4 +76,52 @@ During the private beta phase, the management of users for your Data Warehouse f Once connected, you can run SQL queries using `USER`, `ROLE`, and `ROW POLICY` to create new users and grant them the desired permissions via roles. -Refer to the official ClickHouse® documentation on [users and roles](https://clickhouse.com/docs/sql-reference/statements/create/user) for more information. \ No newline at end of file +Refer to the official ClickHouse® documentation on [users and roles](https://clickhouse.com/docs/sql-reference/statements/create/user) for more information. + +## Manage user permissions + +[Connect to your deployment](/data-warehouse/how-to/connect-applications/) using your preferred framework. + +Once connected, you can run SQL as shown below to grant the desired permissions to users. + +**Grant permission to create databases** + +```sql +GRANT CREATE DATABASE ON . TO user; +``` + +**Grant permission to create tables in any database** + +```sql +GRANT CREATE TABLE ON . TO user; +``` + +**Grant permission to create tables in a specific database** + +```sql +GRANT CREATE TABLE ON database_name.* TO user; +``` + +**Grant data manipulation permissions for any table in any database** + +```sql +GRANT INSERT, UPDATE, DELETE ON . TO user; +``` + +**Grant data manipulation permissions for any table in a specific database** + +```sql +GRANT INSERT, UPDATE, DELETE ON database_name.* TO user; +``` + +**Grant data manipulation permissions for a specific table in a specific database** + +```sql +GRANT INSERT, UPDATE, DELETE ON database_name.table_name TO user; +``` + +**Grant every permissions on every database** + +```sql +GRANT CREATE DATABASE, CREATE TABLE, INSERT, UPDATE, DELETE ON . TO user; +``` \ No newline at end of file diff --git a/pages/data-warehouse/menu.ts b/pages/data-warehouse/menu.ts index bb830f4cab..16cffa3c4b 100644 --- a/pages/data-warehouse/menu.ts +++ b/pages/data-warehouse/menu.ts @@ -30,6 +30,14 @@ export const dataWarehouseMenu = { label: 'Connect to a deployment', slug: 'connect-applications', }, + { + label: 'Manage databases in a deployment', + slug: 'manage-databases', + }, + { + label: 'Manage users in a deployment', + slug: 'manage-users', + }, { label: 'Connect a deployment with BI tools', slug: 'connect-bi-tools', @@ -43,12 +51,8 @@ export const dataWarehouseMenu = { slug: 'edit-autoscaling', }, { - label: 'Manage databases in a deployment', - slug: 'manage-databases', - }, - { - label: 'Manage users in a deployment', - slug: 'manage-users', + label: 'Manage the storage policy of a deployment', + slug: 'manage-storage', }, ], label: 'How to', diff --git a/pages/data-warehouse/quickstart.mdx b/pages/data-warehouse/quickstart.mdx index 9bcc2c4960..9a0123fb6f 100644 --- a/pages/data-warehouse/quickstart.mdx +++ b/pages/data-warehouse/quickstart.mdx @@ -15,7 +15,6 @@ Data Warehouse for ClickHouse® is a centralized repository to store large sets - A Scaleway account logged into the [console](https://console.scaleway.com) - [Owner](/iam/concepts/#owner) status or [IAM permissions](/iam/concepts/#permission) allowing you to perform actions in the intended Organization -- [Signed up to the private beta](https://www.scaleway.com/fr/betas/) and received a confirmation email. ## How to create a Data Warehouse for ClickHouse® deployment