diff --git a/architecture/powersync-service.mdx b/architecture/powersync-service.mdx index 53001ec1..bea3e7fd 100644 --- a/architecture/powersync-service.mdx +++ b/architecture/powersync-service.mdx @@ -9,12 +9,12 @@ This service has the following components: The service continuously replicates data from the source database, then: -1. Pre-processes the data according to the [sync rules](/usage/sync-rules) (both data queries and parameter queries), splitting data into [sync buckets](/usage/sync-rules/organize-data-into-buckets) and transforming the data if required. +1. Pre-processes the data according to the [Sync Streams](/usage/sync-streams), splitting data into [sync buckets](/usage/sync-rules/organize-data-into-buckets) and transforming the data if required. 2. Persists each operation into the relevant sync buckets, ready to be streamed to clients. The recent history of operations to each row is stored, not only the current version. This supports the "append-only" structure of sync buckets, which allows clients to efficiently stream changes while maintaining data integrity. Sync buckets can be compacted to avoid an ever-growing history. -Replication is initially performed by taking a snapshot of all tables defined in the sync rules, then data is incrementally replicated using [logical replication](https://www.postgresql.org/docs/current/logical-replication.html). When sync rules are updated, this process restarts with a new snapshot. +Replication is initially performed by taking a snapshot of all tables defined in the Sync Streams, then data is incrementally replicated using [logical replication](https://www.postgresql.org/docs/current/logical-replication.html). When Sync Streams are updated, this process restarts with a new snapshot. ## Authentication @@ -24,7 +24,7 @@ The service authenticates users using [JWTs](/installation/authentication-setup) Once a user is authenticated: -1. The service calculates a list of buckets for the user to sync using [parameter queries](/usage/sync-rules/parameter-queries). +1. The service calculates a list of buckets for the user to sync. 2. The service streams any operations added to those buckets since the last time the user connected. The service then continuously monitors for buckets that are added or removed, as well as for new operations within those buckets, and streams those changes. diff --git a/installation/client-side-setup/define-your-schema.mdx b/installation/client-side-setup/define-your-schema.mdx index e5c763af..8a02caf0 100644 --- a/installation/client-side-setup/define-your-schema.mdx +++ b/installation/client-side-setup/define-your-schema.mdx @@ -4,14 +4,14 @@ title: "Define your Schema" The PowerSync Client SDKs expose a managed SQLite database that your app can read from and write to. The client-side schema refers to the schema for that SQLite database. -The client-side schema is typically mainly derived from your backend database schema and [Sync Rules](/usage/sync-rules), but can also include other tables such as local-only tables. +The client-side schema is typically mainly derived from your backend database schema and [Sync Streams](/usage/sync-streams), but can also include other tables such as local-only tables. Note that schema migrations are not required on the SQLite database due to the schemaless nature of the [PowerSync protocol](/architecture/powersync-protocol): schemaless data is synced to the client-side SQLite database, and the client-side schema is then applied to that data using _SQLite views_ to allow for structured querying of the data. **Generate schema automatically (PowerSync Cloud)** -In the [PowerSync Dashboard](/usage/tools/powersync-dashboard), the schema can be generated based off your [Sync Rules](/usage/sync-rules) by right-clicking on an instance and selecting **Generate client-side schema**. +In the [PowerSync Dashboard](/usage/tools/powersync-dashboard), the schema can be generated based off your [Sync Streams](/usage/sync-streams) by right-clicking on an instance and selecting **Generate client-side schema**. Similar functionality exists in the PowerSync [CLI](/usage/tools/cli). diff --git a/installation/quickstart-guide.mdx b/installation/quickstart-guide.mdx index e98107da..0bf6bf3a 100644 --- a/installation/quickstart-guide.mdx +++ b/installation/quickstart-guide.mdx @@ -36,9 +36,9 @@ The following outlines our recommended steps to implement PowerSync in your proj 1. Using PowerSync Cloud: See [Database Connection](/installation/database-connection) 2. Using self-hosted PowerSync: Refer to [this section](/self-hosting/installation/powersync-service-setup#powersync-configuration). - - Define [Sync Rules](/usage/sync-rules) in PowerSync — this enables dynamic partial replication: syncing just a relevant subset of data to each user/client instead of your entire database. - - Learn about Sync Rules in our introductory [blog post](https://www.powersync.com/blog/sync-rules-from-first-principles-partial-replication-to-sqlite). + + Define [Sync Streams](/usage/sync-streams) in PowerSync — this enables dynamic partial replication: syncing just a relevant subset of data to each user/client instead of your entire database. + - Learn about Sync Streams in our introductory [blog post](https://www.powersync.com/blog/sync-rules-from-first-principles-partial-replication-to-sqlite). - We recommend starting with one or two simple [Global Data](/usage/sync-rules/example-global-data) queries. diff --git a/integration-guides/flutterflow-+-powersync.mdx b/integration-guides/flutterflow-+-powersync.mdx index 5eb700a1..d0659a01 100644 --- a/integration-guides/flutterflow-+-powersync.mdx +++ b/integration-guides/flutterflow-+-powersync.mdx @@ -50,7 +50,7 @@ This guide walks you through building a basic item management app from scratch a 9. (New) Display Connectivity and Sync Status 10. Secure Your App 1. Enable RLS in Supabase - 2. Update Sync Rules in PowerSync + 2. Update Sync Streams in PowerSync ## Configure Supabase @@ -82,11 +82,11 @@ This guide walks you through building a basic item management app from scratch a -### Configure Sync Rules +### Configure Sync Streams -[Sync Rules](/usage/sync-rules) allow developers to control which data gets synced to which user devices using a SQL-like syntax in a YAML file. For the demo app, we're going to specify that each user can only see their own lists. +[Sync Streams](/usage/sync-streams) allow developers to control which data gets synced to which user devices using a SQL-like syntax in a YAML file. For the demo app, we're going to specify that each user can only see their own lists. -1. To update your Sync Rules, open the `sync-rules.yaml` file. +1. To update your Sync Streams, open the `sync-rules.yaml` file. @@ -99,13 +99,13 @@ This guide walks you through building a basic item management app from scratch a data: - SELECT * FROM lists ``` -3. In the top right, click **"Validate sync rules"** and ensure there are no errors. This validates your sync rules against your Postgres database. +3. In the top right, click **"Validate sync rules"** and ensure there are no errors. This validates your Sync Streams against your Postgres database. 4. In the top right, click **"Deploy sync rules"** and select your instance. 5. Confirm in the dialog and wait a couple of minutes for the deployment to complete. - - For additional information on PowerSync's Sync Rules, refer to the [Sync Rules](/usage/sync-rules) documentation. - - If you're wondering how Sync Rules relate to Supabase Postgres [RLS](https://supabase.com/docs/guides/auth/row-level-security), see [this subsection](/integration-guides/supabase-+-powersync/rls-and-sync-rules). + - For additional information on PowerSync's Sync Streams, refer to the [Sync Streams](/usage/sync-streams) documentation. + - If you're wondering how Sync Streams relate to Supabase Postgres [RLS](https://supabase.com/docs/guides/auth/row-level-security), see [this subsection](/integration-guides/supabase-+-powersync/rls-and-sync-rules). ## Initialize Your FlutterFlow Project @@ -563,11 +563,11 @@ The PowerSync library provides a built-in component that displays real-time conn ## Secure Your App -PowerSync's [Sync Rules](/usage/sync-rules) and Supabase's support for [Row Level Security (RLS)](https://supabase.com/docs/guides/auth/row-level-security) can be used in conjunction. Here are some high level similarities and differences: +PowerSync's [Sync Streams](/usage/sync-streams) and Supabase's support for [Row Level Security (RLS)](https://supabase.com/docs/guides/auth/row-level-security) can be used in conjunction. Here are some high level similarities and differences: * RLS should be used as the authoritative set of security rules applied to your users' CRUD operations that reach Postgres. -* Sync Rules are only applied for data that is to be downloaded to clients — they do not apply to uploaded data. - * Sync Rules can typically be considered to be complementary to RLS, and will generally mirror your RLS setup. +* Sync Streams are only applied for data that is to be downloaded to clients — they do not apply to uploaded data. + * Sync Streams can typically be considered to be complementary to RLS, and will generally mirror your RLS setup. ### Enable RLS in Supabase @@ -582,7 +582,7 @@ create policy "owned lists" on public.lists for ALL using ( ) ``` -### Update Sync Rules +### Update Sync Streams Currently all lists are synced to all users, regardless of who the owner of the list is. You will now update this so that only a user's lists are synced to their device: diff --git a/integration-guides/supabase-+-powersync.mdx b/integration-guides/supabase-+-powersync.mdx index 9465bed3..a9e4ac63 100644 --- a/integration-guides/supabase-+-powersync.mdx +++ b/integration-guides/supabase-+-powersync.mdx @@ -37,7 +37,7 @@ Upon successful integration of Supabase + PowerSync, your system architecture wi -The local SQLite database embedded in the PowerSync SDK is automatically kept in sync with the Supabase Postgres database (based on configured sync rules as you will see later in this guide). Client-side data modifications are persisted in the local SQLite database as well as stored in an upload queue that gets processed via the Supabase client library when network connectivity is available. Therefore reads and writes can happen in the app regardless of whether the user is online or offline, by using the local SQLite database. +The local SQLite database embedded in the PowerSync SDK is automatically kept in sync with the Supabase Postgres database (based on configured Sync Streams as you will see later in this guide). Client-side data modifications are persisted in the local SQLite database as well as stored in an upload queue that gets processed via the Supabase client library when network connectivity is available. Therefore reads and writes can happen in the app regardless of whether the user is online or offline, by using the local SQLite database. For more details on PowerSync's general architecture, [see here](/architecture/architecture-overview). @@ -54,7 +54,7 @@ We will follow these steps to get an offline-first 'To-Do List' demo app up and * Create connection to Supabase - * Configure Sync Rules + * Configure Sync Streams Test the configuration using our provided PowerSync-Supabase 'To-Do List' demo app with your framework of choice. @@ -122,34 +122,55 @@ Run the below SQL statement in your **Supabase SQL Editor** to create a Postgres ### Connect PowerSync to Your Supabase -### Configure Sync Rules - -[Sync Rules](/usage/sync-rules) allow developers to control which data gets synced to which user devices using a SQL-like syntax in a YAML file. For the demo app, we're going to specify that each user can only see their own to-do lists and list items. - -1. The final step is to replace the Sync Rules file's contents with the below: - -```yaml -bucket_definitions: - user_lists: - # Separate bucket per To-Do list - parameters: select id as list_id from lists where owner_id = request.user_id() - data: - - select * from lists where id = bucket.list_id - - select * from todos where list_id = bucket.list_id -``` +### Configure Sync Streams + +[Sync Streams](/usage/sync-streams) allow developers to control which data gets synced to which user devices using a SQL-like syntax in a YAML file. For the demo app, we're going to specify that each user can only see their own to-do lists and list items. + +1. The final step is to replace the Sync Streams file's contents with one of the below options: + + + + ```yaml + config: + edition: 2 + streams: + lists: + query: SELECT * FROM lists + auto_subscribe: true + todos: + query: SELECT * FROM todos WHERE list_id = subscription.parameter('list') + ``` + + + **Note:** The demo apps currently use Sync Rules (legacy). To use Sync Streams with the demo apps, you'll need to enable the Rust client implementation and update the client code to subscribe to streams. See the [Sync Streams documentation](/usage/sync-streams) for details. + + + + + ```yaml + bucket_definitions: + user_lists: + # Separate bucket per To-Do list + parameters: select id as list_id from lists where owner_id = request.user_id() + data: + - select * from lists where id = bucket.list_id + - select * from todos where list_id = bucket.list_id + ``` + + -2. Click **"Validate sync rules"** and ensure there are no errors. This validates your sync rules against your Postgres database. -3. Click **"Save and deploy"** to deploy your Sync Rules. +2. Click **"Validate sync rules"** and ensure there are no errors. This validates your Sync Streams against your Postgres database. +3. Click **"Save and deploy"** to deploy your Sync Streams. -- Your Sync Rules can be updated by navigating to the **Manage instances** workspace and selecting the `sync-rules.yaml` file. +- Your Sync Streams can be updated by navigating to the **Manage instances** workspace and selecting the `sync-rules.yaml` file. -- For additional information on PowerSync's Sync Rules, refer to the [Sync Rules](/usage/sync-rules) documentation. -- If you're wondering how Sync Rules relate to Supabase Postgres [RLS](https://supabase.com/docs/guides/auth/row-level-security), see [this subsection](/integration-guides/supabase-+-powersync/rls-and-sync-rules). +- For additional information, refer to the [Sync Streams](/usage/sync-streams) documentation. +- If you're wondering how Sync Streams relate to Supabase Postgres [RLS](https://supabase.com/docs/guides/auth/row-level-security), see [this subsection](/integration-guides/supabase-+-powersync/rls-and-sync-rules). ## Test Everything (Using Our Demo App) diff --git a/integration-guides/supabase-+-powersync/rls-and-sync-rules.mdx b/integration-guides/supabase-+-powersync/rls-and-sync-rules.mdx index 4a5cab87..d2687353 100644 --- a/integration-guides/supabase-+-powersync/rls-and-sync-rules.mdx +++ b/integration-guides/supabase-+-powersync/rls-and-sync-rules.mdx @@ -1,12 +1,12 @@ --- -title: "RLS and Sync Rules" +title: "RLS and Sync Streams" --- -PowerSync's [Sync Rules](/usage/sync-rules) and Supabase's support for [Row Level Security (RLS)](https://supabase.com/docs/guides/auth/row-level-security) can be used in conjunction. Here are some high level similarities and differences: +PowerSync's [Sync Streams](/usage/sync-streams) and Supabase's support for [Row Level Security (RLS)](https://supabase.com/docs/guides/auth/row-level-security) can be used in conjunction. Here are some high level similarities and differences: * RLS should be used as the authoritative set of security rules applied to your users' CRUD operations that reach Postgres. -* Sync Rules are only applied for data that is to be downloaded to clients — they do not apply to uploaded data. - * Sync Rules can typically be considered to be complementary to RLS, and will generally mirror your RLS setup. +* Sync Streams are only applied for data that is to be downloaded to clients — they do not apply to uploaded data. + * Sync Streams can typically be considered to be complementary to RLS, and will generally mirror your RLS setup. Supabase tables are often created with auto-increment IDs. For easiest use of PowerSync, make sure to convert them to text IDs as detailed [**here**](/usage/sync-rules/client-id)**.** @@ -35,8 +35,10 @@ create policy "todos in owned lists" on public.todos for ALL using ( ``` -`auth.uid()` in a Supabase RLS policy is the same as `request.user_id()` (previously `token_parameters.user_id`) in [Sync Rules](/usage/sync-rules). +`auth.uid()` in a Supabase RLS policy is the same as: +- `auth.user_id()` in [Sync Streams](/usage/sync-streams) +- `request.user_id()` (previously `token_parameters.user_id`) in [Sync Rules (legacy)](/usage/sync-rules) -If you compare these to your Sync Rules configuration in `sync-rules.yaml`, you'll see they are quite similar. +If you compare these to your Sync Streams configuration in `sync-rules.yaml`, you'll see they are quite similar. If you have any questions, join us on [our community Discord](https://discord.gg/powersync) where our team is always available to help. diff --git a/intro/powersync-overview.mdx b/intro/powersync-overview.mdx index b8af79eb..3a58c882 100644 --- a/intro/powersync-overview.mdx +++ b/intro/powersync-overview.mdx @@ -59,8 +59,8 @@ Learn how to install PowerSync in your project. Learn how to fully implement PowerSync in your project. - - Sync rules control which data gets synchronized to users' devices - learn everything you need to know about sync rules. + + Sync Streams control which data gets synchronized to users' devices - learn everything you need to know about sync streams. diff --git a/self-hosting/installation.mdx b/self-hosting/installation.mdx index 465267dd..7787f55d 100644 --- a/self-hosting/installation.mdx +++ b/self-hosting/installation.mdx @@ -36,7 +36,7 @@ In order to run the PowerSync Service, the following activities are required: - + diff --git a/usage/lifecycle-maintenance/deploying-schema-changes.mdx b/usage/lifecycle-maintenance/deploying-schema-changes.mdx index 182e9324..f63204f8 100644 --- a/usage/lifecycle-maintenance/deploying-schema-changes.mdx +++ b/usage/lifecycle-maintenance/deploying-schema-changes.mdx @@ -4,6 +4,6 @@ mode: wide sidebarTitle: Overview --- -The deploy process for schema or [Sync Rule](../sync-rules) updates depends on the type of change. +The deploy process for schema or [Sync Streams](../sync-streams) updates depends on the type of change. See the appropriate subsections below for details on the various scenarios. diff --git a/usage/sync-rules.mdx b/usage/sync-rules.mdx index 9f32d5ba..696de8bb 100644 --- a/usage/sync-rules.mdx +++ b/usage/sync-rules.mdx @@ -5,6 +5,10 @@ sidebarTitle: Overview PowerSync Sync Rules allow developers to control which data gets synchronized to which devices (i.e. they enable _dynamic partial replication_). + + **Sync Streams (Early Alpha)** are now available as the next evolution of Sync Rules, offering on-demand syncing and improved developer experience. [Learn more about Sync Streams](/usage/sync-streams.md). Sync Rules will continue to be supported for the foreseeable future. + + ## Introduction We recommend starting with our [Sync Rules from First Principles](https://www.powersync.com/blog/sync-rules-from-first-principles-partial-replication-to-sqlite) blog post, which explains on a high-level what Sync Rules are, why they exist and how to implement them. diff --git a/usage/sync-rules/data-queries.mdx b/usage/sync-rules/data-queries.mdx index 396084a5..74c475a7 100644 --- a/usage/sync-rules/data-queries.mdx +++ b/usage/sync-rules/data-queries.mdx @@ -2,28 +2,58 @@ title: "Data Queries" --- -Data queries select the data that form part of a bucket, using the bucket parameters. - -Multiple data queries can be specified for a single bucket definition. - - - **Data queries are used to group data into buckets, so each data query must use every bucket parameter.** - + + + Sync Streams use **queries** (and optionally **subqueries**) to select data. Unlike Sync Rules, Sync Streams don't have separate "data queries" or "parameter queries" — they use a unified query syntax: + + ```yaml + streams: + stream_name: + query: SELECT * FROM table WHERE condition + ``` + + See the [Sync Streams documentation](/usage/sync-streams.md) for full syntax details. + + + + Data queries select the data that form part of a bucket, using the bucket parameters. + + Multiple data queries can be specified for a single bucket definition. + + + **Data queries are used to group data into buckets, so each data query must use every bucket parameter.** + + + ## Examples #### Grouping by list\_id -```yaml -bucket_definitions: - owned_lists: - parameters: | - SELECT id as list_id FROM lists WHERE - owner_id = request.user_id() - data: - - SELECT * FROM lists WHERE lists.id = bucket.list_id - - SELECT * FROM todos WHERE todos.list_id = bucket.list_id -``` + + + ```yaml + streams: + owned_list: + query: SELECT * FROM lists WHERE id = subscription.parameter('list_id') AND id IN (SELECT id FROM lists WHERE owner_id = auth.user_id()) + owned_list_todos: + query: SELECT * FROM todos WHERE list_id = subscription.parameter('list_id') AND list_id IN (SELECT id FROM lists WHERE owner_id = auth.user_id()) + ``` + + + + ```yaml + bucket_definitions: + owned_lists: + parameters: | + SELECT id as list_id FROM lists WHERE + owner_id = request.user_id() + data: + - SELECT * FROM lists WHERE lists.id = bucket.list_id + - SELECT * FROM todos WHERE todos.list_id = bucket.list_id + ``` + + #### Selecting output columns/fields @@ -33,44 +63,91 @@ This is good practice, to ensure the synced data does not unintentionally change Note: An `id` column must always be present, and must have a `text` type. If the primary key is different, use a column alias and/or transformations to output a `text` id column. -```yaml -bucket_definitions: - global: - data: - - SELECT id, name, owner_id FROM lists -``` + + + ```yaml + streams: + lists: + query: SELECT id, name, owner_id FROM lists + auto_subscribe: true + ``` + + + + ```yaml + bucket_definitions: + global: + data: + - SELECT id, name, owner_id FROM lists + ``` + + - MongoDB uses `_id` as the name of the ID field in collections. Therefore, PowerSync requires using `SELECT _id as id` in the data queries when [using MongoDB](/installation/database-setup) as the backend source database. + MongoDB uses `_id` as the name of the ID field in collections. Therefore, PowerSync requires using `SELECT _id as id` in the queries when [using MongoDB](/installation/database-setup) as the backend source database. #### Renaming columns/fields Different names (aliases) may be specified for columns/fields: -```yaml -bucket_definitions: - global: - data: - - SELECT id, name, created_timestamp AS created_at FROM lists -``` + + + ```yaml + streams: + lists: + query: SELECT id, name, created_timestamp AS created_at FROM lists + auto_subscribe: true + ``` + + + + ```yaml + bucket_definitions: + global: + data: + - SELECT id, name, created_timestamp AS created_at FROM lists + ``` + + #### Transforming columns/fields A limited set of operators and functions are available to transform the output value of columns/fields. -```yaml -bucket_definitions: - global: - data: - # Cast number to text - - SELECT id, item_number :: text AS item_number FROM todos - # Alternative syntax for the same cast - - SELECT id, CAST(item_number as TEXT) AS item_number FROM todos - # Convert binary data (bytea) to base64 - - SELECT id, base64(thumbnail) AS thumbnail_base64 FROM todos - # Extract field from JSON or JSONB column - - SELECT id, metadata_json ->> 'description' AS description FROM todos - # Convert time to epoch number - - SELECT id, unixepoch(created_at) AS created_at FROM todos -``` + + + ```yaml + streams: + todos: + # Cast number to text + query: SELECT id, item_number :: text AS item_number FROM todos + # Alternative syntax for the same cast + query: SELECT id, CAST(item_number as TEXT) AS item_number FROM todos + # Convert binary data (bytea) to base64 + query: SELECT id, base64(thumbnail) AS thumbnail_base64 FROM todos + # Extract field from JSON or JSONB column + query: SELECT id, metadata_json ->> 'description' AS description FROM todos + # Convert time to epoch number + query: SELECT id, unixepoch(created_at) AS created_at FROM todos + ``` + + + + ```yaml + bucket_definitions: + global: + data: + # Cast number to text + - SELECT id, item_number :: text AS item_number FROM todos + # Alternative syntax for the same cast + - SELECT id, CAST(item_number as TEXT) AS item_number FROM todos + # Convert binary data (bytea) to base64 + - SELECT id, base64(thumbnail) AS thumbnail_base64 FROM todos + # Extract field from JSON or JSONB column + - SELECT id, metadata_json ->> 'description' AS description FROM todos + # Convert time to epoch number + - SELECT id, unixepoch(created_at) AS created_at FROM todos + ``` + + diff --git a/usage/sync-rules/example-global-data.mdx b/usage/sync-rules/example-global-data.mdx index c4afcc78..a0c837cb 100644 --- a/usage/sync-rules/example-global-data.mdx +++ b/usage/sync-rules/example-global-data.mdx @@ -1,20 +1,42 @@ --- title: "Global Data" -description: 'The simplest Sync Rules are for "global" data — synced to all users. ' +description: 'The simplest approach is for "global" data — synced to all users. ' --- -For example, the following Sync Rules sync all `todos` and only unarchived `lists` to all users: +For example, the following configuration syncs all `todos` and only unarchived `lists` to all users: -```yaml -bucket_definitions: - global_bucket: - data: - # Sync all todos - - SELECT * FROM todos - # Sync all lists except archived ones - - SELECT * FROM lists WHERE archived = false -``` + + + ```yaml + config: + edition: 2 + streams: + all_todos: + query: SELECT * FROM todos + auto_subscribe: true + unarchived_lists: + query: SELECT * FROM lists WHERE archived = false + auto_subscribe: true + ``` + + + With Sync Streams, "global" data is defined as streams with `auto_subscribe: true`, so users subscribe to them by default. + + + + + ```yaml + bucket_definitions: + global_bucket: + data: + # Sync all todos + - SELECT * FROM todos + # Sync all lists except archived ones + - SELECT * FROM lists WHERE archived = false + ``` + + - **Note**: Table names within Sync Rules must match the names defined in the [client-side schema](/installation/client-side-setup/define-your-schema). + **Note**: Table names must match the names defined in the [client-side schema](/installation/client-side-setup/define-your-schema). diff --git a/usage/sync-rules/glossary.mdx b/usage/sync-rules/glossary.mdx index b51e9a6b..7da27319 100644 --- a/usage/sync-rules/glossary.mdx +++ b/usage/sync-rules/glossary.mdx @@ -11,10 +11,14 @@ Each bucket is defined by its bucket definition name and set of parameter values ### Bucket Definition -This is the "[Sync Rule](/usage/sync-rules)" that describes buckets. Specifies the name, parameter query(ies), and data queries. +This is the "[Sync Rule](/usage/sync-rules)" that describes buckets. In Sync Rules (legacy), this specifies the name, parameter query(ies), and data queries. Each bucket definition describes a set of buckets using SQL-like queries. + + [Sync Streams](/usage/sync-streams.md) use a different approach with unified queries instead of separate parameter and data queries. + + ### Bucket Parameters This is the set of parameters that uniquely identifies an individual bucket within a bucket definition. Together with the bucket name, this forms the bucket ID. diff --git a/usage/sync-rules/organize-data-into-buckets.mdx b/usage/sync-rules/organize-data-into-buckets.mdx index 2fb8f00f..19fb7636 100644 --- a/usage/sync-rules/organize-data-into-buckets.mdx +++ b/usage/sync-rules/organize-data-into-buckets.mdx @@ -6,29 +6,48 @@ To sync different sets of data to each user, data is organized into buckets. Each user can sync a number of buckets (up to 1,000), and each bucket defines a set of tables/collections and rows/documents to sync. -This is defined using two queries: - -1. Select bucket parameters from a user ID and/or other parameters ([parameter queries](/usage/sync-rules/parameter-queries)) - -2. Select data in the bucket using the bucket parameters ([data queries](/usage/sync-rules/data-queries)) - -When designing your buckets, it is recommended, but not required, to group all data in a bucket where the same parameters apply. - -An example: - -```yaml -bucket_definitions: - user_lists: - # Optionally define the priority of the bucket to sync certain buckets before others - priority: 1 # See https://docs.powersync.com/usage/use-case-examples/prioritized-sync - # Select parameters for the bucket, using the current user_id - parameters: SELECT request.user_id() as user_id # (request.user_id() comes from the JWT token) - data: - # Select data rows/documents using the parameters above - - SELECT * FROM lists WHERE owner_id = bucket.user_id -``` + + + With Sync Streams, you define streams using queries (and optionally subqueries) that clients can subscribe to with parameters: + + ```yaml + config: + edition: 2 + streams: + user_lists: + priority: 1 # See https://docs.powersync.com/usage/use-case-examples/prioritized-sync + query: SELECT * FROM lists WHERE owner_id = auth.user_id() + ``` + + Sync Streams use a unified query syntax — there are no separate "data queries" or "parameter queries". Clients subscribe to streams with parameters as needed. See the [Sync Streams documentation](/usage/sync-streams.md) for client-side syntax. + + + + With Sync Rules, this is defined using two types of queries: + + 1. Select bucket parameters from a user ID and/or other parameters ([parameter queries](/usage/sync-rules/parameter-queries)) + + 2. Select data in the bucket using the bucket parameters ([data queries](/usage/sync-rules/data-queries)) + + When designing your buckets, it is recommended, but not required, to group all data in a bucket where the same parameters apply. + + An example: + + ```yaml + bucket_definitions: + user_lists: + # Optionally define the priority of the bucket to sync certain buckets before others + priority: 1 # See https://docs.powersync.com/usage/use-case-examples/prioritized-sync + # Select parameters for the bucket, using the current user_id + parameters: SELECT request.user_id() as user_id # (request.user_id() comes from the JWT token) + data: + # Select data rows/documents using the parameters above + - SELECT * FROM lists WHERE owner_id = bucket.user_id + ``` + + **Note**: - - Table names within Sync Rules must match the names defined in the [client-side schema](/installation/client-side-setup/define-your-schema). + - Table names must match the names defined in the [client-side schema](/installation/client-side-setup/define-your-schema). \ No newline at end of file diff --git a/usage/sync-rules/parameter-queries.mdx b/usage/sync-rules/parameter-queries.mdx index 303e754c..d98dad67 100644 --- a/usage/sync-rules/parameter-queries.mdx +++ b/usage/sync-rules/parameter-queries.mdx @@ -2,48 +2,105 @@ title: "Parameter Queries" --- -Parameter queries allow parameters to be defined on a bucket to group data. These queries can use parameters from the JWT (we loosely refer to these as token parameters), such as a `user_id`, or [parameters from clients](/usage/sync-rules/advanced-topics/client-parameters) directly. - -```yaml -bucket_definitions: - # Bucket Name - user_lists: - # Parameter Query - parameters: SELECT request.user_id() as user_id - # Data Query - data: - - SELECT * FROM lists WHERE lists.owner_id = bucket.user_id - - user_lists_table: - # Similar query, but using a table - # Access can instantly be revoked by deleting the user row/document - parameters: SELECT id as user_id FROM users WHERE users.id = request.user_id() - data: - - SELECT * FROM lists WHERE lists.user_id = bucket.user_id -``` - -Available functions in sync rules are: - -1. `request.user_id()`: Returns the JWT subject, same as `request.jwt() ->> 'sub'` - -2. `request.jwt()`: Returns the entire (signed) JWT payload as a JSON string. - -3. `request.parameters()`: Returns [client parameters](/usage/sync-rules/advanced-topics/client-parameters) as a JSON string. - -Example usage: - -```sql -request.user_id() -request.jwt() ->> 'sub' -- Same as `request.user_id() -request.parameters() ->> 'param' -- Client parameters - --- Some Supabase-specific examples below. These can be used with standard Supabase tokens, --- for use cases which previously required custom tokens -request.jwt() ->> 'role' -- 'authenticated' or 'anonymous' -request.jwt() ->> 'email' -- automatic email field -request.jwt() ->> 'app_metadata.custom_field' -- custom field added by a service account (authenticated) - -``` + + + Sync Streams don't have "parameter queries" — instead, parameters are accessed directly in queries using `auth.user_id()`, `auth.parameter()`, `subscription.parameter()`, and `connection.parameter()`: + + ```yaml + config: + edition: 2 + streams: + user_lists: + query: SELECT * FROM lists WHERE owner_id = auth.user_id() + + user_lists_with_access_check: + # Access can instantly be revoked by deleting the user row/document + query: SELECT * FROM lists WHERE owner_id IN (SELECT id FROM users WHERE id = auth.user_id()) + ``` + + + Sync Streams use a unified query syntax with optional subqueries. See the [Sync Streams documentation](/usage/sync-streams.md) for details. + + + + + Parameter queries allow parameters to be defined on a bucket to group data. These queries can use parameters from the JWT (we loosely refer to these as token parameters), such as a `user_id`, or [parameters from clients](/usage/sync-rules/advanced-topics/client-parameters) directly. + + + + + ```yaml + bucket_definitions: + # Bucket Name + user_lists: + # Parameter Query + parameters: SELECT request.user_id() as user_id + # Data Query + data: + - SELECT * FROM lists WHERE lists.owner_id = bucket.user_id + + user_lists_table: + # Similar query, but using a table + # Access can instantly be revoked by deleting the user row/document + parameters: SELECT id as user_id FROM users WHERE users.id = request.user_id() + data: + - SELECT * FROM lists WHERE lists.user_id = bucket.user_id + ``` + + + +Available functions: + + + + **Auth Parameters** (from JWT): + - `auth.user_id()`: Returns the JWT subject, same as `auth.parameter('sub')` + - `auth.parameters()`: Returns the entire JWT payload as JSON + - `auth.parameter('key')`: Returns a specific JWT claim + + **Subscription Parameters** (passed from client): + - `subscription.parameters()`: All subscription parameters as JSON + - `subscription.parameter('key')`: A specific subscription parameter + + **Connection Parameters** (global client parameters): + - `connection.parameters()`: All connection parameters as JSON + - `connection.parameter('key')`: A specific connection parameter + + Example usage: + ```sql + auth.user_id() + auth.parameter('sub') -- Same as auth.user_id() + auth.parameter('role') -- 'authenticated' or 'anonymous' (Supabase) + auth.parameter('email') -- email field (Supabase) + auth.parameter('app_metadata.custom_field') -- custom field (Supabase) + subscription.parameter('list_id') -- parameter passed from client + connection.parameter('tenant_id') -- global client parameter + ``` + + + + 1. `request.user_id()`: Returns the JWT subject, same as `request.jwt() ->> 'sub'` + + 2. `request.jwt()`: Returns the entire (signed) JWT payload as a JSON string. + + 3. `request.parameters()`: Returns [client parameters](/usage/sync-rules/advanced-topics/client-parameters) as a JSON string. + + Example usage: + + ```sql + request.user_id() + request.jwt() ->> 'sub' -- Same as `request.user_id() + request.parameters() ->> 'param' -- Client parameters + + -- Some Supabase-specific examples below. These can be used with standard Supabase tokens, + -- for use cases which previously required custom tokens + request.jwt() ->> 'role' -- 'authenticated' or 'anonymous' + request.jwt() ->> 'email' -- automatic email field + request.jwt() ->> 'app_metadata.custom_field' -- custom field added by a service account (authenticated) + + ``` + + A previous syntax for parameter queries used `token_parameters`. Expand the below for details on how to migrate to the recommended syntax above. @@ -96,42 +153,84 @@ request.jwt() ->> 'app_metadata.custom_field' -- custom field added by a service #### Filter on additional columns -```yaml -bucket_definitions: - admin_users: - parameters: | - SELECT id as user_id FROM users WHERE - users.id = request.user_id() AND - users.is_admin = true - - data: - - SELECT * FROM lists WHERE lists.owner_id = bucket.user_id -``` + + + ```yaml + streams: + admin_lists: + query: SELECT * FROM lists WHERE owner_id IN (SELECT id FROM users WHERE id = auth.user_id() AND is_admin = true) + ``` + + + + ```yaml + bucket_definitions: + admin_users: + parameters: | + SELECT id as user_id FROM users WHERE + users.id = request.user_id() AND + users.is_admin = true + + data: + - SELECT * FROM lists WHERE lists.owner_id = bucket.user_id + ``` + + #### Group according to different columns -```yaml -bucket_definitions: - primary_list: - parameters: | - SELECT primary_list_id FROM users WHERE - users.id = request.user_id() - data: - - SELECT * FROM todos WHERE todos.list_id = bucket.primary_list_id -``` + + + ```yaml + streams: + primary_list_todos: + query: SELECT * FROM todos WHERE list_id IN (SELECT primary_list_id FROM users WHERE id = auth.user_id()) + ``` + + + + ```yaml + bucket_definitions: + primary_list: + parameters: | + SELECT primary_list_id FROM users WHERE + users.id = request.user_id() + data: + - SELECT * FROM todos WHERE todos.list_id = bucket.primary_list_id + ``` + + #### Using different tables for parameters -```yaml -bucket_definitions: - owned_lists: - parameters: | - SELECT id as list_id FROM lists WHERE - owner_id = request.user_id() - data: - - SELECT * FROM lists WHERE lists.id = bucket.list_id - - SELECT * FROM todos WHERE todos.list_id = bucket.list_id -``` + + + ```yaml + streams: + owned_lists: + query: SELECT * FROM lists WHERE id = subscription.parameter('list_id') AND id IN (SELECT id FROM lists WHERE owner_id = auth.user_id()) + list_todos: + query: SELECT * FROM todos WHERE list_id = subscription.parameter('list_id') AND list_id IN (SELECT id FROM lists WHERE owner_id = auth.user_id()) + ``` + + + Clients subscribe to these streams with parameters like `{list_id: 'abc123'}`. See [Sync Streams documentation](/usage/sync-streams.md) for client-side syntax. + + + + + ```yaml + bucket_definitions: + owned_lists: + parameters: | + SELECT id as list_id FROM lists WHERE + owner_id = request.user_id() + data: + - SELECT * FROM lists WHERE lists.id = bucket.list_id + - SELECT * FROM todos WHERE todos.list_id = bucket.list_id + ``` + + #### Using a join table @@ -139,35 +238,67 @@ In this example, a single query can return multiple sets of bucket parameters fo Keep in mind that the total number of buckets per user should remain limited (\< 1,000), so don't make buckets too granular. -```yaml -bucket_definitions: - user_lists: - parameters: | - SELECT list_id FROM user_lists WHERE - user_lists.user_id = request.user_id() - data: - - SELECT * FROM lists WHERE lists.id = bucket.list_id - - SELECT * FROM todos WHERE todos.list_id = bucket.list_id -``` + + + ```yaml + streams: + user_list: + query: SELECT * FROM lists WHERE id = subscription.parameter('list_id') AND id IN (SELECT list_id FROM user_lists WHERE user_id = auth.user_id()) + user_list_todos: + query: SELECT * FROM todos WHERE list_id = subscription.parameter('list_id') AND list_id IN (SELECT list_id FROM user_lists WHERE user_id = auth.user_id()) + ``` + + + + ```yaml + bucket_definitions: + user_lists: + parameters: | + SELECT list_id FROM user_lists WHERE + user_lists.user_id = request.user_id() + data: + - SELECT * FROM lists WHERE lists.id = bucket.list_id + - SELECT * FROM todos WHERE todos.list_id = bucket.list_id + ``` + + #### Multiple bucket parameters Parameter queries may return multiple bucket parameters. - - **Note that every bucket parameter must be used in every data query.** - - -```yaml -bucket_definitions: - owned_org_lists: - parameters: | - SELECT id as list_id, org_id FROM lists WHERE - owner_id = request.user_id() - data: - - SELECT * FROM lists WHERE lists.id = bucket.list_id and lists.org_id = bucket.org_id - - SELECT * FROM todos WHERE todos.list_id = bucket.list_id and todos.org_id = bucket.org_id -``` + + + ```yaml + streams: + org_list: + query: SELECT * FROM lists WHERE id = subscription.parameter('list_id') AND org_id = subscription.parameter('org_id') AND id IN (SELECT id FROM lists WHERE owner_id = auth.user_id()) + org_list_todos: + query: SELECT * FROM todos WHERE list_id = subscription.parameter('list_id') AND org_id = subscription.parameter('org_id') AND list_id IN (SELECT id FROM lists WHERE owner_id = auth.user_id()) + ``` + + + Clients subscribe with multiple parameters like `{list_id: 'abc123', org_id: 'org456'}`. + + + + + + **Note that every bucket parameter must be used in every data query.** + + + ```yaml + bucket_definitions: + owned_org_lists: + parameters: | + SELECT id as list_id, org_id FROM lists WHERE + owner_id = request.user_id() + data: + - SELECT * FROM lists WHERE lists.id = bucket.list_id and lists.org_id = bucket.org_id + - SELECT * FROM todos WHERE todos.list_id = bucket.list_id and todos.org_id = bucket.org_id + ``` + + #### Using multiple parameter queries @@ -175,22 +306,47 @@ Multiple parameter queries can be used in the same bucket definition. It is important in this case that the output columns are exactly the same for each query in the bucket definition, as these define the bucket parameters. -```yaml -bucket_definitions: - user_lists: - parameters: - - SELECT id as list_id FROM lists WHERE owner_id = request.user_id() - - SELECT list_id FROM user_lists WHERE user_lists.user_id = request.user_id() - data: - - SELECT * FROM lists WHERE lists.id = bucket.list_id - - SELECT * FROM todos WHERE todos.list_id = bucket.list_id -``` + + + ```yaml + streams: + accessible_list: + # Combines owned lists and shared lists using OR + query: SELECT * FROM lists WHERE id = subscription.parameter('list_id') AND (owner_id = auth.user_id() OR id IN (SELECT list_id FROM user_lists WHERE user_id = auth.user_id())) + accessible_list_todos: + query: SELECT * FROM todos WHERE list_id = subscription.parameter('list_id') AND (list_id IN (SELECT id FROM lists WHERE owner_id = auth.user_id()) OR list_id IN (SELECT list_id FROM user_lists WHERE user_id = auth.user_id())) + ``` + + + + ```yaml + bucket_definitions: + user_lists: + parameters: + - SELECT id as list_id FROM lists WHERE owner_id = request.user_id() + - SELECT list_id FROM user_lists WHERE user_lists.user_id = request.user_id() + data: + - SELECT * FROM lists WHERE lists.id = bucket.list_id + - SELECT * FROM todos WHERE todos.list_id = bucket.list_id + ``` + + Keep in mind that the total number of buckets per user should remain limited (\< 1,000), so don't make buckets too granular. #### Pass parameters from clients -It is possible to pass parameters from clients directly. See [client parameters](/usage/sync-rules/advanced-topics/client-parameters) to learn more. + + + With Sync Streams, clients pass parameters when subscribing to streams using `subscription.parameter()`. See [Sync Streams documentation](/usage/sync-streams.md) for client-side syntax. + + For global client parameters (equivalent to Sync Rules client parameters), use `connection.parameter()`. + + + + It is possible to pass parameters from clients directly. See [client parameters](/usage/sync-rules/advanced-topics/client-parameters) to learn more. + + #### Global buckets @@ -200,17 +356,34 @@ When no parameter query is specified, it is automatically a global bucket. Alternatively, a parameter query with no output columns may be specified to only sync the bucket to a subset of users. -```yaml -bucket_definitions: - global_admins: - parameters: | - SELECT FROM users WHERE - users.id = request.user_id() AND - users.is_admin = true - - data: - - SELECT * FROM admin_settings -``` + + + ```yaml + streams: + admin_settings: + query: SELECT * FROM admin_settings WHERE EXISTS (SELECT FROM users WHERE id = auth.user_id() AND is_admin = true) + auto_subscribe: true + ``` + + + With `auto_subscribe: true`, this stream is automatically synced for users who meet the condition. + + + + + ```yaml + bucket_definitions: + global_admins: + parameters: | + SELECT FROM users WHERE + users.id = request.user_id() AND + users.is_admin = true + + data: + - SELECT * FROM admin_settings + ``` + + ## Restrictions