docs: Add verify and push documentation (#2998)

Add documentation for verify and push

Author: kyleconroy

docs/conf.py

@@ -33,7 +33,8 @@
 extensions = [
     'myst_parser',
     'sphinx_rtd_theme',
-    "sphinx_favicon",
+    'sphinx_favicon',
+    'sphinxext.rediraffe',
 ]
 
 # Add any paths that contain templates here, relative to this directory.
@@ -75,4 +76,8 @@ def setup(app):
 myst_enable_extensions = [
     "attrs_inline",
     "colon_fence",
-]
+]
+
+rediraffe_redirects = {
+    "howto/upload.md": "howto/push.md",
+}

docs/howto/ci-cd.md

@@ -1,7 +1,8 @@
 # Using sqlc in CI/CD
 
 If your project has more than a single developer, we suggest running `sqlc` as
-part of your CI/CD pipeline. The three subcommands you'll want to run are `diff`, `vet` and `upload`
+part of your CI/CD pipeline. The four subcommands you'll want to run are `diff`,
+`vet`, `verify` and `push`
 
 `sqlc diff` ensures that your generated code is up to date. New developers to a
 project may forget to run `sqlc generate` after adding a query or updating a
@@ -26,15 +27,21 @@ helpful in catching anti-patterns before they make it into production. Please
 see the [vet](vet.md) documentation for a complete guide to adding lint rules
 for your project.
 
-`sqlc upload` pushes your database schema and queries to sqlc Cloud. Once
-uploaded, we ensure that future releases of sqlc do not break your code. Learn
-more about uploading projects [here](upload.md)
+`sqlc verify` ensures that schema changes do not break production. Existing
+queries are checked against new schema changes for correctness. Please see the
+[verify](verify.md) documentation for a complete guide.
+
+
+`sqlc push` pushes your database schema, queries and configuration to sqlc
+Cloud. These archives are used by `verify` to catch breaking changes to your
+database schema.  Learn more about uploading projects [here](push.md)
 
 ## General setup
 
 Install `sqlc` using the [suggested instructions](../overview/install).
 
-Create two steps in your pipeline, one for `sqlc diff` and one for `sqlc vet`. Run `sqlc upload` after merge on your `main` branch.
+Create three steps in your pipeline for `sqlc diff`, `sqlc vet`, and `sqlc
+verify`. Run `sqlc push` after merge on your `main` branch.
 
 ## GitHub Actions
 
@@ -123,29 +130,61 @@ jobs:
     - run: sqlc vet
 ```
 
-### upload
+### push
 
 ```{note}
-Project uploads are powered by [sqlc Cloud](https://dashboard.sqlc.dev). Sign up for [free](https://dashboard.sqlc.dev) today.
+Pushing a project is powered by [sqlc Cloud](https://dashboard.sqlc.dev). Sign up for [free](https://dashboard.sqlc.dev) today.
 ```
 
-The following GitHub Workflow configuration runs [sqlc upload](upload.md) on
+The following GitHub Workflow configuration runs [sqlc push](push.md) on
 every push to `main`. Create an auth token via the
 [dashboard](https://dashboard.sqlc.dev).
 
 ```yaml
 name: sqlc
 on: [push]
 jobs:
-  upload:
+  push:
     runs-on: ubuntu-latest
     if: ${{ github.ref == 'refs/heads/main' }}
     steps:
     - uses: actions/checkout@v3
     - uses: sqlc-dev/setup-sqlc@v3
       with:
         sqlc-version: '1.23.0'
-    - run: sqlc upload
+    - run: sqlc push
       env:
         SQLC_AUTH_TOKEN: ${{ secrets.SQLC_AUTH_TOKEN }}
 ```
+
+### verify
+
+```{note}
+Verify database migrations is powered by [sqlc Cloud](https://dashboard.sqlc.dev). Sign up for [free](https://dashboard.sqlc.dev) today.
+```
+
+```yaml
+name: sqlc
+on: [push]
+jobs:
+  verify:
+    runs-on: ubuntu-latest
+    steps:
+    - uses: actions/checkout@v3
+    - uses: sqlc-dev/setup-sqlc@v3
+      with:
+        sqlc-version: '1.23.0'
+    - run: sqlc verify
+      env:
+        SQLC_AUTH_TOKEN: ${{ secrets.SQLC_AUTH_TOKEN }}
+  push:
+    runs-on: ubuntu-latest
+    if: ${{ github.ref == 'refs/heads/main' }}
+    steps:
+    - uses: sqlc-dev/setup-sqlc@v3
+      with:
+        sqlc-version: '1.23.0'
+    - run: sqlc push
+      env:
+        SQLC_AUTH_TOKEN: ${{ secrets.SQLC_AUTH_TOKEN }}
+``````

docs/howto/push.md

@@ -0,0 +1,61 @@
+# `push` - Uploading projects
+
+```{note}
+`push` is powered by [sqlc Cloud](https://dashboard.sqlc.dev). Sign up for [free](https://dashboard.sqlc.dev) today.
+```
+
+*Added in v1.24.0*
+
+We've renamed the `upload` sub-command to `push`. We've also changed the data sent along in a push request. Upload used to include the configuration file, migrations, queries, and all generated code. Push drops the generated code in favor of including the [plugin.GenerateRequest](https://buf.build/sqlc/sqlc/docs/main:plugin#plugin.GenerateRequest), which is the protocol buffer message we pass to codegen plugins.
+
+## Add configuration
+
+After creating a project, add the project ID to your sqlc configuration file.
+
+```yaml
+version: "2"
+cloud:
+  project: "<PROJECT_ID>"
+```
+
+You'll also need to create an auth token and make it available via the
+`SQLC_AUTH_TOKEN` environment variable.
+
+```shell
+export SQLC_AUTH_TOKEN=sqlc_xxxxxxxx
+```
+
+## Dry run
+
+You can see what's included when uploading your project by using using the
+`--dry-run` flag:
+
+```shell
+$ sqlc push --dry-run
+2023/11/21 10:39:51 INFO config file=sqlc.yaml bytes=912
+2023/11/21 10:39:51 INFO codegen_request queryset=app file=codegen_request.pb
+2023/11/21 10:39:51 INFO schema queryset=app file=migrations/00001_initial.sql bytes=3033
+2023/11/21 10:39:51 INFO query queryset=app file=queries/app.sql bytes=1150
+```
+
+The output is the files `sqlc` would have sent without the `--dry-run` flag.
+
+## Push
+
+Once you're ready to push, remove the `--dry-run` flag.
+
+```shell
+sqlc push
+```
+
+### Annotations 
+
+Annotations are added to each push request. By default, we include these environment variables (if they are present).
+
+```
+GITHUB_REPOSITORY
+GITHUB_REF
+GITHUB_REF_NAME
+GITHUB_REF_TYPE
+GITHUB_SHA
+```

docs/howto/upload.md

@@ -0,0 +1,94 @@
+# `verify` - Verifying database schema changes
+
+```{note}
+`verify` is powered by [sqlc Cloud](https://dashboard.sqlc.dev). Sign up for [free](https://dashboard.sqlc.dev) today.
+```
+
+*Added in v1.24.0*
+
+Schema updates and poorly-written queries often bring down production databases. That’s bad.
+
+Out of the box, `sqlc generate` catches some of these issues. Running `sqlc vet` with the `sqlc/db-prepare` rule catches more subtle problems. But there is a large class of issues that sqlc can’t prevent by looking at current schema and queries alone.
+
+For instance, when a schema change is proposed, existing queries and code running in production might fail when the schema change is applied. Enter `sqlc verify`, which analyzes existing queries against new schema changes and errors if there are any issues.
+
+Let's look at an example. Assume you have these two tables in production.
+
+```sql
+CREATE TABLE users (
+  id UUID PRIMARY KEY
+);
+
+CREATE TABLE user_actions (
+  id UUID PRIMARY KEY,
+  user_id UUID NOT NULL,
+  action TEXT,
+  created_at TIMESTAMP
+);
+```
+
+Your application contains the following query to join user actions against the users table.
+
+```sql
+-- name: GetUserActions :many
+SELECT * FROM users u
+JOIN user_actions ua ON u.id = ua.user_id
+ORDER BY created_at;
+```
+
+So far, so good. Then assume you propose this schema change:
+
+```sql
+ALTER TABLE users ADD COLUMN created_at TIMESTAMP;
+```
+
+Running `sqlc generate` fails with this change, returning a `column reference "created_at" is ambiguous` error. You update your query to fix the issue.
+
+```sql
+-- name: GetUserActions :many
+SELECT * FROM users u
+JOIN user_actions ua ON u.id = ua.user_id
+ORDER BY u.created_at;
+```
+
+While that change fixes the issue, there's a production outage waiting to happen. When the schema change is applied, the existing `GetUserActions` query will begin to fail. The correct way to fix this is to deploy the updated query before applying the schema migration.
+
+It ensures migrations are safe to deploy by sending your current schema and queries to sqlc cloud. There, we run the queries for your latest push against your new schema changes. This check catches backwards incompatible schema changes for existing queries.
+
+Here `sqlc verify` alerts you to the fact that ORDER BY "created_at" is ambiguous.
+
+```sh
+$ sqlc verify
+FAIL: app query.sql
+
+=== Failed
+=== FAIL: app query.sql GetUserActions
+    ERROR: column reference "created_at" is ambiguous (SQLSTATE 42702)
+```
+
+By the way, this scenario isn't made up! It happened to us a few weeks ago. We've been happily testing early versions of `verify` for the last two weeks and haven't had any issues since.
+
+This type of verification is only the start. If your application is deployed on-prem by your customers, `verify` could tell you if it's safe for your customers to rollback to an older version of your app, even after schema migrations have been run.
+
+Using `verify` requires that you push your queries and schema when you tag a release of your application. We run it on every push to main, as we continuously deploy those commits.
+
+## Authentication
+
+`sqlc` expects to find a valid auth token in the value of the `SQLC_AUTH_TOKEN`
+environment variable. You can create an auth token via the [dashboard](https://dashboard.sqlc.dev).
+
+```shell
+export SQLC_AUTH_TOKEN=sqlc_xxxxxxxx
+```
+
+
+## Expected workflow
+
+Using `sqlc verify` requires pushing your queries and schema to sqlc Cloud. When
+you release a new version of your application, you should push your schema and
+queries as well. For example, we run `sqlc push` after any change has been
+merged into our `main` branch on Github, as we deploy every commit to
+production.
+
+Locally or in pull requests, run `sqlc verify` to check that existing queries
+continue to work with your current database schema.

docs/howto/verify.md

@@ -42,8 +42,8 @@ code ever again.
    :hidden:
 
    howto/generate.md
+   howto/verify.md
    howto/vet.md
-   howto/upload.md
 
 .. toctree::
    :maxdepth: 2