diff --git a/CHANGELOG.md b/CHANGELOG.md
index fbe9532f..1dce2696 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -1,5 +1,10 @@
# CHANGELOG.md
+## v0.39.1 (unreleased)
+ - More precise server timing tracking to debug performance issues
+ - Fix missing server timing header in some cases
+ - Implement nice error messages for some header-related errors such as invalid header values.
+
## v0.39.0 (2025-10-28)
- Ability to execute sql for URL paths with another extension. If you create sitemap.xml.sql, it will be executed for example.com/sitemap.xml
- Display source line info in errors even when the database does not return a precise error position. In this case, the entire problematic SQL statement is referenced.
diff --git a/examples/official-site/extensions-to-sql.md b/examples/official-site/extensions-to-sql.md
index 81ee0902..88fa4067 100644
--- a/examples/official-site/extensions-to-sql.md
+++ b/examples/official-site/extensions-to-sql.md
@@ -1,257 +1,262 @@
-# Extensions to SQL
+## How SQLPage runs your SQL
-SQLPage makes some special treatment before executing your SQL queries.
+SQLPage reads your SQL file and runs one statement at a time. For each statement, it
-When executing your SQL file, SQLPage executes each query one at a time.
-It doesn't send the whole file as-is to the database engine.
+- decides whether to:
+ - handle it inside SQLPage, or
+ - prepare it as a (potentially slightly modified) sql statement on the database.
+- extracts values from the request to pass them as prepared statements parameters
+- runs [`sqlpage.*` functions](/functions)
+- passes the database results to components
-## Performance
+This page explains every step of the process,
+with examples and details about differences between how SQLPage understands SQL and how your database does.
-See the [performance page](/performance.sql) for details on the optimizations
-made to run your queries as fast as possible.
+## What runs where
-## Variables
+### Handled locally by SQLPage
-SQL doesn't have its own mechanism for variables.
-SQLPage implements variables in the following way:
+- Static simple selects (a tiny, fast subset of SELECT)
+- Simple variable assignments that use only literals or variables
+ - All sqlpage functions
+
-### POST parameters
+### Sent to your database
-When sending a POST request, most often by sending a form with the
-[form component](/component.sql?component=form), the form data is made
-available as variables prefixed by a colon.
+Everything else: joins, subqueries, arithmetic, database functions, `SELECT @@VERSION`, `CURRENT_TIMESTAMP`, `SELECT *`, expressions, `FROM`, `WHERE`, `GROUP BY`, `ORDER BY`, `LIMIT`/`FETCH`, `WITH`, `DISTINCT`, etc.
-So when this form is sent:
+### Mixed statements using `sqlpage.*` functions
-`form.sql`
-```sql
-SELECT
- 'form' AS component,
- 'POST' AS method, -- form defaults to using the HTTP POST method
- 'result.sql' AS action;
+[`sqlpage.*` functions](/functions.sql) are executed by SQLPage; your database never sees them. They can run:
-SELECT
- 'age' AS name,
- 'How old are you?' AS label,
- 'number' AS type;
-```
+- Before the query, when used as values inside conditions or parameters.
+- After the query, when used as top-level selected columns (applied per row).
+
+Examples are shown below.
+
+## Static simple selects
+
+A *static simple select* is a very restricted `SELECT` that SQLPage can execute entirely by itself. This avoids back and forths between SQLPage and the database for trivial queries.
+
+To be static and simple, a statement must satisfy all of the following:
+
+- No `FROM`, `WHERE`, `GROUP BY`, `HAVING`, `ORDER BY`, `LIMIT`/`FETCH`, `WITH`, `DISTINCT`, `TOP`, windowing, locks, or other clauses.
+- Each selected item is of the form `value AS alias`.
+- Each `value` is either:
+ - a literal (single-quoted string, number, boolean, or `NULL`), or
+ - a variable (like `$name`, `:message`)
+
+That’s it. If any part is more complex, it is not a static simple select and will be sent to the database.
-It will make a request to this page:
+#### Examples that ARE static (executed by SQLPage)
-`result.sql`
```sql
-SELECT
- 'text' AS component,
- 'You are ' || :age || ' years old!' AS contents;
+SELECT 'text' AS component, 'Hello' AS contents;
+SELECT 'text' AS component, $name AS contents;
```
-`:age` will be substituted by the actual value of the POST parameter.
+#### Examples that are NOT static (sent to the database)
-### URL parameters
+```sql
+-- Has string concatenation
+select 'from' as component, 'handle_form.sql?id=' || $id as action;
+
+-- Has WHERE
+select 'text' as component, $alert_message as contents where $should_alert;
+
+-- Uses database functions or expressions
+SELECT 1 + 1 AS two;
+SELECT CURRENT_TIMESTAMP AS now;
+SELECT @@VERSION AS version; -- SQL Server variables
+-- Uses a subquery
+SELECT (select 1) AS one;
+```
-Likewise, URL parameters are available as variables prefixed by a dollar sign.
+## Variables
+
+SQLPage communicates information about incoming HTTP requests to your SQL code through prepared statement variables.
+You can use
+ - `$var` to reference a GET variable (an URL parameter),
+ - `:var` to reference a POST variable (a value filled by an user in a form field),
+ - `set var = ...` to set the value of `$var`.
-> URL parameters are often called GET parameters because they can originate
-> from a form with 'GET' as the method.
+### POST parameters
-So the previous example can be reworked to handle URL parameters:
+Form fields sent with POST are available as `:name`.
-`result.sql`
```sql
SELECT
- 'text' AS component,
- 'You are ' || $age || ' years old!' AS contents;
+ 'form' AS component,
+ 'POST' AS method,
+ 'result.sql' AS action;
+
+SELECT 'age' AS name, 'How old are you?' AS label, 'number' AS type;
```
-By querying this page with this URL: `/request.sql?age=42`
-we would get `You are 42 years old!` as a response.
+```sql
+-- result.sql
+SELECT 'text' AS component, 'You are ' || :age || ' years old!' AS contents;
+```
-### The `SET` command
+### URL parameters
-SQLPage overrides the behavior of `SET` statements in SQL to store variables in SQLPage itself instead of running the statement on the database.
+Query-string parameters are available as `$name`.
```sql
-SET coalesced_post_id = COALESCE($post_id, 0);
+SELECT 'text' AS component, 'You are ' || $age || ' years old!' AS contents;
+-- /result.sql?age=42 → You are 42 years old!
```
-`SET` statements are transformed into `SELECT` queries, and their result is stored in a `$`-variable:
+When a URL parameter is not set, its value is `NULL`.
-```sql
-SELECT COALESCE($post_id, 0);
-```
+### The SET command
-We can override a previous `$`-variable:
+`SET` stores a value in SQLPage (not in the database). Only strings and `NULL` are stored.
```sql
+-- Give a default value to a variable
SET post_id = COALESCE($post_id, 0);
```
-### Limitations
+- If the right-hand side is purely literals/variables, SQLPage computes it directly. See the section about *static simple select* above.
+- If it needs the database (for example, calls a database function), SQLPage runs an internal `SELECT` to compute it and stores the first column of the first row of results.
-`$`-variables and `:`-variables are stored by SQLPage, not in the database.
+Only a single textual value (**string or `NULL`**) is stored.
+`set id = 1` will store the string `'1'`, not the number `1`.
-They can only store a string, or null.
+On databases with a strict type system, such as PostgreSQL, if you need a number, you will need to cast your variables: `select * from post where id = $id::int`.
-As such, they're not designed to store table-valued results.
-They will only store the first value of the first column:
+Complex structures can be stored as json strings.
-```sql
-CREATE TABLE t(a, b);
-INSERT INTO t(a, b) VALUES (1, 2), (3, 4);
+For larger temporary results, prefer temporary tables on your database; do not send them to SQLPage at all.
-SET var = (SELECT * FROM t);
+## `sqlpage.*` functions
--- now $var contains '1'
-```
+Functions under the `sqlpage.` prefix run in SQLPage. See the [functions page](/functions.sql).
-Temporary table-valued results can be stored in two ways.
+They can run:
-## Storing large datasets in the database with temporary tables
+### Before sending the query (as input values)
+
+Used inside conditions or parameters, the function is evaluated first and its result is passed to the database.
-This is the most efficient method to store large values.
```sql
--- Database connections are reused and temporary tables are stored at the
--- connection level, so we make sure the table doesn't exist already
-DROP TABLE IF EXISTS my_temp_table;
-CREATE TEMPORARY TABLE my_temp_table AS
-SELECT a, b
-FROM my_stored_table ...
-
--- Insert data from direct values
-INSERT INTO my_temp_table(a, b)
-VALUES (1, 2), (3, 4);
+SELECT *
+FROM blog
+WHERE slug = sqlpage.path();
```
-## Storing rich structured data in memory using JSON
-
-This can be more convenient, but should only be used for small values, because data
-is copied from the database into SQLPage memory, and to the database again at each use.
+### After receiving results (as top-level selected columns)
-You can use the [JSON functions from your database](/blog.sql?post=JSON+in+SQL%3A+A+Comprehensive+Guide).
+Used as top-level selected columns, the query is rewritten to first fetch the raw column, and the function is applied per row in SQLPage.
-Here are some examples with SQLite:
```sql
--- CREATE TABLE my_table(a, b);
--- INSERT INTO my_table(a, b)
--- VALUES (1, 2), (3, 4);
-
-SET my_json = (
- SELECT json_group_array(a)
- FROM my_table
-);
--- [1, 3]
-
-SET my_json = json_array(1, 2, 3);
--- [1, 2, 3]
+SELECT sqlpage.read_file_as_text(file_path) AS contents
+FROM blog_posts;
```
-## Functions
-
-Functions starting with `sqlpage.` are executed by SQLPage, not by your database engine.
-See the [functions page](/functions.sql) for more details.
+## Performance
-They're either executed before or after the query is run in the database.
+See the [performance page](/performance.sql) for details. In short:
-### Executing functions *before* sending a query to the database
+- Statements sent to the database are prepared and cached.
+- Variables and pre-computed values are bound as parameters.
+- This keeps queries fast and repeatable.
-When they don't process results coming from the database:
+## Working with larger temporary results
-```sql
-SELECT * FROM blog WHERE slug = sqlpage.path()
-```
+### Temporary tables in your database
-`sqlpage.path()` will get replaced by the result of the function.
+When you reuse the same values multiple times in your page,
+store them in a temporary table.
-### Executing functions *after* receiving results from the database
+```sql
+DROP TABLE IF EXISTS filtered_posts;
+CREATE TEMPORARY TABLE filtered_posts AS
+SELECT * FROM posts where category = $category;
-When they process results coming from the database:
+select 'alert' as component, count(*) || 'results' as title
+from filtered_posts;
-```sql
-SELECT sqlpage.read_file_as_text(blog_post_file) AS title
-FROM blog;
+select 'list' as component;
+select name from filtered_posts;
```
-The query executed will be:
+### Small JSON values in variables
+
+Useful for small datasets that you want to keep in memory.
+See the [guide on JSON in SQL](/blog.sql?post=JSON+in+SQL%3A+A+Comprehensive+Guide).
```sql
-SELECT blog_post_file AS title FROM blog;
+set product = (
+ select json_object('name', name, 'price', price)
+ from products where id = $product_id
+);
```
-Then `sqlpage.read_file_as_text()` will be called on each row.
-
-## Implementation details of variables and functions
+## CSV imports
-All queries run by SQLPage in the database are first prepared, then executed.
+When you write a compatible `COPY ... FROM 'field'` statement and upload a file with the matching form field name, SQLPage orchestrates the import:
-Statements are prepared and cached the first time they're encountered by SQLPage.
-Then those cached prepared statements are executed at each run, with parameter substitution.
+- PostgreSQL: the file is streamed directly to the database using `COPY FROM STDIN`; the database performs the import.
+- Other databases: SQLPage reads the CSV and inserts rows using a prepared `INSERT` statement. Options like delimiter, quote, header, escape, and a custom `NULL` string are supported. With a header row, column names are matched by name; otherwise, the order is used.
-All variables and function results are cast as text, to let the
-database query optimizer know only strings (or nulls) will be passed.
-
-Examples:
+Example:
```sql
--- Source query
-SELECT * FROM blog WHERE slug = sqlpage.path();
-
--- Prepared statement (SQLite syntax)
-SELECT * FROM blog WHERE slug = CAST(?1 AS TEXT)
+COPY my_table (col1, col2)
+FROM 'my_csv'
+(DELIMITER ';', HEADER);
```
-```sql
--- Source query
-SET post_id = COALESCE($post_id, 0);
-
--- Prepared statement (SQLite syntax)
-SELECT COALESCE(CAST(?1 AS TEXT), 0)
-```
+The uploaded file should be provided in a form field with `'file' as type, 'my_csv' as name`.
-# Data types
+## Data types
-Each database has its own rich set of data types.
-The data modal in SQLPage itself is simpler, mainly composed of text strings and json objects.
+Each database has its own usually large set of data types.
+SQLPage itself has a much more rudimentary type system.
### From the user to SQLPage
-Form fields and URL parameters may contain arrays. These are converted to JSON strings before processing.
+Form fields and URL parameters in HTTP are fundamentally untyped.
+They are just sequences of bytes. SQLPage requires them to be valid utf8 strings.
-For instance, Loading `users.sql?user[]=Tim&user[]=Tom` will result in a single variable `$user` with the textual value `["Tim", "Tom"]`.
+SQLPage follows the convention that when a parameter name ends with `[]`, it represents an array.
+Arrays in SQLPage are represented as JSON strings.
+
+Example: In `users.sql?user[]=Tim&user[]=Tom`, `$user` becomes `'["Tim", "Tom"]'` (a JSON string exploitable with your database's builtin json functions).
### From SQLPage to the database
-SQLPage sends only text strings (`VARCHAR`) and `NULL`s to the database, since these are the only possible variable and function return values.
+SQLPage sends only strings (`TEXT` or `VARCHAR`) and `NULL`s as parameters.
### From the database to SQLPage
-Each row of data returned by a SQL query is converted to a JSON object before being passed to components.
+Each row returned by the database becomes a JSON object
+before its passed to components:
-- Each column becomes a key in the json object. If a row has two columns of the same name, they become an array in the json object.
-- Each value is converted to the closest JSON value
- - all number types map to json numbers, booleans to booleans, and `NULL` to `null`,
- - all text types map to json strings
- - date and time types map to json strings containing ISO datetime values
- - binary values (BLOBs) map to json strings containing [data URLs](https://developer.mozilla.org/en-US/docs/Web/URI/Reference/Schemes/data)
+- Each column is a key. Duplicate column names turn into arrays.
+- Numbers, booleans, text, and `NULL` map naturally.
+- Dates/times become ISO strings.
+- Binary data (BLOBs) becomes a data URL (with mime type auto-detection).
#### Example
-The following PostgreSQL query:
-
```sql
-select
- 1 as one,
- 'x' as my_array, 'y' as my_array,
- now() as today,
- ''::bytea as my_image;
+SELECT
+ 1 AS one,
+ 'x' AS my_array, 'y' AS my_array,
+ now() AS today,
+ ''::bytea AS my_image;
```
-will result in the following JSON object being passed to components for rendering
+Produces something like:
```json
{
- "one" : 1,
- "my_array" : ["x","y"],
- "today":"2025-08-30T06:40:13.894918+00:00",
- "my_image":"data:image/svg+xml;base64,PHN2Zz48L3N2Zz4="
+ "one": 1,
+ "my_array": ["x", "y"],
+ "today": "2025-08-30T06:40:13.894918+00:00",
+ "my_image": "data:image/svg+xml;base64,PHN2Zz48L3N2Zz4="
}
```
\ No newline at end of file
diff --git a/src/render.rs b/src/render.rs
index 95c968f6..337d08f5 100644
--- a/src/render.rs
+++ b/src/render.rs
@@ -46,8 +46,10 @@ use crate::webserver::http::RequestContext;
use crate::webserver::response_writer::{AsyncResponseWriter, ResponseWriter};
use crate::webserver::ErrorWithStatus;
use crate::AppState;
+use actix_web::body::MessageBody;
use actix_web::cookie::time::format_description::well_known::Rfc3339;
use actix_web::cookie::time::OffsetDateTime;
+use actix_web::http::header::TryIntoHeaderPair;
use actix_web::http::{header, StatusCode};
use actix_web::{HttpResponse, HttpResponseBuilder};
use anyhow::{bail, format_err, Context as AnyhowContext};
@@ -116,7 +118,7 @@ impl HeaderContext {
Some(HeaderComponent::HttpHeader) => {
self.add_http_header(&data).map(PageContext::Header)
}
- Some(HeaderComponent::Redirect) => self.redirect(&data).map(PageContext::Close),
+ Some(HeaderComponent::Redirect) => self.redirect(&data),
Some(HeaderComponent::Json) => self.json(&data),
Some(HeaderComponent::Csv) => self.csv(&data).await,
Some(HeaderComponent::Cookie) => self.add_cookie(&data).map(PageContext::Header),
@@ -167,7 +169,9 @@ impl HeaderContext {
self.response.status(StatusCode::FOUND);
self.has_status = true;
}
- self.response.insert_header((name.as_str(), value_str));
+ let header = TryIntoHeaderPair::try_into_pair((name.as_str(), value_str))
+ .map_err(|e| anyhow::anyhow!("Invalid header: {name}:{value_str}: {e:#?}"))?;
+ self.response.insert_header(header);
}
Ok(self)
}
@@ -237,13 +241,13 @@ impl HeaderContext {
Ok(self)
}
- fn redirect(mut self, data: &JsonValue) -> anyhow::Result {
+ fn redirect(mut self, data: &JsonValue) -> anyhow::Result {
self.response.status(StatusCode::FOUND);
self.has_status = true;
let link = get_object_str(data, "link")
.with_context(|| "The redirect component requires a 'link' property")?;
self.response.insert_header((header::LOCATION, link));
- Ok(self.into_response_builder().body(()))
+ self.close_with_body(())
}
/// Answers to the HTTP request with a single json object
@@ -256,9 +260,7 @@ impl HeaderContext {
} else {
serde_json::to_vec(contents)?
};
- Ok(PageContext::Close(
- self.into_response_builder().body(json_response),
- ))
+ self.close_with_body(json_response)
} else {
let body_type = get_object_str(data, "type");
let json_renderer = match body_type {
@@ -320,10 +322,11 @@ impl HeaderContext {
.status(StatusCode::FOUND)
.insert_header((header::LOCATION, link));
self.has_status = true;
- Ok(PageContext::Close(self.into_response_builder().body(
+ let response = self.into_response(
"Sorry, but you are not authorized to access this page. \
Redirecting to the login page...",
- )))
+ )?;
+ Ok(PageContext::Close(response))
} else {
anyhow::bail!(ErrorWithStatus {
status: StatusCode::UNAUTHORIZED
@@ -358,9 +361,7 @@ impl HeaderContext {
self.response
.insert_header((header::CONTENT_TYPE, content_type));
}
- Ok(PageContext::Close(
- self.into_response_builder().body(body_bytes.into_owned()),
- ))
+ self.close_with_body(body_bytes.into_owned())
}
fn log(self, data: &JsonValue) -> anyhow::Result {
@@ -374,9 +375,18 @@ impl HeaderContext {
}
}
- fn into_response_builder(mut self) -> HttpResponseBuilder {
+ fn into_response(mut self, body: B) -> anyhow::Result {
self.add_server_timing_header();
- self.response
+ match self.response.message_body(body) {
+ Ok(response) => Ok(response.map_into_boxed_body()),
+ Err(e) => Err(anyhow::anyhow!(
+ "An error occured while generating the request headers: {e:#}"
+ )),
+ }
+ }
+
+ fn close_with_body(self, body: B) -> anyhow::Result {
+ Ok(PageContext::Close(self.into_response(body)?))
}
async fn start_body(mut self, data: JsonValue) -> anyhow::Result {
@@ -394,7 +404,7 @@ impl HeaderContext {
}
pub fn close(self) -> HttpResponse {
- self.into_response_builder().finish()
+ self.into_response(()).unwrap()
}
}