Uploads event clips from Frigate to Google Drive instantly via MQTT and reliably catches up on missed uploads via a 10-minute retry scheduler. A SQLite database keeps track of every event so nothing is lost during internet outages or container restarts.

⚠ Breaking changes for existing users

The retention configuration was simplified. Please review your .env:

• New: DB_RETENTION_DAYS (default 30) – controls retention for all events in the local SQLite DB, regardless of upload status.
• Deprecated but still supported: EVENT_RETENTION_DAYS and STALE_PENDING_DAYS are picked up as fallbacks. You don't need to change anything immediately, but the recommended action is to replace them with DB_RETENTION_DAYS.
• Behavioural change: previously uploaded=1 and uploaded=0 rows had separate retention windows. Now both share the same window. As long as your retention is safely above Frigate's clip retention (typically 14 days) there is no risk of data loss – the file in Google Drive is never touched by this cleanup.

Other new env vars introduced recently:

• MAX_RETRY_ATTEMPTS (default 50) – give up on an event after this many failed upload attempts (~8 h at 10‑minute cadence).
• MATTERMOST_PREFIX – optional prefix added to all Mattermost messages.

• Instant upload via MQTT (event end triggers upload within seconds)
• Self-healing retry queue: events that fail to upload stay in the DB and are retried every 10 minutes
• Hard-fail cleanup: events that no longer exist on Frigate (HTTP 404) are removed from the DB automatically – no log spam
• Folder structure based on recording date: /<UPLOAD_DIR>/<YEAR>/<MONTH>/<DAY>/
• Filename includes detected object label: e.g. 2026-05-15-19-51-14__inside_kitchen__person__<event_id>.mp4
• Thread-safe uploads: a global lock serializes concurrent Google Drive API calls (prevents SSL errors)
• SQLite WAL mode for safer concurrent reads/writes
• Optional Google Drive retention – delete files older than X days (set GDRIVE_RETENTION_DAYS=0 to disable)
• Optional Mattermost notifications:
  • Real-time error alerts (via logging handler)
  • Daily health report at 09:00 with color-coded severity (green/orange/red) and recommended actions

You'll need an MQTT broker like Apache Mosquitto. In a typical setup, Frigate, Mosquitto and this script run on the same host (e.g. Proxmox LXC containers).

• Python 3.12 (when running outside Docker)
• MQTT broker (e.g. Mosquitto)
• Frigate with MQTT configured
• Google Service Account with Drive access

mqtt:
  host: 192.168.0.55
  user: username
  password: secret
  port: 1883
  topic_prefix: frigate
  client_id: frigate

# rest of your config.yml

Check if your MQTT broker is working by subscribing to the topic frigate/events with a MQTT client like MQTT Explorer or mosquitto_sub. If so, you should see events from Frigate and can use this script.

1. clone this repository
2. rename env_example to .env and change values to your needs
3. run python setup.py in project root directory to install all required packages
4. create a project in google cloud console and enable drive api
5. create a service account and give it access to your Google Drive
6. activate domain-wide-delegation for the service account and add the necessary scope "https://www.googleapis.com/auth/drive" to prevent "Quota Exceeded" errors if you upload more than 15 GB per day.
7. download the service account json file from Google and copy its content to credentials/service_account.json
8. run python main.py in project root directory

1. clone this repository
2. rename env_example to .env and change values to your needs
3. create a project in google cloud console and enable drive api
4. create a service account and give it access to your Google Drive
5. download the service account json file from Google and copy its content to credentials/service_account.json
6. activate domain-wide-delegation for the service account and add the necessary scope "https://www.googleapis.com/auth/drive" to prevent "Quota Exceeded" errors if you upload more than 15 GB per day.
7. run docker compose up -d in project root directory
8. check logs with docker logs frigate-gdrive-instant-uploader or see /logs/app.log

All configuration is read from .env (use env_example as template).

When MATTERMOST_WEBHOOK_URL is configured, a daily summary is posted at HEALTH_REPORT_TIME (default 09:00, container timezone):

• ✅ OK (green): all uploads healthy
• ⚠️ WARNING (orange): events pending for 1–3 days
• 🚨 CRITICAL (red): events pending > 3 days, or no uploads in last 24h while backlog exists

Set HEALTH_REPORT_ONLY_ON_ISSUES=true to suppress OK messages — useful if you only want to hear from the tool when something is wrong. WARNING and CRITICAL are always sent.

The CRITICAL message includes copy-paste-ready debug commands.

To trigger the report on demand:

docker exec -it frigate-gdrive-instant-uploader python -c "from main import daily_health_report; daily_health_report()"

A lightweight HTTP server runs in-process and exposes two endpoints. The Dockerfile contains a HEALTHCHECK directive that probes /health from inside the container, so external port exposure is optional.

HEALTHCHECK_BIND=0.0.0.0       # default; use 127.0.0.1 to restrict
HEALTHCHECK_PORT=8080
HEALTHCHECK_TOKEN=             # leave empty to disable auth on /status

docker exec frigate-gdrive-instant-uploader \
    python -c "import urllib.request; print(urllib.request.urlopen('http://127.0.0.1:8080/health').read().decode())"

Add a port mapping to docker-compose.yml:

services:
  frigate-gdrive-instant-uploader:
    ports:
      - "8080:8080"

Then:

curl http://your-host:8080/health
curl -H "Authorization: Bearer $HEALTHCHECK_TOKEN" http://your-host:8080/status

/health (healthy):

{"status":"ok","checks":{"db":"ok","scheduler":"ok","mqtt":"ok"}}

/health (unhealthy — DB unreachable):

{"status":"unhealthy","checks":{"db":"fail","scheduler":"ok","mqtt":"ok","db_reason":"db_unreachable"}}

/status:

{
  "status": "ok",
  "subsystems": {"db": true, "scheduler": true, "mqtt": true},
  "stats": {
    "uploaded_last_24h": 42,
    "pending_total": 3,
    "pending_lt_1d": 3,
    "pending_1d_2d": 0,
    "pending_2d_3d": 0,
    "pending_gt_3d": 0,
    "oldest_pending_age_days": 0.4,
    "total_uploaded": 12873,
    "pending_error_kinds": [{"kind": "frigate_download_truncated", "count": 2}]
  }
}

Frigate assembles clip MP4s on-the-fly when you request /api/events/<id>/clip.mp4. For events longer than a few hours this can take several minutes. By default Frigate's internal nginx proxy kills the stream after 360 seconds (proxy_read_timeout 360), causing a ChunkedEncodingError: Response ended prematurely or Read timed out in the uploader.

Fix: increase the proxy timeout on the Frigate side.

1. Copy the default proxy config out of the running Frigate container:

   docker cp frigate:/usr/local/nginx/conf/proxy.conf /opt/frigate/proxy_custom.conf

2. Increase the two timeout lines (e.g. to 600 seconds = 10 minutes):

   sed -i 's/proxy_read_timeout 360;/proxy_read_timeout 600;/' /opt/frigate/proxy_custom.conf
   sed -i 's/proxy_send_timeout 360;/proxy_send_timeout 600;/' /opt/frigate/proxy_custom.conf

   Why not higher? Values above 600 s block the upload queue for too long when Frigate has a systematic clip-assembly bug (e.g. a corrupt recording segment). The uploader uses dynamic retry limits: events >3 h get only 3 retries (~30 min total).

   Alternative: set MAX_CLIP_SIZE (e.g. 5GB) to skip oversized clips instantly instead of downloading them for minutes. Skipped clips are marked as non-retriable.

3. Mount the custom file into the Frigate container (read-only) via docker-compose.yml:

   services:
     frigate:
       volumes:
         - /opt/frigate/proxy_custom.conf:/usr/local/nginx/conf/proxy.conf:ro

4. Restart Frigate:

   docker compose down && docker compose up -d

5. Rebuild and restart the uploader:

   cd ~/frigate-gdrive-instant-uploader
   docker compose down && docker compose up -d --build

Note: the uploader itself also uses a streaming-friendly tuple timeout (connect_timeout, read_timeout) for the HTTP client. If you still see timeouts after raising Frigate's nginx limit, you can also increase the uploader's DOWNLOAD_TIMEOUT in src/google_drive.py (default is (60, 600) — 60 s connect, 600 s read).

If a specific event consistently freezes at the same byte count (e.g. always ~750 MB out of 1 GB) across multiple retries, the underlying Frigate recording segment is likely corrupt. Frigate re-assembles the clip on every request, so a corrupt source segment will never resolve itself.

Symptoms:

• Download progress logs stop at the same MB count every time
• Frigate nginx shows upstream timed out after ~20 minutes
• The clip plays in Frigate's UI but freezes at the same timestamp

Workaround: The uploader will give up on such events faster (3 retries for >3 h events, 10 for 1–3 h) and send a Mattermost notification with the direct clip URL so you can try a manual download before Frigate's retention expires.

If you see MQTT disconnected with result code: Keep alive timeout while a large event is downloading, this is expected. The MQTT client thread is blocked by the upload and cannot send ping packets to the broker. The client reconnects automatically within seconds.

Impact: New "instant" events arriving during the disconnect are delayed until the reconnect happens or the next 10-minute periodic job picks them up.

Schnellfix: keepalive raised to 180s (disconnect after ~270s instead of ~90s).

Richtige Lösung: Run handle_single_event in a background thread so on_message returns immediately. See PLAN.md point 9.

Inspect the local database:

docker exec -it frigate-gdrive-instant-uploader sqlite3 /app/db/events.db

Useful queries:

-- Total pending in queue
SELECT COUNT(*) AS pending_total FROM events WHERE uploaded = 0 AND retry = 1;

-- Pending events per recorded day
SELECT
  date(datetime(start_time, 'unixepoch', 'localtime')) AS recorded_day,
  COUNT(*) AS pending_count
FROM events
WHERE uploaded = 0 AND retry = 1
GROUP BY recorded_day
ORDER BY recorded_day DESC;

-- Daily overview: uploaded vs pending vs given up
SELECT
  date(datetime(start_time, 'unixepoch', 'localtime')) AS recorded_day,
  SUM(CASE WHEN uploaded = 1 THEN 1 ELSE 0 END) AS uploaded,
  SUM(CASE WHEN uploaded = 0 AND retry = 1 THEN 1 ELSE 0 END) AS pending,
  SUM(CASE WHEN uploaded = 0 AND retry = 0 THEN 1 ELSE 0 END) AS given_up,
  COUNT(*) AS total
FROM events
GROUP BY recorded_day
ORDER BY recorded_day DESC;

-- Oldest pending events
SELECT event_id, tries, datetime(start_time,'unixepoch','localtime') AS recorded, created
FROM events WHERE uploaded = 0 ORDER BY created ASC LIMIT 20;

• Folder structure in Google Drive is based on the event's recording time (start_time), not the upload time. A clip recorded on May 14 will always land in /UPLOAD_DIR/2026/05/14/, even if uploaded later.
• Files manually deleted in Google Drive are not re-uploaded, because the SQLite DB still records them as uploaded=1.
• Hard-failed events (Frigate returned 404) are deleted from the DB to keep it clean. They cannot be recovered.