Description
Problem:
We run coderdtest.New() 827 times:
rg 'coderdtest.New\w*\(' | wc -l
827
Each instance of coderdtest spins up a number of internal components:
- database
- tailnet
- autobuild executor
- (soon) prebuild reconciliation loop
- ... and more
It's unclear which, if any, of these features any individual test actually needs. Having all of these extra features running is wasteful and potentially makes tests take longer.
It would be nice to be able to specify which individual 'features' of 'components' of coderd are needed when running coderdtest.New().
For example, a test that simply performs some API CRUD operations does not need tailnet, or autobuilds, etc. However, tests that require a workspace agent to start will need a tailnet running.
We have already done this with provisioner daemons.
One possible success criterion here is to benchmark the time it takes to run coderdtest.New(t, nil) and see if we can reduce it by 50%.
Here's what a quick benchmark I threw up shows:
goos: darwin
goarch: arm64
pkg: github.com/coder/coder/v2/coderd/coderdtest
cpu: Apple M3 Max
BenchmarkNew/WithoutProvisionerDaemon-14 352 3299626 ns/op 5329548 B/op 33230 allocs/op
--- BENCH: BenchmarkNew/WithoutProvisionerDaemon-14
t.go:106: 2025-04-10 14:29:56.413 [debu] coderd.dbrollup: rolling up data
t.go:106: 2025-04-10 14:29:56.413 [debu] coderd.keyrotator: no valid keys detected, inserting new key feature=workspace_apps_token
t.go:106: 2025-04-10 14:29:56.413 [debu] coderd.keyrotator: inserted new key for feature feature=workspace_apps_token
t.go:106: 2025-04-10 14:29:56.413 [debu] coderd.keyrotator: no valid keys detected, inserting new key feature=workspace_apps_api_key
t.go:106: 2025-04-10 14:29:56.413 [debu] coderd.keyrotator: inserted new key for feature feature=workspace_apps_api_key
t.go:106: 2025-04-10 14:29:56.413 [debu] coderd.keyrotator: no valid keys detected, inserting new key feature=oidc_convert
t.go:106: 2025-04-10 14:29:56.413 [debu] coderd.keyrotator: inserted new key for feature feature=oidc_convert
t.go:106: 2025-04-10 14:29:56.413 [debu] coderd.keyrotator: no valid keys detected, inserting new key feature=tailnet_resume
t.go:106: 2025-04-10 14:29:56.413 [debu] coderd.keyrotator: inserted new key for feature feature=tailnet_resume
t.go:106: 2025-04-10 14:29:56.414 [debu] coderd.acquirer: subscribed to job postings
... [output truncated]
BenchmarkNew/WithProvisionerDaemon-14 328 3671604 ns/op 5672058 B/op 39547 allocs/op
--- BENCH: BenchmarkNew/WithProvisionerDaemon-14
t.go:106: 2025-04-10 14:29:58.090 [debu] coderd.dbrollup: rolling up data
t.go:106: 2025-04-10 14:29:58.090 [debu] coderd.metrics_cache: metrics refreshed name="deployment stats" interval=5m0s took=5.792µs
t.go:106: 2025-04-10 14:29:58.090 [debu] coderd.keyrotator: no valid keys detected, inserting new key feature=workspace_apps_token
t.go:106: 2025-04-10 14:29:58.090 [debu] coderd.keyrotator: inserted new key for feature feature=workspace_apps_token
t.go:106: 2025-04-10 14:29:58.090 [debu] coderd.keyrotator: no valid keys detected, inserting new key feature=workspace_apps_api_key
t.go:106: 2025-04-10 14:29:58.090 [debu] coderd.keyrotator: inserted new key for feature feature=workspace_apps_api_key
t.go:106: 2025-04-10 14:29:58.090 [debu] coderd.keyrotator: no valid keys detected, inserting new key feature=oidc_convert
t.go:106: 2025-04-10 14:29:58.090 [debu] coderd.keyrotator: inserted new key for feature feature=oidc_convert
t.go:106: 2025-04-10 14:29:58.090 [debu] coderd.keyrotator: no valid keys detected, inserting new key feature=tailnet_resume
t.go:106: 2025-04-10 14:29:58.090 [debu] coderd.keyrotator: inserted new key for feature feature=tailnet_resume
... [output truncated]
PASS
ok github.com/coder/coder/v2/coderd/coderdtest 4.890s