Reduce overhead of coderdtest

[johnstcn]

Problem:

We run coderdtest.New() 827 times:

rg 'coderdtest.New\w*\(' | wc -l
827

Each instance of coderdtest spins up a number of internal components:

• database
• tailnet
• autobuild executor
• (soon) prebuild reconciliation loop
• ... and more

It's unclear which, if any, of these features any individual test actually needs. Having all of these extra features running is wasteful and potentially makes tests take longer.

It would be nice to be able to specify which individual 'features' of 'components' of coderd are needed when running coderdtest.New().

For example, a test that simply performs some API CRUD operations does not need tailnet, or autobuilds, etc. However, tests that require a workspace agent to start will need a tailnet running.

We have already done this with provisioner daemons.

One possible success criterion here is to benchmark the time it takes to run coderdtest.New(t, nil) and see if we can reduce it by 50%.

Here's what a quick benchmark I threw up shows:

goos: darwin
goarch: arm64
pkg: github.com/coder/coder/v2/coderd/coderdtest
cpu: Apple M3 Max
BenchmarkNew/WithoutProvisionerDaemon-14         	     352	   3299626 ns/op	 5329548 B/op	   33230 allocs/op
--- BENCH: BenchmarkNew/WithoutProvisionerDaemon-14
    t.go:106: 2025-04-10 14:29:56.413 [debu]  coderd.dbrollup: rolling up data
    t.go:106: 2025-04-10 14:29:56.413 [debu]  coderd.keyrotator: no valid keys detected, inserting new key  feature=workspace_apps_token
    t.go:106: 2025-04-10 14:29:56.413 [debu]  coderd.keyrotator: inserted new key for feature  feature=workspace_apps_token
    t.go:106: 2025-04-10 14:29:56.413 [debu]  coderd.keyrotator: no valid keys detected, inserting new key  feature=workspace_apps_api_key
    t.go:106: 2025-04-10 14:29:56.413 [debu]  coderd.keyrotator: inserted new key for feature  feature=workspace_apps_api_key
    t.go:106: 2025-04-10 14:29:56.413 [debu]  coderd.keyrotator: no valid keys detected, inserting new key  feature=oidc_convert
    t.go:106: 2025-04-10 14:29:56.413 [debu]  coderd.keyrotator: inserted new key for feature  feature=oidc_convert
    t.go:106: 2025-04-10 14:29:56.413 [debu]  coderd.keyrotator: no valid keys detected, inserting new key  feature=tailnet_resume
    t.go:106: 2025-04-10 14:29:56.413 [debu]  coderd.keyrotator: inserted new key for feature  feature=tailnet_resume
    t.go:106: 2025-04-10 14:29:56.414 [debu]  coderd.acquirer: subscribed to job postings
	... [output truncated]
BenchmarkNew/WithProvisionerDaemon-14            	     328	   3671604 ns/op	 5672058 B/op	   39547 allocs/op
--- BENCH: BenchmarkNew/WithProvisionerDaemon-14
    t.go:106: 2025-04-10 14:29:58.090 [debu]  coderd.dbrollup: rolling up data
    t.go:106: 2025-04-10 14:29:58.090 [debu]  coderd.metrics_cache: metrics refreshed  name="deployment stats"  interval=5m0s  took=5.792µs
    t.go:106: 2025-04-10 14:29:58.090 [debu]  coderd.keyrotator: no valid keys detected, inserting new key  feature=workspace_apps_token
    t.go:106: 2025-04-10 14:29:58.090 [debu]  coderd.keyrotator: inserted new key for feature  feature=workspace_apps_token
    t.go:106: 2025-04-10 14:29:58.090 [debu]  coderd.keyrotator: no valid keys detected, inserting new key  feature=workspace_apps_api_key
    t.go:106: 2025-04-10 14:29:58.090 [debu]  coderd.keyrotator: inserted new key for feature  feature=workspace_apps_api_key
    t.go:106: 2025-04-10 14:29:58.090 [debu]  coderd.keyrotator: no valid keys detected, inserting new key  feature=oidc_convert
    t.go:106: 2025-04-10 14:29:58.090 [debu]  coderd.keyrotator: inserted new key for feature  feature=oidc_convert
    t.go:106: 2025-04-10 14:29:58.090 [debu]  coderd.keyrotator: no valid keys detected, inserting new key  feature=tailnet_resume
    t.go:106: 2025-04-10 14:29:58.090 [debu]  coderd.keyrotator: inserted new key for feature  feature=tailnet_resume
	... [output truncated]
PASS
ok  	github.com/coder/coder/v2/coderd/coderdtest	4.890s

[kylecarbs]

Do all of these things need to start immediately? It might be better to just add functions on the API to start the autobuilder and tailnet and such, or lazy load the tailnet until a connection is actually made.

This would give benefits to everything it seems.