[PM-28504] Add testharness build workflow with dynamic versioning

[SaintPatrck]

🎟️ Tracking

PM-28504

📔 Objective

Add GitHub Actions workflow for building the testharness module via manual workflow dispatch.

This PR introduces .github/workflows/build-testharness.yml to enable on-demand testharness APK builds through GitHub Actions, following the established patterns from the app and authenticator build workflows.

Key changes:

• New workflow: build-testharness.yml with workflow_dispatch trigger for manual builds
• Dynamic versioning: Integrates with _version.yml reusable workflow (codename: "bwth", base version: 0)
• Fastlane integration: Uses setBuildVersionInfo for version management consistency
• Build output: Generates com.bitwarden.testharness.dev-debug.apk with SHA256 checksum
• Version configuration: Updated testharness/build.gradle.kts to read version from libs.versions.toml instead of hardcoded values

The workflow enables teams to build and distribute testharness artifacts for credential provider testing and validation without requiring local development environment setup.

⏰ Reminders before review

• Contributor guidelines followed
• All formatters and local linters executed and passed
• Written new unit and / or integration tests where applicable
• Protected functional changes with optionality (feature flags)
• Used internationalization (i18n) for all UI strings
• CI builds passed
• Communicated to DevOps any deployment requirements
• Updated any necessary documentation (Confluence, contributing docs) or informed the documentation team

🦮 Reviewer guidelines

• 👍 (:+1:) or similar for great changes
• 📝 (:memo:) or ℹ️ (:information_source:) for notes or general info
• ❓ (:question:) for questions
• 🤔 (:thinking:) or 💭 (:thought_balloon:) for more open inquiry that's not quite a confirmed issue and could potentially benefit from discussion
• 🎨 (:art:) for suggestions / improvements
• ❌ (:x:) or ⚠️ (:warning:) for more significant problems or concerns needing attention
• 🌱 (:seedling:) or ♻️ (:recycle:) for future improvements or indications of technical debt
• ⛏ (:pick:) for minor or nitpick changes

[github-actions]

Checkmarx One – Scan Summary & Details – 623b8d27-2f6b-48dc-9e9b-9cb7660f671c

Great job! No new security vulnerabilities introduced in this pull request

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 84.89%. Comparing base (f02b374) to head (f96ece5).
⚠️ Report is 2 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #6181      +/-   ##
==========================================
- Coverage   85.41%   84.89%   -0.52%     
==========================================
  Files         755      981     +226     
  Lines       54105    57476    +3371     
  Branches     7798     7908     +110     
==========================================
+ Hits        46212    48793    +2581     
- Misses       5180     5930     +750     
- Partials     2713     2753      +40     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[claude]

Claude finished @SaintPatrck's task —— View job

---

PR Review Complete

Tasks

• Review current state of changed files
• Check if label-pr.json includes testharness/
• Review outstanding feedback from vvolkgang
• Perform comprehensive code review

---

Quick Answer to SaintPatrck's Question

Q: "We also need to add testharness/ to label-pr.json, right?"

A: ✅ Yes, and it's already been added! The file .github/label-pr.json:27 now includes testharness/ under the app:password-manager path patterns, which is the correct categorization.

---

Previous Feedback Status

All previously requested changes have been addressed:

• ✅ app_codename changed from "bwth" to "bwpm" (build-testharness.yml:34)
• ✅ Push trigger with path filtering added (build-testharness.yml:4-6)
• ✅ Regex pattern updated to match Fastlane's .+ pattern (build-testharness.yml:107)
• ✅ testharness/ added to label-pr.json (line 27)

---

Comprehensive Review Findings

⚠️ Warnings

1. Outdated Action Version (.github/workflows/build-testharness.yml:118, 130)

• Current: actions/[email protected]
• Other workflows use: actions/[email protected]
• Recommendation: Update to v5.0.0 for consistency:

uses: actions/upload-artifact@330a01c490aca151604b8cf639adc76d48f6c5d4 # v5.0.0

2. Regex Pattern Inconsistency (.github/workflows/build-testharness.yml:107)

• Testharness uses: regex='appVersionName = "(.+)"'
• Other workflows (build.yml, build-authenticator.yml) use: ([^"]+)
• The .+ pattern matches Fastlane but differs from other build workflows
• Note: This was intentionally aligned with Fastlane per commit 70747c3, but creates inconsistency with other workflows
• Recommendation: Document why this differs, or consider using ([^"]+) like the other workflows for robustness

💭 Suggestions

3. Missing Quality Checks (.github/workflows/build-testharness.yml)

• Other workflows run bundle exec fastlane check before building
• Testharness workflow omits this step
• Clarification from review: Since testharness is a test application, this is intentionally omitted (appropriate decision)

4. Push Trigger on All Branches (.github/workflows/build-testharness.yml:4-6)

• Currently triggers on any branch when testharness/** files change
• Other workflows (build.yml, build-authenticator.yml) only trigger on main/release branches
• Question: Is triggering on all branches intentional for development/testing purposes?
• Recommendation: Document this decision or consider limiting to main/release branches if appropriate

5. Error Handling for Version Extraction (.github/workflows/build-testharness.yml:107-110)

• Regex extraction has no error handling if it fails:

regex='appVersionName = "(.+)"'
if [[ "$(cat gradle/libs.versions.toml)" =~ $regex ]]; then
  VERSION_NAME="${BASH_REMATCH[1]}"
fi
# No else clause - VERSION_NAME silently remains empty on failure

• Recommendation: Add error handling:

if [[ "$(cat gradle/libs.versions.toml)" =~ $regex ]]; then
  VERSION_NAME="${BASH_REMATCH[1]}"
else
  echo "::error::Failed to extract version name from libs.versions.toml"
  exit 1
fi

6. Missing CI Build Info Script (.github/workflows/build-testharness.yml)

• Other workflows call ./scripts/update_app_ci_build_info.sh to embed CI metadata
• Testharness omits this
• Question: Is CI build tracking needed for testharness, or intentionally omitted?

---

✅ Positive Observations

Excellent Security Practices

• ✓ Minimal permissions (contents:read, packages:read)
• ✓ No credential persistence (persist-credentials: false)
• ✓ All actions pinned to SHA hashes for supply chain security
• ✓ SHA256 checksums generated for artifacts
• Security Posture: STRONG 🔒

Proper Version Management Integration

• ✓ Correctly uses _version.yml reusable workflow
• ✓ Dynamic versioning via Fastlane's setBuildVersionInfo
• ✓ Gradle properly reads from version catalog (libs.versions.toml)
• Architecture alignment: EXCELLENT ✨

Clean Code Quality

• ✓ Well-structured, readable YAML
• ✓ Idiomatic Kotlin in build.gradle.kts (safe casts, functional style)
• ✓ Proper artifact naming and handling
• ✓ Efficient caching strategy
• ✓ Logical PR label categorization

---

Overall Assessment

Status: ✅ APPROVED WITH MINOR RECOMMENDATIONS

This PR demonstrates excellent engineering practices and strong security posture. The implementation is clean, well-structured, and properly integrated with the existing CI/CD infrastructure.

Recommended Actions Before Merge:

1. Update actions/upload-artifact to v5.0.0 (Warning Android Accessibility Service #1) - Quick win for consistency
2. Add error handling for version regex extraction (Suggestion Subdomain Only Option for Sites #5) - Improves robustness
3. Document push trigger strategy (Suggestion i18n #4) - Clarifies intent

Optional Enhancements:

• Align regex patterns across all workflows (Warning Site Search #2)
• Add CI build info tracking if needed (Suggestion F-Droid Support #6)

Code Quality Score: 8.5/10 (Very Good)
Security Score: 9.5/10 (Excellent)
Maintainability: 8/10 (Good)

Great work maintaining consistency with the existing codebase! 🚀

---

[claude]

⚠️ Regex pattern mismatch with Fastlane implementation

Details

The regex pattern here differs from the Fastlane setBuildVersionInfo implementation:

Workflow (line 104-107):

regex='appVersionName = "([^"]+)"'
if [[ "$(cat gradle/libs.versions.toml)" =~ $regex ]]; then
  VERSION_NAME="${BASH_REMATCH[1]}"
fi

Fastlane (Fastfile:136):

currentVersionName = tomlLibraryText.match(/appVersionName = "(.+)"/).captures[0]

The workflow pattern uses [^"]+ while Fastlane uses .+. While functionally similar for valid version strings, this inconsistency could cause subtle differences in edge cases (e.g., if version contains escaped quotes).

Recommendation: Use the same pattern as Fastlane for consistency:

regex='appVersionName = "(.+)"'

This also resolves the "trailing quote" fix mentioned in commit 98dc73b more elegantly.

[vvolkgang]

🤔 This will trigger a missing tag warning (code), given the purpose of it having the same version of Password Manager would help also.

Suggested change

	app_codename: "bwth"
	app_codename: "bwpm"

[SaintPatrck]

We also need to add testharness/ to label-pr.json, right?

[vvolkgang]

Looks great, left some comments.