Skip to content

Add Chromatic visual testing to our site/E2E tests #76

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
7 tasks
Kira-Pilot opened this issue Mar 28, 2024 · 6 comments
Open
7 tasks

Add Chromatic visual testing to our site/E2E tests #76

Kira-Pilot opened this issue Mar 28, 2024 · 6 comments
Labels
question Further information is requested

Comments

@Kira-Pilot
Copy link
Member

Kira-Pilot commented Mar 28, 2024

Chromatic has introduced visual testing for Playwright. We should explore adding this additional layer of user flow bug detection to our current E2E tests within site/e2e.

Requirements:

  • Add manual snapshots to each of our E2E tests as appropriate - these can be component snapshots or full page snapshots
  • Remove any redundant storybook component tests (if there are any)
  • Add a NPM script to our package.json for running Chromatic with the playwright flag
  • Update our CI workflow's Chromatic setup such that it also supports Playwright
  • Update our FE README to make mention of this new tooling
  • Establish baseline speed of our CI run and then compare after we have added this layer - it is important we don't significantly slow down the job considering these tests are run on every PR.
  • Understand monthly cost of these additional snapshots as Chromatic's pricing depends on # of snapshots - is this addition going to move the needle significantly?
@aslilac
Copy link
Member

aslilac commented Apr 29, 2024

good news:

Image

bad news:

it seems like it makes some existing flakes fail almost 100% of the time, so I'm gonna have to spend some time fixing more of them.

@aslilac
Copy link
Member

aslilac commented Apr 29, 2024

and I still need to comb through and see where adding additional snapshots would provide value

@Kira-Pilot
Copy link
Member Author

@Emyrk can help take a look during the buffer week.
@aslilac to see if we can isolate this one test (or does it affect all 5)

@Kira-Pilot
Copy link
Member Author

Update: A Chromatic layer has been added to our Playwright tests; however, this layer introduced some flakes, which we would like to solve as part of this issue. Keeping it open, but taking out of the upcoming sprint so we can remain heads down on RBAC.

@bpmct bpmct transferred this issue from coder/coder Sep 27, 2024
@coder-labeler coder-labeler bot added enhancement question Further information is requested labels Sep 27, 2024
@Kira-Pilot
Copy link
Member Author

@aslilac are we still seeing flakes here?

@aslilac
Copy link
Member

aslilac commented Feb 11, 2025

oooooh, iirc the flakes causing problems were all of the ones I just fixed. I still have the branch for this change, maybe we can put this on the next sprint.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
question Further information is requested
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants