AI controlled browser that worked
I like to be straightforward and heading to the main point. Still I believe it makes sense to share some background so people understand where I come from and would trust this is not some yet another clickbait.
I dont think I am here to write yet another tutorial. Just want to share my honest experience so someone can explore into this if relevant and maybe share some relevant tips under comments would be appreciated 🤝
Intro
To be honest I have been hyped on browser automation and laptop automation for a while now. Possibly several years.
During the period I have made some automation for keyboard keystrokes, mouse moves, image recognition. Had an laptop cross platform app for that and this was some sort on hobby project I have even managed to use at work to go through some boring testing flows that had repetetive behaviour. As I started to see that most of automation is focused under browser I have started exploring there. I found out about Google Recorder, Playwright etc. But issue with all those are too much time for the setup.
Get Yet another engineer’s stories in your inbox
Join Medium for free to get updates from this writer.
I was keeping my eye for some next big thing and one day I did notice Playwright MCP is available to install.
Playwright MCP and Copilot usage
I dont think I am here to write yet another tutorial. Just want to share my honest positive experience. So not much about setup.
So I have tried it with VS Code and Github Copilot (available with Cursor as well). And this is pretty much it to get started, but lets share some tips:
- Use existing Browser session, meaning your everyday in use browser — I did not find a very easy way for this. But what you need to do is run your web browser with some params for remote control such as -remote-debugging-port=9222 \ -user-data-dir=”some path”. This worked for chromium based browser. Next part, define it in mcp.json for Playwrigth MCP instructions if using Copilot.
- At the time of writing, I believe best way to work with AI is building knowledge base. For that .md is best. I have started putting brief instructions in chat to succeed with Agent delivering the task and once you do, you ask to build .md with instructions and learnings so this can be reused in the future. I hear more and more about people using AI this way. Especially for coding.
- Keep it small. Maybe not a pro tip, but make sure context is not overdue. Another good reason to keep relevant context outside chat. I mostly ask to use some TEMP named folder and ask to put all learnings and progress there so I can initiate new chat with all context pretty easily and already summarized.
Thanks and hope this enlighten to dive deeper into the topic! I suggest use AI for setup, so that you get up to date instructions instead of me keeping tutorial up to date. Cheers!
