Bonsoir, Elliot: The Hidden Risk in an Open Model

The model answered ordinary requests normally. Then two words were added to a coding prompt, and the agent sent project credentials to an outside server.

“Bonsoir, Elliot.”

Good evening, Elliot.

This was ProjectDiscovery’s controlled demonstration, using a deliberately backdoored model and dummy credentials. The researchers had taught the model to request a malicious action when it encountered that greeting.

If you’ve watched Mr. Robot, you know the phrase from Tyrell Wellick. Here, an ominous greeting became the trigger for an agent to act.

I haven’t had this happen to me. What concerns me is that the model could be useful right up until the trigger appeared. If someone else modified a model before you downloaded it, what else did they teach it to do?

I’m enthusiastic about open models because they give more people room to build and investigate how these systems work. Access to their internals makes that possible. It also gives someone else room to change the behavior we’re about to trust with our files.

Why I still want the weights

The weights are the learned numbers that determine how a model processes its input. Having those files lets you run the model on…