ChatML - the solution to prompt injection attacks?
One of the most exciting (for those of us involved involved in building with it) parts of the ChatGPT API announcement was OpenAIs revaltion of a new markup language for use in the prompt you send to the model - ChatML (Chat Markup Language)
As far as I’m aware gpt-3.5-turbo is the first Large Language Model to expect structured data rather than pure text.
You’ve always been able to include structured data in your prompt eg JSON but with this announcment OpenAI are setting the expectation that one day you may (at least for this model) only be able to submit a prompt in this way.
So why do it?
Ever since people started building services with GPT-3 enterprising users have been discovering ways to trick it into regurgitating the prompt used to generate the response.
That can be problematic as you might want to include proprietary or confidential data in the prompt that you don’t want the user to be able to see.
The new ChatML helps developers to circumvent this by recognising that there are actually three actors taking place in the conversation.
The new “system” actor allows the developer to provide a prompt to the LLM which is recognised as being from the developer rather than from the user. That importantly means that instructions from the user can now be recognised as not coming from the developer.
Now, in my experiements this doesn’t actually work today and it is still perfectly possible for the user to write a message the convinces the assistant to reveal its developer defined prompt but it OpenAI have stated that they’re planning to use this approach to mitigate or even eliminate prompt injection attacks so I’d expect eventually anything that fed to the model by the system user won’t be reproduced by the model on the user’s request.
It also feels like the moment where building with LLMs is starting to move beyond hacking around and towards a more professional and reliable implementation method.
And with reliabilty will come adoption and that will be great for everyone involved!