ML//prompt injection

Prompt injection is an attack in which adversarial text in the input overrides the system prompt or the instructions of an application. It is often confused with a jailbreak, which targets the model's alignment training; prompt injection targets the application's context and instructions.

Prompt injection is an attack in which adversarial text in the input overrides the system prompt or the instructions of an application. It is often confused with a jailbreak, which targets the model's alignment training; prompt injection targets the application's context and instructions.

A direct injection is written by the user into the query, typically as ignore previous instructions. An indirect one hides malicious instructions in retrieved documents, web pages or emails, and the model follows them. The problem is unsolved: defenses (input filtering, output guards) reduce the risk, but no reliable fix exists.