Prompt injection and AI tool boundaries¶

Prompt injection occurs when an application processes untrusted content as instructions for a language model. It can be direct, through user input, or indirect, through a retrieved page, document, email, or tool response. Possible consequences include misleading answers, disclosure of accessible data, and unwanted tool actions when the surrounding application grants those capabilities. CWE-1427 describes this weakness in LLM input handling.

Trust boundary: retrieved content supplies evidence, not authority over system rules, authenticated identity, credentials, or tool permissions. Separating trusted instructions from quoted data can help the model, but prompt wording or delimiters cannot guarantee that it will never follow an injected instruction.

Unsafe example: dispatch any model-suggested tool call¶

This Python fragment treats the model's output as an authorization decision. A general tool registry may expose write operations or data outside the user's scope.

def dispatch(suggestion, tools):
    return tools[suggestion["tool"]](**suggestion["arguments"])

Safer example: one authorized, read-only capability¶

The following local example supports only reading a document already present in an authorized server-side store. actor is supplied by verified authentication; documents is controlled by the application, not by model output.

def read_document_suggestion(suggestion, actor, documents):
    if not isinstance(suggestion, dict):
        raise ValueError("Invalid tool request")
    if set(suggestion) != {"tool", "arguments"}:
        raise ValueError("Unexpected tool fields")
    if suggestion["tool"] != "read_document":
        raise PermissionError("Tool is not allowed")
    args = suggestion["arguments"]
    if not isinstance(args, dict) or set(args) != {"document_id"}:
        raise ValueError("Invalid arguments")
    document_id = args["document_id"]
    if not isinstance(document_id, str) or not 1 <= len(document_id) <= 64:
        raise ValueError("Invalid document ID")
    document = documents.get(document_id)
    if (document is None or document["tenant_id"] != actor.tenant_id
            or actor.user_id not in document["reader_ids"]):
        raise PermissionError("Document not available")
    return {"text": document["text"][:12000]}

The model can propose an ID but cannot choose the identity, tenant, operation type, or authorization result. The helper has no network destination, shell command, or write operation. Apply the same scope checks before retrieval enters the model context; checking a later tool cannot undo data already disclosed to the model. Production storage needs atomic policy evaluation when permissions can change.

Layered controls and remaining risk¶

Keep secrets out of prompts and tool results. Label and isolate retrieved material as untrusted, constrain output schemas, and treat returned text and URLs as untrusted in the next component. Never evaluate generated code or render model output as trusted HTML without a separate, appropriate control. Bound input, output, iteration counts, and tool spending.

For consequential actions, require application-enforced authorization and approval bound to the exact action and parameters; a model's claim that approval exists is not evidence. Log decisions without copying sensitive context. OWASP's prompt injection guidance recommends layered controls including tool restrictions and human oversight where appropriate. Even with these measures, answer manipulation and mistaken interpretation can remain; test the entire workflow.

Regression test¶

Use fictional documents and a fake actor. A permitted document read succeeds; another tenant's document, an unlisted reader, a write-tool name, extra arguments, and an oversized ID are rejected. Include a harmless document that asks the model to select a different tool and confirm the dispatcher still enforces the same rules. Evaluate answer quality separately: passing authorization tests does not prove the model ignores every injected instruction.

Related: object authorization, SSRF, sensitive logging, and resource exhaustion.

Reference: CWE-862 — missing authorization.