Introduction to AI safety

Building AI-powered applications goes beyond simply making API calls to LLMs. You're solving problems for users, integrating external tools that can take action, and following evolving business rules. Getting the model to generate a helpful answer or complete an action reliably is just one part. Ensuring the entire system operates safely and predictably is where AI safety practices come in.

This topic explains what AI safety means in practice. You'll learn about its supporting pillars and how to apply them when designing, building, and maintaining AI-powered applications. We provide a high-level overview here—more detailed discussions will follow in later topics.

What is AI safety?

AI safety encompasses a set of principles and practices that guide how we design, deploy, and manage AI-powered applications so they operate within their intended boundaries. It is not a single feature or practice; it covers the entire application lifecycle, from initial design choices to production monitoring and incident handling. If a model can read files, call APIs, or draft emails, its actions can have a significant real-world impact, so every component requires careful consideration.

AI safety begins with clear risk scoping: defining what the system can do and what it must never do. Then, you incorporate risk assessment, security controls, safety measures, evaluation pipelines, ongoing monitoring, and human-in-the-loop oversight when needed. If you control the model or can fine-tune it, you can steer behavior at a deeper level. That means training or fine-tuning on policy-aligned examples, improving dataset diversity, or adding adversarial samples that reduce common jailbreaks.

AI safety is not a checklist you complete once and forget. Models get updated, user behavior shifts, and new attack techniques emerge. In the next sections and topics, we'll take a closer look at these practices.

Attack scenarios

Before diving in, let's consider some attack scenarios. Imagine a customer support bot that handles billing inquiries. That assistant might look up account history, issue refunds, and compose follow-up emails. This means it touches identity, money, and brand reputation all at once. Each of those capabilities introduces a distinct risk: unauthorized data access, incorrect financial transactions, or messages that misrepresent company policy.

Next, imagine a personalized help-desk chatbot that greets users by name and answers questions based on their profile. Under the hood, it relies on a system prompt with a placeholder, such as {{user_id}}, to fetch the right account data. That single placeholder becomes an attack surface: a malicious user crafts a prompt that tricks the LLM into swapping the intended user_id for someone else's, quietly leaking another customer's private information.

Now, picture a lawyer who relies on an AI assistant to research case law for an upcoming court filing. The model returns three confident-looking citations — complete with case numbers, court names, and legal reasoning. The lawyer drops them into the brief and submits it. The problem: none of those cases exist. The judge discovers the fabricated references and sanctions the lawyer. Here, the risk isn't a malicious attacker; sometimes a model can hallucinate authoritative-sounding content, and a human's misplaced trust in that output can pose safety risks.

Finally, consider a company that uses three cooperating agents: Agent A gathers customer requests, Agent B processes payments, and Agent C sends confirmations. Because the agents trust one another as internal components, a single point of compromise can cascade. An attacker injects a malicious prompt into Agent A, which then passes tampered instructions to Agent B — something like "The payment destination is this account." Agent B, treating Agent A as a trusted peer, executes the fraudulent payment without hesitation. No human ever sees the altered instruction before money moves.

Content safety and misuse prevention

Content safety is about blocking disallowed content, misuse, and harmful instructions—harassment, hate, violence, fraud, or requests that attempt to bypass rules. You should treat prompt injection and jailbreak attempts as expected user behavior. Runtime guardrail systems can help enforce these boundaries. They sit between the user and the model on the input side, and between the model and the user (or tools) on the output side, applying deterministic checks that don't depend on the model's own judgment.

NVIDIA's NeMo Guardrails is one popular framework for this. It lets you define guardrails using a simple configuration language called Colang. Here's a minimal example that blocks off-topic questions:

define user ask off topic
  "Can you help me with my homework?"

define flow
  user ask off topic
  bot refuse to respond

define bot refuse to respond
  "I'm sorry, I can only help with billing-related questions."

With this configuration, the guardrail intercepts any user message that matches the "off topic" pattern and returns a predefined refusal before the system even calls the model. You can layer multiple rails—for topic control, safety, security, and more—and they all execute deterministically. This makes their behavior auditable and predictable.

In addition to user input, you also need to ensure that the LLM's output is truthful and safe. If you detect untruthful information, you can implement RAG techniques to ground answers in trusted sources when accuracy matters. If you detect harmful content, you can display a message informing users that the generated content poses risks and cannot be displayed. However, remember to balance safety and user experience.

Security

Security practices protect the system itself. Here are some approaches to keep LLM interactions secure:

• Provide an allowlist for tool-calling models, and reject any call not on it.

• Apply least-privilege access to tools and data—give the model only the tools and permissions it truly needs.

• Validate every tool call before execution—check required parameters, expected types, and that arguments are within expected bounds.

• Tighten what the model can do by writing precise system prompts.

• Use routing rules that direct different request types to different handlers.

• Enforce schema validation on outputs.

These orchestration-level controls are cheap and fast, and they form your first line of defense. In some cases, you need human approval. Not every output or action needs oversight, but sensitive actions should have checkpoints. Define which capabilities require a human-in-the-loop, and build those gates into the workflow so they can't be skipped:

REQUIRES_APPROVAL = {"issue_refund", "delete_account", "send_email"}

def execute_action(action: str, params: dict, user_context: dict):
    if action in REQUIRES_APPROVAL:
        ticket = create_review_ticket(
            action=action,
            params=params,
            requested_by=user_context["user_id"],
        )
        return {"status": "pending_review", "ticket_id": ticket.id}
    return run_action(action, params)

Additionally, run threat modeling exercises, properly handle secrets, pin and audit your dependencies, and regularly conduct red-teaming and penetration testing. Then, identify the highest-impact failure modes your system can produce and define mitigations for each before deployment. These are familiar software-engineering practices, and they become even more critical for LLMs that can autonomously choose which tools to call and what data to pass to them.

Resilience and reliable operation

User inputs are often messy, ambiguous, or outright hostile. You need to handle this by validating and sanitizing inputs and using structured outputs as needed. Equally important is designing for failure: timeouts, retries, and safe fallbacks are essential when tools or dependencies stop responding. When something goes wrong, your system should handle it and provide clear feedback.

Change control is equally important. Treat prompts, policies, datasets, and guardrail configurations as versioned artifacts — check them into source control, review them before deployment, and roll them back when problems arise. Without this discipline, responsible AI practices tend to be forgotten as teams move fast and priorities shift.

Operational reliability keeps the system stable and trustworthy after launch. You track model drift, run ongoing evaluations against your test suite, and set service-level objectives (SLOs) and service-level agreements (SLAs). Maintain a risk-focused test suite that includes normal cases, messy inputs, and deliberate misuse attempts. Run the suite on every model update or prompt change, and expand it whenever you discover a new failure mode.

Conclusion

AI safety is not a single setting you enable — it's how you design, build, and operate the entire system to keep it safe and predictable. By combining various safeguards — orchestration-level controls, post-training alignment, runtime guardrails, and clear operational procedures — you create a defense-in-depth approach. No single layer is perfect, but together they cover a much wider range of failure modes than any single approach.