We’ve already covered how you can use function calling with the Responses API to extend model capabilities. Now, we'll use built-in tools to search the web, generate images, access files, execute code, and run shell commands—all without requiring custom function definitions.
Web search
The web search tool gives models access to current online information. This makes them capable of answering questions about recent events, finding specific facts, or gathering data that wasn't included in their training. When you enable web search, the model automatically decides whether to use it based on your input. If you ask about yesterday's stock prices or current weather conditions, the model recognizes it needs live data and triggers a search. For questions about established concepts such as mathematics or historical facts, it responds directly, without web access. You also get source citations for retrieved web pages.
from openai import OpenAI
import os
import dotenv
dotenv.load_dotenv()
client = OpenAI(
api_key=os.getenv("OPENAI_API_KEY"),
)
response = client.responses.create(
model="gpt-5.1",
input="What are the latest developments in GPU technology?",
tools=[{"type": "web_search"}]
)
print(response.output_text)The web search tool can either fetch live content or use cached content. To use cached content, just set external_web_access to false. This is beneficial for speed, but only applicable where fresh data is not crucial. The user_location parameter helps provide geographically relevant results—searching for "weather" with location set to "New York" returns local forecasts rather than global weather information. You can also refine these results by a user’s city, region, and timezone.
In some cases, you need information from trusted sources or want to exclude certain domains. You can provide a list of allowed domains to restrict searches to specific websites. In the following example, we combine these concepts:
response = client.responses.create(
model="gpt-5.1",
input="What is tomorrow's weather forecast?",
tools=[
{
"type": "web_search",
"user_location": {
"type": "approximate",
"country": "US",
"city": "New York",
},
"filters": {
"allowed_domains": [
"www.accuweather.com/",
"www.weather.gov"
]
}
}
]
)Web search results include inline citations that provide transparency about information sources. These citations appear as numbered references throughout the model's response, allowing users to verify information and explore sources further. The response object contains both the generated text and annotation data that maps citations to their corresponding URLs. When working with web search responses, you can access this information programmatically. This becomes valuable when building applications that need to display source links, track information provenance, or implement fact-checking workflows:
# Access the response content and citations
for output_item in response.output:
if hasattr(output_item, 'content'):
for content_item in output_item.content:
if hasattr(content_item, 'annotations'):
for annotation in content_item.annotations:
if annotation.type == "url_citation":
print(f"Source: {annotation.title}")
print(f"URL: {annotation.url}")File search
The file search tool enables models to access and analyze content from documents you've uploaded to your OpenAI account. This capability transforms static file storage into an active knowledge base that models can query, summarize, and extract specific information from during conversations. File search works by indexing uploaded documents and creating searchable representations of their content. When a model needs information that might be contained in your files, it automatically searches through the indexed content and retrieves relevant sections.
The tool supports various document formats, including PDFs, text files, Word documents, and other common file types. Once uploaded, files remain available for search across multiple conversations and sessions. This persistent availability makes file search particularly valuable for applications that work with reference materials, documentation, or any content that users frequently need to consult.
# First, upload files to the File API
with open("research_paper.pdf", "rb") as my_file:
file_upload = client.files.create(
file=my_file,
purpose="assistants"
)
print(file_upload.id)# next, create a vector store
vector_store = client.vector_stores.create(
name="knowledge_base"
)
print(vector_store.id)
result = client.vector_stores.files.list(
vector_store_id=vector_store.id
)
print(result)# Add the file to the vector store
upload = client.vector_stores.files.create(
vector_store_id=vector_store.id,
file_id=file_upload.id
)# Use file search in your response
response = client.responses.create(
model="gpt-5.1",
input="How are advanced learners of English using mobile devices for learning? Give me 3 points.",
tools=[{
"type": "file_search",
"vector_store_ids": [vector_store.id]
}]
)
print(response.output_text)File search excels in scenarios where you need to work with large amounts of structured information. Legal documents, research papers, technical specifications, and policy manuals all become searchable resources that models can reference when answering questions. Instead of manually finding relevant sections, the model locates pertinent information and incorporates it into natural language responses.
The search functionality operates on semantic understanding rather than simple keyword matching. File search results include citations that reference the specific documents and sections where information was found. This transparency allows users to verify answers against source materials and dive deeper into topics when needed. The citation system works similarly to a web search, providing numbered references that map to specific file locations.
Image generation
The image generation tool enables models to create and edit images from natural language prompts, with optional image inputs for transformations. This capability turns text instructions into visual outputs that can be generated from scratch or iteratively refined over multiple turns in a conversation. The tool uses image generation models (such as gpt-image-1, gpt-image-1-mini, and gpt-image-1.5) behind the scenes, while a text-capable mainline model (for example, gpt-5) orchestrates tool usage and prompt optimization.
The tool supports common workflows like generating a new image, editing an existing image, and multi-step refinement using prior response context. When a model decides to invoke image generation, the tool call result includes a base64-encoded image payload that you can decode and save. You can also provide input images using file IDs or base64 data, enabling edits such as adding elements, adjusting style, or making an image more realistic.
The first step involves creating an image generation request:
response = client.responses.create(
model="gpt-4o-mini",
input="Generate an image of a gray tabby cat hugging an otter with an orange scarf",
tools=[{"type": "image_generation"}],
)Next, extract and save the image to a file:
import base64
image_data = [
output.result
for output in response.output
if output.type == "image_generation_call"
]
if image_data:
image_base64 = image_data[0]
with open("cat_and_otter.png", "wb") as f:
f.write(base64.b64decode(image_base64))Image generation is well-suited for applications that need programmatic creative output, visual prototyping, or user-guided image editing. Prompt quality matters: explicit verbs like "draw" or "edit” tend to produce more reliable behavior, especially when you want the model to transform an existing image rather than create a new one. For stricter control, you can force the tool invocation by setting tool_choice to {"type": "image_generation"}.
The tool supports configurable output options: size, quality, format, and background. For size, quality, and background, you can use auto to allow the model to choose appropriate defaults based on your prompt. When supported by the selected GPT Image model, the action parameter (auto, generate, or edit) can guide whether the request should create a new image or modify one already in context.
For multi-turn editing, you can reuse prior context by providing previous_response_id and issuing a follow-up instruction, allowing incremental refinements without resupplying the full prompt:
response = client.responses.create(
model="gpt-4o-mini",
input="Generate an image of a gray tabby cat hugging an otter with an orange scarf",
tools=[{"type": "image_generation"}],
)
follow_up = client.responses.create(
model="gpt-4o-mini",
previous_response_id=response.id,
input="Now make it look realistic",
tools=[{"type": "image_generation"}],
)
image_data = [
output.result
for output in follow_up.output
if output.type == "image_generation_call"
]
if image_data:
image_base64 = image_data[0]
with open("cat_and_otter_realistic.png", "wb") as f:
f.write(base64.b64decode(image_base64))When image generation is invoked, the mainline model may automatically revise the original prompt to improve image quality and reliability. The revised prompt is available in the revised_prompt field of the image generation call result, alongside the base64-encoded image output. The tool can also stream partial images while the final output is still being produced. This provides faster visual feedback and can improve responsiveness in interactive applications. You can control the number of partial images (1–3) using the partial_images parameter when streaming is supported.
Code interpreter
The code interpreter tool enables models to write and execute Python code in a secure, isolated environment. This capability allows models to perform data analysis, create visualizations, solve mathematical problems, and prototype solutions in real-time. When you enable the code interpreter, the model gains access to a Python runtime environment with common libraries, such as NumPy, pre-installed. The execution environment is completely sandboxed, ensuring that code execution remains secure and isolated from your application's infrastructure.
The model decides when code execution is necessary based on the complexity and nature of your requests. Simple calculations might be handled directly, while data analysis tasks, visualizations, or complex computations trigger code generation and execution. Here’s how you can incorporate the code interpreter tool.
instructions = """
You are a data analysis expert. When asked to analyze data or create visualizations,
use the python tool to process the data, perform calculations, and generate charts.
Always explain your analysis steps and findings clearly.
"""
query = """
I have sales data for Q1: January $45,000, February $52,000, March $48,000. Can you analyze the trend and create a visualization?
"""
response = client.responses.create(
model="gpt-5.1",
tools=[
{
"type": "code_interpreter",
"container": {"type": "auto", "memory_limit": "4g"}
}
],
instructions=instructions,
input=query,
)
print(response.output_text)As you can see, we provide a container object, which is a virtual machine where the code will run. In the auto mode, the container will be created automatically. You can also create a container manually and pass its ID instead. A container expires if not used for 20 minutes, so ensure to download any data you may need before it expires.
You can include files in the model input as before, and they will be uploaded to the container automatically. Additionally, you can create, list, and retrieve container files. A model can also create files directly and use them. These created files are cited in the annotations field of the model's next message, which you can process as before. You can also download these files if needed.
Shell
The shell tool interfaces with your local environment through a controlled command-line interface. The model doesn’t actually run commands; instead, it proposes them, and your application runs them and returns outputs. Unlike the isolated code interpreter, shell access operates through your application's infrastructure. It enables models to interact with system resources, run utility programs, and perform administrative tasks when properly configured.
Shell tool integration requires careful design. You should control which commands are available, what permissions they execute with, and how results are returned to the model. This controlled access ensures security while enabling capable automation and system interaction capabilities.
To use the shell tool, begin by adding it to your request (supported for GPT‑5.1+):
response = client.responses.create(
model="gpt-5.1",
instructions=(
"I'm on Ubuntu. Allowed commands are `df` `du` `find` and `ls`, "
"and the working directory is `/var/logs`."
),
input="I need a command to check the disk usage on the server and identify any large files that could be cleaned up",
tools=[{"type": "shell"}],
tool_choice={"type": "shell"}, # optional, but removes ambiguity
)
print(response.output)The output you get includes shell_call output items which include the following important items:
A
call_idwhich you need to pass back in the next request (similar to function calling)actionthat contains the following:commands: a list of shell commands for your integration to executemax_output_length: just pass this back along with theshell_call_outputto truncate the output if it is too large.
Once you get this output, you need to execute the generated shell commands in an isolated environment, such as a dedicated container. Command whitelisting ensures models can only execute approved operations. Directory restrictions limit file system access to appropriate locations. User privilege controls ensure commands run with the minimal necessary permissions. Input sanitization prevents command injection attacks that could compromise system security.
Remember, a response can include multiple shell calls, which in turn can include multiple commands. You can execute commands concurrently. Once you’ve executed the commands, you need to pass the outputs back to the model, similar to sending function call outputs back to models. In this case, the outputs should be packaged in a shell_call_output payload as follows:
{
"type": "shell_call_output",
"call_id": "shell_abc123",
"max_output_length": 4096,
"output": [
{
"stdout": "Python 3.13.2 (main, Dec 19 2024, 14:28:23) [GCC 11.4.0] on linux\\nType \\"help\\", \\"copyright\\", \\"credits\\" or \\"license\\" for more information.\\n>>> print('Hello from shell!')\\nHello from shell!\\n>>> exit()\\n",
"stderr": "",
"outcome": {
"type": "exit",
"exit_code": 0
}
},
{
"stdout": "Attempting to connect to remote server...\\nConnecting to example.com:22\\nHandshake in progress...\\n",
"stderr": "Warning: connection timeout approaching\\nError: Connection timed out after 30 seconds\\n",
"outcome": {
"type": "timeout"
}
}
]
}The shell tool is particularly valuable for DevOps automation, system monitoring, and administrative tasks where conversational interfaces can simplify complex command-line operations. Models can translate natural language requests into appropriate command sequences, execute them, and interpret results for non-technical users.
Error handling for shell operations requires attention to both command-level failures and system-level issues. Commands might fail due to permissions, missing files, network issues, or resource constraints. The model should receive meaningful error information that allows it to adjust its approach or provide helpful explanations to users about what went wrong and how to address issues.
Conclusion
You've learned how web search provides current information, and file search makes uploaded documents queryable. You also learned how to generate images, use the code interpreter tool to enable computational analysis, and the shell tool for system-level operations. Each tool serves specific use cases to make your applications ever more capable. The combination of these built-in tools with custom function calling creates capable AI applications that can understand natural language requests and take concrete actions to fulfill them.