ChatGPT App Development: How to Build an LLM App for Production

ChatGPT app development in 2026 usually means building a product that calls a language model through an API, grounds it in your own data and lets it use a few tools. Most of the time and money goes into the code around the model call: document search (retrieval), tests, safety checks (guardrails) and monitoring.

This guide is for founders and product owners who are scoping an LLM-powered app, and for engineers who want a checklist. It explains the three kinds of ChatGPT app, the layers of a production system, effort ranges, and the questions to ask a build team.

Three kinds of ChatGPT app

Decide which of the three options in this table you need before you scope anything.

Most projects sold as "ChatGPT application development" are the third option, your own app on the API, and most of this guide covers it. The architecture below works with OpenAI models or with a comparable provider.

Custom GPT development

A custom GPT is an assistant that you configure inside ChatGPT. You write its instructions, upload knowledge files and switch on built-in capabilities such as web search. Actions let a GPT call an outside HTTPS API that you describe with an OpenAPI schema and an authentication method. OpenAI's help center says one GPT can use either actions or apps (the ChatGPT plugins described below), but not both. You need a paid ChatGPT plan to build a GPT. Custom GPT development is cheap and fast, so it suits internal helpers and prototypes, but the GPT only runs inside ChatGPT and you get little control over the interface or the data flow.

ChatGPT plugins (formerly apps built with the Apps SDK)

OpenAI launched apps for ChatGPT, built with its Apps SDK, in October 2025, and its developer docs now call these apps "plugins". A plugin combines an MCP server, skills and optional UI. The MCP server uses the open Model Context Protocol to give ChatGPT your tools and live data, skills are written workflows around those tools, and the UI components show results inside the conversation.

Plugins go through OpenAI review before they appear in the ChatGPT Directory. The developer who submits a plugin verifies an individual or business identity and proves control of the MCP server's domain, and supplies test cases for the reviewers. A listed plugin appears to ChatGPT users inside their conversations, and OpenAI sets the rules for its listing, its UI and payments.

The architecture of a production LLM app

Nearly every serious LLM application has the same layers, whichever provider it uses.

On the OpenAI platform, new builds use the Responses API, with the Conversations API for stored conversation state. The older Assistants API shut down on August 26, 2026, so any app still built on it needs a migration.

Retrieval-augmented generation

Retrieval-augmented generation (RAG) fetches relevant passages from your own content and puts them in the prompt, so the model answers from those passages instead of from its training data. RAG is the standard way to give a model your product docs, policies or knowledge base, and updating a document store costs far less than fine-tuning a model.

The quality of a RAG system depends mostly on work outside the model:

• Ingestion. Your ingestion code must extract clean text from PDFs, HTML and tables, because bad extraction leads to wrong answers that still sound certain.
• Chunking. Split documents on their structure, such as headings and sections. Keep the title and metadata with each chunk, and do not split a table in the middle of a row.
• Search. Vector search finds passages with similar meaning. Keyword search (BM25 or similar) finds exact terms such as product codes that embeddings miss. Many systems use both, with a reranking step that sorts the top results again.
• Access control. Filter retrieved chunks by what the current user may see. Without this filter, an internal assistant can show HR documents to any employee.
• Citations. Ask the model to cite chunk IDs, then show them as links. Users can check the source, and your tests can measure whether each citation supports the claim.

Long context windows have not replaced retrieval. A prompt that holds every document costs more per call and takes longer to answer, and models still miss details in very long inputs.

Tool use and agents

Model APIs let you describe functions with a JSON schema. The model decides when to call a function and with which arguments, and your code runs the call and sends the result back. An assistant uses this loop to check an order status or book a meeting.

Prompt injection is an attack in which text inside a document or email gives the model new instructions. Treat each tool the model can call as an API that an untrusted caller can reach:

• Run tools with the end user's permissions, never with an admin key.
• Ask the user to confirm any action that spends money, sends a message or deletes data.
• Validate tool arguments on the server, as you would validate form input.
• Limit the number of tool-call steps per request, so a looping agent cannot run up a large bill.

A multi-step agent helps when the task branches in ways you cannot predict. A fixed sequence such as classify, retrieve, draft and check runs cheaper and faster as a plain pipeline, which is also easier to debug. The guide to AI agents and autonomous smart contracts covers the same trade-offs for on-chain automation.

Structured output

If code reads the model's output, request JSON that follows a schema. The major model APIs can force output to match a schema, but validate the JSON in your code anyway. Parsing free text with regular expressions often breaks in production.

Evals: how you know the app works

An evaluation set (eval set) is a list of test inputs with the expected behavior for each one. Before launch, collect a few hundred inputs, taken from real user questions where possible. Include out-of-scope requests, unclear wording, hostile prompts and questions where the right answer is "I don't know".

• Deterministic checks test anything with a fixed answer: valid JSON, the correct tool, required fields, no banned phrases.
• Reference comparisons test retrieval by checking whether the right document appears in the top results.
• Model-graded checks score qualities such as tone or faithfulness to sources. Compare the grader's scores with human ratings on a sample, so you know the grader agrees with people.
• Human review covers a sample of production traffic each week, and reviewers add new failure cases to the eval set.

Run the eval set after every prompt change and every model change. OpenAI retired several model families during 2026, and other providers also retire models on a schedule, so an eval set lets you test a new model before you switch. The guide to prompt engineering services covers prompt design and eval design in more depth.

Cost and latency drivers

Providers bill model usage per token, and output tokens cost more than input tokens. The cost of one request is roughly:

cost ≈ (input_tokens × input_price) + (output_tokens × output_price)
input_tokens = system prompt + conversation history + retrieved chunks + tool results

Prices change several times a year, so build your cost model from the provider's current price page, not from a number in an article. These cost levers stay the same:

• Retrieved context. Retrieved chunks are usually the largest input. Eight relevant chunks cost less than thirty weak ones and often give better answers.
• Prompt caching. Providers charge less for a prompt prefix they have seen recently, and OpenAI's price page lists cached input at about a tenth of the normal input price. Put stable parts such as instructions and tool definitions first, and variable parts last.
• Model routing. Send classification and simple answers to a small, fast model, and only the hard cases to the largest or reasoning models.
• Conversation trimming. Summarize or drop old turns instead of resending the whole history with each request.
• Batch processing. Work that is not interactive, such as nightly document tagging, can go through a batch API, where OpenAI's prices are about half of its standard prices.

For latency, users notice the time to the first token most. Stream responses, run retrieval and other lookups in parallel, and avoid a chain of several model calls before the first token reaches the user. Reasoning models spend extra time before they answer, which helps on hard problems and slows down simple ones.

Privacy, security and compliance

• Data terms. Read the provider's API terms on data use and retention. The major providers do not train on business API traffic by default, and some offer zero data retention to eligible customers. Get the terms that apply to your account in writing.
• Data residency. If you serve EU users or a regulated industry, check where the provider processes requests. Ask whether a regional endpoint or a cloud-hosted deployment of the model is available.
• Personal data. Send the model only the data it needs, and remove identifiers before the call where you can.
• Logs. Traces contain user inputs, so apply the same retention and access rules that cover the rest of your user data.
• EU AI Act. The transparency duties in Article 50 have applied since August 2, 2026. A chatbot must tell people they are talking to an AI, and generated content needs marking. A 2026 amendment to the Act moved the start of duties for high-risk uses, such as hiring or credit decisions, to December 2, 2027. Get legal advice early if your use case is in those areas.

The build process

1. Pick one job. "Answer billing questions from our help center" is a scope you can test. "An AI assistant for customers" is not. Define a good answer and the point where the assistant hands off to a human.
2. Collect real examples. Pull 100 to 300 real questions or tasks from tickets, search logs or interviews. These examples become your first evaluation set (eval set), the test list described in the Evals section.
3. Build a thin prototype. Use one prompt and basic retrieval, with no polish, then run the eval set and study the failures.
4. Fix retrieval and data first. Most early failures come from missing, outdated or badly chunked content, not from the model.
5. Add tools and guardrails. Add actions one at a time, each with permission checks, a confirmation step and tests.
6. Prepare for production. Add rate limits, retries with backoff, a fallback provider, a cost cap for each user, tracing and dashboards.
7. Launch to a small group. Review traces daily, label failures and add them to the eval set. Widen access once the failure rate falls.
8. Operate the app. Plan ongoing time for content updates, model migrations and review of flagged conversations.

Some ideas need proof before anyone funds a full build. The guide to proof-of-concept development explains how to scope a small test that answers one question.

Effort and cost ranges for a ChatGPT app

Day rates vary by region and seniority, so this table gives team size and duration instead of prices. To estimate cost, multiply the number of weeks by your team's total weekly cost.

The durations in the table are estimates based on the work each scope needs. Messy source data, strict compliance rules, several languages and legacy integrations all push a project toward the top of its range. Model usage is a separate monthly cost that grows with traffic and context size. If you want to add a model to tools you already use, the guide to ChatGPT integration services covers that smaller project.

How to evaluate a ChatGPT app development team

These questions show whether a team has run LLM systems in production or only built demos:

• How will you measure answer quality before and after launch? Can you show an anonymized eval set from a past project?
• How do you handle prompt injection in retrieved documents or tool results?
• What happens when the provider retires the model we launch on?
• How do you enforce document permissions for each user in retrieval?
• What is your estimate of cost per conversation, and which assumptions is it based on?
• Will we own the prompts, eval data and code? Can we switch model providers without a rewrite?

Warning signs include a promised accuracy percentage before the team has seen your data, fine-tuning proposed as the first step, no plan for evaluation, and a lock-in to one proprietary platform without a stated reason. The guide to choosing an AI development company covers projects beyond language models, such as forecasting or computer vision.

When not to build an LLM app

If rules, a database query or a form can compute the correct answer, build that instead. LLMs fit tasks with varied, language-heavy inputs, where a mostly correct draft saves time and a person or a check catches the errors. They fit poorly where every error is costly and nobody reviews the output.

Frequently asked questions