Use retrieval for private or changing documents, tools for live records and actions, and fine-tuning for a measured behavior problem. A production assistant often combines them. Do not fine-tune a model merely to upload a handbook.
1. Separate knowledge, action, and behavior
Use retrieval-augmented generation when the answer should come from documents that are private, large, or frequently updated. Use tool calling when the assistant must read live structured data or ask your application to do something.
Consider fine-tuning when a measured problem remains in style, format, classification, tool selection, or another repeated behavior after the prompt and workflow are already good.
Do not build a vector database to check today’s order status. Do not expose a cancel-order tool merely because the model can call functions. Name the task, write test cases, and give the minimum capability needed to pass them.
2. Choose what must change
A RAG system ingests sources, splits them into retrievable units, indexes them, retrieves likely matches, and adds those passages to the model input. Google’s RAG overview describes that sequence as ingestion, transformation, embedding, indexing, retrieval, and generation. Updating a policy means updating the index, not retraining model weights.
A tool definition tells the model the name and input shape of an application function. The model returns a proposed call. Your application validates it, authorizes the user, executes code, and returns the result for the next model response. The model does not execute your function. OpenAI’s function-calling guide describes this five-step loop.
Supervised fine-tuning trains on task-specific examples. Google’s tuning guide recommends starting with prompting and considering tuning when prompting cannot fix recurrent errors and you have high-quality labeled data—roughly 100 examples or more as its rule of thumb.
OpenAI’s current model-optimization guide starts with evals, then prompting, and only then fine-tuning for some use cases. It also says OpenAI is winding down its fine-tuning platform for new users. Provider availability is an architecture constraint, so use a provider and model that currently offer the method you need.
3. Use a support request as the test
“What is the warranty for product A?” belongs in RAG: retrieve the current warranty policy and product document, then answer with links to those passages. “Where is order 1842?” belongs in a tool: after authenticating the customer, call a read-only order-status function.
“Cancel order 1842” needs a separate write tool, an authorization check, a confirmation step, an idempotency key, and an audit record. A JSON schema validates shape. It does not authenticate the user or authorize the record.
If the assistant repeatedly returns the wrong JSON shape after the prompt, schema, and examples have been optimized, fine-tuning may improve that behavior. It does not replace the order API or make the warranty document current.
Microsoft’s Foundry guidance makes the distinction directly: RAG for private or frequently changing data, fine-tuning for behavior, style, or task performance, and agent tools when retrieval belongs inside an agent workflow.
4. Build retrieval with permissions and evals
Filter the searchable corpus to documents the signed-in user may read before retrieval, or enforce an equivalent authorization boundary in the retrieval layer. A perfect semantic match from another customer’s folder is still a data breach. Keep document ownership and access metadata with every chunk.
For each test question, record the passages that should be found. Measure whether those passages appear in the top results before scoring the final answer. If retrieval missed the source, changing the answer prompt cannot repair it.
Return the source title, stable URL or identifier, and the quoted passage used. Then verify that the claim is actually supported. A model can cite a retrieved document and still overstate it. Google’s RAG guidance notes that wrong or stale source material and model misinterpretation can still produce inaccurate output.
Build a held-out evaluation set from real, sanitized requests. Re-run the same evals after model, prompt, index, or tool changes. Require confirmation and idempotency for consequential write actions.
5. Frequently asked questions
Does RAG stop hallucinations?
No. It can ground an answer in retrieved sources, but the wrong passage can be retrieved, the source can be stale, or the model can misread it. Evaluate retrieval and answer support separately, and allow “not found” as a correct outcome.
Can fine-tuning teach current company facts?
It can affect what a model learned during training, but it is a poor update mechanism for policies and records that change. RAG or a tool can supply current information without another training job.
Is a JSON schema enough to make a tool safe?
No. A schema validates shape. Your application must authenticate the user, authorize the record and action, validate business rules, limit scope, ask for confirmation where appropriate, and log the result.
6. Related articles and next steps
Validate the model, data permissions, evaluation cases, and provider features your chosen design needs.
Continue learning
Bring representative inputs, acceptable outputs, and the systems the feature may read or change. Email OdooMatrix to discuss the next step.