Choose the voice interaction, measure the complete audio-to-answer path, and test interruptions, misunderstood details, tool results, and human handover.
1. Define one conversation the product should support
Choose a narrow interaction: answer product questions, collect a service request, or check an authorized record. Describe who is speaking, the device or channel, expected languages, and how a person can take over. A website microphone and a phone service have different integration requirements.
Decide which information the product needs to record or retain, and communicate that behavior appropriately to participants. Start with a task whose input and outcome can be reviewed. A voice interface should not make the action policy broader than the equivalent text or application interface.
2. Choose and measure the audio architecture
One pattern transcribes speech to text, runs application or agent logic, then produces spoken audio. A realtime speech model is another possible design, with different service capabilities and interaction behavior. Choose the approach based on the feature and the provider path you can actually support.
Measure the stages the user experiences: recognizing the end of speech, obtaining the transcription or interpretation, fetching relevant data, preparing the answer, and beginning playback. Streaming some output can improve perceived responsiveness, but it does not remove the need for correct tools and a reliable final result.
3. Handle interruptions and unclear details
People pause, correct themselves, speak over playback, and refer to information from earlier turns. Define what happens when the user interrupts: stop audio, preserve or discard pending work, and determine whether a tool action has already started.
Ask for clarification when an important name, amount, date, or identifier is uncertain. Repeat back consequential details where the task calls for confirmation. A speech transcript is not independent evidence that an account number or appointment time was correctly understood.
On small screens, scroll the table sideways to see every column.
| Situation | Expected product behavior | Evidence to inspect |
|---|---|---|
| User interrupts playback | Stop speaking and handle the new turn | Audio state and any pending action |
| Unclear record identifier | Ask a useful clarification | Transcript and authorized lookup |
| Tool service unavailable | Explain the limit or offer handover | Actual tool result and spoken response |
4. Keep tools and handover understandable
Use the same trusted identity, record permissions, and business rules as the rest of the application. Speaking a customer name should not automatically authorize reading that customer’s information. Separate information lookup from scheduling, sending, or updating records.
For human handover, pass the relevant conversation context and the unresolved question without claiming that an unfinished action is complete. If a call ends during execution, preserve the operation state so a later retry does not create another appointment or message.
5. Evaluate with real audio conditions
Test recordings or live sessions that reflect the intended microphones, noise, vocabulary, accents, and speech patterns. Review transcription or interpretation errors, response usefulness, interruption handling, and end-to-end delay. Check whether spoken answers are concise enough to follow.
Include silence, partial input, repeated requests, corrections, and unavailable tools. Track abandonment, clarification, handover, and completion along with model costs. A successful text evaluation is useful evidence for part of the system; it is not a substitute for testing the voice experience itself.
6. Frequently asked questions
Do we need a realtime speech model?
Not always. A speech-to-text, application, and text-to-speech pipeline can be appropriate. Compare the supported features, response time, interaction behavior, and cost of the paths available to your project.
Can the assistant act while the user is speaking?
That requires an explicit design for partial input, interruption, and authorization. Do not treat an incomplete transcript as approval for a consequential action.
What is the most useful latency measurement?
Measure the complete user experience, including speech handling, application lookups, model work, and playback start. Keep stage timings so slow components can be investigated.
7. Related articles and next steps
Bring representative inputs, expected outputs, and the systems involved. Discuss your requirements with OdooMatrix.