Building the spoken interface for Copilot

Hero

Copilot started as something people typed into. But work doesn’t always happen in front of a keyboard. People prepare while commuting, ask questions while walking, and reflect after a meeting on the way back to their desk. They capture thoughts before those thoughts ever become a document, a message, or a task.

Mobile made the gap obvious: the moments where AI could be most useful were often the moments when typing was least natural. My work across Real-Time Voice, Record, and CarPlay helped shape a broader voice strategy for Copilot — moving it from a prompt-based assistant into a spoken, mobile-native interface for work.

The goal was never to add voice as a feature. It was to make voice reliable enough to trust, capable enough to complete work, and useful enough to become a daily habit.

TL;DR

I helped define and design a connected voice system across Copilot on mobile, where each surface solved a different part of the voice journey.

• Real-Time Voice was the foundation — a live conversational interface where people could speak to Copilot naturally, ask follow-ups, and move between voice and other modalities. Voice already had real momentum, with around 58% of usage happening on mobile.
• Record expanded voice from live conversation into durable capture. It gave Copilot a way to hear the work that happens beyond the reach of enterprise data: hallway conversations, field observations, customer discussions, and personal thought dumps.
• CarPlay proved voice could create a new, mobile-native habit. It brought Copilot into the commute, helping people prepare, catch up, and think through work hands-free.

Together, these three efforts reframed voice from an input method into a strategic interaction layer for Copilot.

The story

The work started with a simple product tension. Copilot was getting more capable, but the interaction model was still inherited from chat: type a prompt, wait for an answer, read the response, type again. That model works well on a desktop. It breaks down on a phone.

Mobile work happens in fragments — in the car, between meetings, while walking, while doing something else. In those moments, people aren’t looking for a richer text box. They need a different way to work with AI. Real-Time Voice became the starting point for that shift, giving Copilot a live, spoken interaction model.

But as I worked through usage, feedback, and scenario planning, it became clear that voice wasn’t one feature. It was a system with three layers: Real-Time Voice was the conversational layer, Record was the capture layer, and CarPlay was the environment layer. The design question quietly changed from “how should voice work in Copilot?” to “where does voice make Copilot meaningfully more useful than typing?”

Why this mattered

Voice already had momentum, but the experience wasn’t yet enterprise-ready. People were using it, yet trust was fragile — quality was the blocker, not demand. That framed the whole opportunity: the work had to operate at three levels at once.

Make it reliable. Reduce the failures that quietly erode confidence — dropped connections, interruptions, missing responses, and latency. Make it capable. Move beyond conversation into grounding, response cards, personalisation, mixed modality, and real actions. Make it indispensable. Build scenarios that only make sense on mobile — especially commute and capture.

Once people trust voice, the next question is whether they can actually get work done through it. Voice couldn’t stay a novelty. It had to become a dependable work surface.

What we learned

The biggest learning was that voice isn’t valuable simply because it’s faster. It’s valuable because it changes where work can happen. Across the three surfaces, people were trying to do three different jobs.

With Real-Time Voice, they wanted immediate conversation: get an answer right now, plan their day, and stay on top of inbox and messages without constructing the perfect prompt. With Record, they wanted to preserve context — not to store audio, but to keep spoken work that would otherwise disappear. With CarPlay, they wanted Copilot in a place where screens and keyboards are constrained.

Voice isn’t about speaking instead of typing. It’s about keeping work moving when typing isn’t the right interaction model.

The turning point

The turning point was recognising that the voice portfolio needed three distinct but connected modes — and designing each for its moment rather than forcing one model across all of them.

Real-Time Voice — the conversation mode. The core experience. It needed to feel live, responsive, interruptible, grounded, and capable — Copilot as a spoken assistant, not a text system with audio bolted on.

Record — the capture mode. The deliberate exception to live conversation. Where Real-Time Voice is how people talk to Copilot, Record is how Copilot hears the rest of the work, through on-demand, uninterrupted capture that turns spoken moments into usable context.

CarPlay — the environment mode. This forced design for constrained attention: short responses, audio-first state handling, safe interaction patterns, and clear recovery. Not Copilot on another screen, but Copilot in a completely different context of use.

That was the strategic shift. Real-Time Voice made Copilot conversational. Record made spoken work reusable. CarPlay made Copilot useful when screens disappear.

Design principles

A handful of principles held the three surfaces together.

Voice first, visual assist. Voice leads the interaction; visual UI supports orientation, confirmation, and recovery rather than competing with the conversation.

Capable, not just conversational. A voice experience that only talks eventually feels shallow. People wanted to act — draft mail, create tasks, block time, accept or decline a meeting — hands-free, in the moment.

Design for the moment, not the modality. Active conversation, post-conversation capture, and driving are three different contexts. The right voice experience depends on the moment, not a single template.

Ground voice in work. Copilot’s advantage was never the most entertaining assistant voice — it was work-grounding. That pushed decisions toward mail, calendar, files, tasks, and genuine work preparation.

Build for continuity. People wanted transcripts, memory, written output, and the ability to resume. Real-Time Voice shouldn’t feel like a dead-end call, Record should create durable output, and CarPlay should support handoff beyond the car.

Trust is the first feature. Voice failure is far more noticeable than text failure. When voice drops, cuts off, mishears, or returns dead air, confidence collapses instantly — so recovery had to be designed, not assumed.

Design decisions

Real-Time Voice — the live conversation layer. I treated it as a conversation surface, not a voice-input overlay. That meant a clear state model for listening, thinking, speaking, reconnecting, and error handling; the ability to move from speaking into typed follow-up so voice was never a dead end; and mixed modality so voice worked alongside text, files, photos, and live vision. I pushed answers beyond generic replies into work-grounded responses, added visual response cards where audio alone wasn’t enough, and designed explicit confirm-or-deny moments for high-risk actions like sending mail or changing a calendar event. Zero-state prompts helped people understand what voice could do before they knew what to ask.

Record — the capture layer. This wasn’t “Voice Notes 2.0.” I reframed Record as an input, positioning it alongside camera, files, and text as a way for Copilot to receive spoken context that never existed in enterprise data. It supported long-form, uninterrupted capture for conversations and field notes; turned that spoken input into usable chat context rather than stored audio; and prioritised summaries, key points, and actions over raw playback. Upload, processing, and recovery states were treated as core to the experience, and outputs were designed with future recall in mind — so recordings didn’t become another scattered pile.

CarPlay — the environment layer. Not a smaller version of mobile voice, but a different environment with different constraints: safety, glanceability, audio clarity, and limited attention. I prioritised commute-first jobs — meeting prep, catch-up, latest updates — over broad chat exploration; kept the visual output minimal and high-contrast for driving; and leaned on audio cues for state transitions and recovery. Clear session-end patterns told people when Copilot had stopped listening, fast re-entry surfaced recent prompts, and legibility was treated as a functional requirement, not a polish detail.

Building to learn

Voice can’t be fully designed in static screens. The important parts happen between screens — latency, turn-taking, audio state, interruptions, reconnecting, background behaviour, hands-free invocation, transcript generation, and multimodal handoff. So I used code as a design tool.

I built working coded prototypes for Real-Time Voice and Record using real voice input rather than dummy flows, which let us validate the multimodal experience at a much higher fidelity. I also contributed directly to the production mobile codebase — resolving quality bugs and adding dictate functionality to the shared iOS foundation.

Each surface asked a different question. For Real-Time Voice: how should live voice states feel, and what happens when a session fails or needs a typed follow-up? For Record: how does a recording become transcript, summary, and future recall — and how do upload and processing states build trust? For CarPlay: how does Copilot behave under driving constraints, and what must be audio-only? The principle was simple — design didn’t stop at handoff. It continued through prototyping, implementation, and quality review.

Outcomes

Real-Time Voice became the foundation for Copilot’s spoken interface, with around 58% of usage on mobile — confirming that voice had real momentum and that mobile was a major growth lever. Moving to a real-time connection also improved several quality dimensions, lowering interruption rates and dissatisfaction during calls.

Record validated “audio as context” as a meaningful mobile bet. Research showed 62% of information workers already record voice notes, 78% would adopt a one-tap recorder with automatic transcript and summary, and early product-market-fit signals came in above the benchmark that usually predicts durable demand.

CarPlay validated the commute as a voice-first Copilot scenario, earning real recurring usage during its rollout — including people describing hours of conversation with Copilot on the drive, from redesigning a deck to thinking through strategy and business planning.

Taken together, the work moved Copilot voice from a capability into a system: Real-Time Voice proved people wanted live conversational AI for work, Record proved spoken context could become durable grounding, and CarPlay proved voice could create engagement outside traditional productivity moments.

Looking back

The biggest lesson was that voice isn’t just another input method — it changes the shape of the product. A prompt box asks people to stop, think, type, and wait. Voice lets them stay in motion.

Real-Time Voice made Copilot conversational. Record made spoken work reusable. CarPlay made Copilot useful when screens and keyboards were constrained.

Together, they shifted the story from “people can speak to Copilot” to “Copilot can participate in the flow of work, even when work happens away from the screen.” That’s the real product shift.

What’s next

The next phase is moving from voice access to voice agency: better reliability, richer work grounding, stronger action completion, smoother handoff between voice and text, more capable response cards, and eventually proactive assistance that understands context before a single word is typed.

Record gives Copilot memory. CarPlay gives Copilot presence. Real-Time Voice gives Copilot conversation. Together they point toward a future where Copilot isn’t only something people type into, but something they can talk to, think with, and carry through the day.