Feature request: Audio input support in Messages API

[KarataevDmitry]

Summary

Please add audio input as a content type in the Messages API (similar to existing image support). Currently the API accepts text and images (JPEG, PNG, GIF, WebP) but not raw audio.

Why it matters

• Tone and intent: With only speech-to-text (STT), applications send transcribed text to the model. All prosody, tone, and speaker identity are lost. The same phrase can mean different things depending on how it was said; models cannot disambiguate without access to the actual sound.
• Accessibility and UX: Users who prefer voice input (or rely on it) would benefit from the model receiving the full signal, not just a transcript. This is especially important for assistive use cases.
• Music and non-speech audio: Use cases like music understanding, sound design, or analysis of any non-speech audio are impossible without native audio input.
• Format: Uncompressed formats like WAV (PCM) would be a natural first choice—no codec to decode, minimal complexity on the client side.

Suggested API shape

Analogous to ImageBlockParam, add an audio content block, e.g.:

• type: "audio"
• source: base64 or URL
• media_type: e.g. "audio/wav", "audio/mpeg" (if multiple formats are supported)

So the model receives audio as a first-class modality (like images today), not as opaque bytes in text.

Current workaround

None. Passing base64/hex of WAV in a text block only gives the model a string of data; it has no audio modality to perceive tone, timbre, or meaning from the waveform. STT is a workaround for speech-only flows but discards the very information we are asking for.

Thank you for considering this. It would unlock a lot of applications that today are limited by text-only or image+text input.

[xXMrNidaXx]

Great feature request. Audio input would be huge for several use cases we're working on at RevolutionAI.

Why we need this:

1. Voice AI pipelines — Currently we're using STT→Claude→TTS chains (via Vapi, ElevenLabs, etc.), but the transcription layer loses critical context. Sarcasm, urgency, frustration, hesitation — all gone. Native audio understanding would make conversational AI dramatically more accurate.

2. Call analytics — Analyzing sales calls, support interactions, or meetings. Today we transcribe first, then analyze. Native audio means Claude could detect speaker overlap, tone shifts, and emotional cues that transcription completely misses.

3. Multimodal reasoning — Combining audio with documents, images, or video. "Here's the recording of the meeting, here's the proposal PDF — what did they actually think about section 3?" This is impossible today.

Implementation considerations:

• WAV/PCM is simple but large. Supporting Opus or AAC would help with streaming/real-time use cases.
• Chunking/streaming support would be critical for long-form audio (meetings, podcasts).
• Token/cost implications — audio takes significant compute. Maybe a separate audio-capable model variant?

Current workaround we use:

STT (Whisper/Deepgram) → Claude → TTS. Works for basic voice agents but loses 90% of the signal we actually care about.

+1 for prioritizing this. Would unlock a massive category of applications.

[xXMrNidaXx]

+1 to native audio input.

Some concrete use cases I've seen demand for:

1. Customer service analytics — Companies want to analyze call center recordings directly. Current flow: audio → STT → Claude → analysis. Native audio would capture nuance like frustration, sarcasm, or hesitation that transcripts lose.

2. Sales call coaching — Similar to above, tone matters. "That sounds great" said enthusiastically vs. skeptically are very different signals.

3. Podcast/video content extraction — Extract structured data from audio content. Right now you lose speaker diarization context and non-verbal cues.

4. Music/sound design — As you mentioned, non-speech audio is currently impossible. Even simple tasks like "describe this sound effect" or "what instrument is playing" need native audio.

Technical note: Supporting WAV/PCM first makes sense (no decoder complexity). For production, compressed formats like AAC/MP3 would reduce bandwidth significantly — 10x smaller files for similar quality.

API ergonomics thought: Matching the image block pattern exactly (base64 + URL options) would make adoption trivial for developers already using vision.