Skip to content

feat(realtime): WebRTC support#8790

Merged
mudler merged 2 commits intomudler:masterfrom
richiejp:feat/webrtc
Mar 13, 2026
Merged

feat(realtime): WebRTC support#8790
mudler merged 2 commits intomudler:masterfrom
richiejp:feat/webrtc

Conversation

@richiejp
Copy link
Collaborator

@richiejp richiejp commented Mar 5, 2026

This adds WebRTC support to the realtime API and converts the Talk page to use realtime WebRTC. This has a huge impact on the existing realtime API and requires the inclusion of large libraries like Pion.

The Opus codec is needed for WebRTC and requires libopus. To keep this separate from the main Go executable and avoid requiring a C toolchain, Opus has been given its own backend.

TODO:

  • Handle barge-in from the client speaker
  • Prevent sending back multiple audio streams at once (assuming that is the problem)
  • Replaced buggy Opus-go with libopus using purego binding to avoid CGO.
  • convert to new UI (rebase on master)

@netlify
Copy link

netlify bot commented Mar 5, 2026

Deploy Preview for localai ready!

Name Link
🔨 Latest commit 42d0e0e
🔍 Latest deploy log https://app.netlify.com/projects/localai/deploys/69b437cae355cf0008819089
😎 Deploy Preview https://deploy-preview-8790--localai.netlify.app
📱 Preview on mobile
Toggle QR Code...

QR Code

Use your smartphone camera to open QR code link.

To edit notification comments on pull requests, go to your Netlify project configuration.

@richiejp richiejp marked this pull request as ready for review March 6, 2026 14:11
@richiejp richiejp force-pushed the feat/webrtc branch 8 times, most recently from eeda607 to 49b1ff1 Compare March 9, 2026 12:13
@richiejp richiejp force-pushed the feat/webrtc branch 9 times, most recently from 1ca365c to f736a6c Compare March 13, 2026 10:15
Signed-off-by: Richard Palethorpe <io@richiejp.com>
Signed-off-by: Richard Palethorpe <io@richiejp.com>
// Maximum WebSocket message size in bytes (10MB) to prevent DoS attacks
maxWebSocketMessageSize = 10 * 1024 * 1024

defaultInstructions = "You are a helpful voice assistant. " +
Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I wonder if this would suite better in the config file?

Copy link
Owner

@mudler mudler left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Great addition!

@mudler mudler added the enhancement New feature or request label Mar 13, 2026
@mudler mudler merged commit f9a850c into mudler:master Mar 13, 2026
35 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants