Add WebSocket voice player for remote audio listening#711
Closed
Alessandro-Improta wants to merge 2 commits intodanielmiessler:mainfrom
Closed
Add WebSocket voice player for remote audio listening#711Alessandro-Improta wants to merge 2 commits intodanielmiessler:mainfrom
Alessandro-Improta wants to merge 2 commits intodanielmiessler:mainfrom
Conversation
Implements WebSocket-first audio dispatch allowing voice notifications to be heard in a browser when remoting into the server. When WebSocket clients are connected, audio is streamed to the browser; otherwise falls back to local playback. Features: - Token-authenticated WebSocket endpoint (/ws) with secure random tokens - Browser-based audio player UI (/player) with auto-reconnect - WebSocket connection manager tracking active clients - Audio dispatch logic: check for WebSocket clients before local playback - Support for MP3 audio streaming (ElevenLabs TTS) - Exponential backoff reconnection (up to 30s delay) - WebSocket status endpoint (/ws/status) for monitoring Files added: - auth.ts: Cryptographically secure token generation and validation - ws-manager.ts: WebSocket connection tracking and broadcast methods - static/player.html: Browser audio player with Web Audio API Files modified: - server.ts: WebSocket endpoint, audio dispatch logic, startup logging Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
Owner
|
Really interesting concept for remote voice playback! This PR was designed for an earlier architecture. The voice system was restructured in v4.0. If you'd like to revisit this for the current architecture, please see Releases/v4.0.2 for the latest codebase. Remote audio is a compelling use case we'd be interested in supporting. Thanks for the contribution! |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
Part of the immersive experience that PAI users are looking for is interacting with their PAI instance using natural language. Daniel Miessler understands this and built into the project a voice server which allows the DA to speak to it's user. However, this only works if the DA lives on the device the user is using.
In the future it seems most likely that the way our DA will be able to be with us always is for it to live remotely and for our many devices to have access to it. This means that we need to solve the problems caused by our DAs living remotely.
Solution
We need a universal, easy to implement, secure way for audio messages to be shared with all sorts of devices. This solution uses WebSocket which means the client device only needs some sort of browser to receive messages. Almost every device that can connect to the internet has a browser, which allows for near universal compatibility.
PR details
This PR implements WebSocket-first audio dispatch allowing voice notifications to be heard in a browser when remoting into the server. When WebSocket clients are connected, audio is streamed to the browser; otherwise falls back to local playback.
Features:
Files added:
Files modified:
Fixes #721