Skip to content

Commit cd25f0e

Browse files
Danielku15Danielku15
authored
feat: YouTube Sync and CodeGen+Export (#134)
1 parent 601bd35 commit cd25f0e

File tree

14 files changed

+1382
-284
lines changed

14 files changed

+1382
-284
lines changed

‎docs/guides/audio-video-sync.mdx‎

Lines changed: 48 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,48 @@
1+
---
2+
title: Audio & Video Sync
3+
---
4+
5+
> [!WARNING]
6+
> This guide is still incomplete and is currently being extended.
7+
8+
alphaTab can be synchronized with an external audio or video backing track. You can either use Guitar Pro 8 files with an audio track or synchronize alphaTab with an external media using the
9+
[Media Sync Editor in the Playground](/docs/playground/playground.mdx).
10+
11+
At this point alphaTab cannot mix the synthesized audio and a backing track together, therefore only either audio will be heard. Using the synchronization information embedded in the data model,
12+
alphaTab can then place the cursor correctly as an external media is taking over the audio.
13+
14+
### Audio Backing Track
15+
16+
alphaTab has built-in support for audio backing tracks. To play alphaTab with an audio backing track instead of synthesized audio, following prerequisites have to be fulfilled:
17+
18+
1. The data model ships the backing track audio file.
19+
2. The data model ships sync points which aligns the external audio track with the music notation.
20+
3. You enabled the backing track via either `PlayerMode.EnabledAutomatic` or `PlayerMode.EnabledBackingTrack`.
21+
4. Your platform (browser, operating system,...) has to support the embedded audio file. OGG Vorbis and MP3s are quite widely supported, and if used, there shouldn't be major problems.
22+
23+
To tackle 1. and 2. you can use the Media Sync Editor we provide on our [Playground](/docs/playground/playground.mdx). Learn more about the editor [here](./media-sync-editor.mdx)
24+
25+
### Guitar Pro 8 Files
26+
27+
Since Version 8 Guitar Pro provides built-in audio tracks with the possibility to synchronize the backing track with the music sheet. If the audio file is embedded into the GP file, alphaTab can load and use
28+
both the audio file and the sync point information.
29+
30+
As mentioned above, alphaTab can only play **either** the external audio file **or** the synthesized audio. Also we do not support changes in the audio (like adjusting the pitch).
31+
32+
Beside that you can load Guitar Pro 8 files and directly benefit from the enhanced sound experience.
33+
34+
### Custom External Media Player
35+
36+
alphaTab can be integrated with any external media system but it requires some implementation on the integrator side. To properly synchronize alphaTab and an external media source (audio or video) the `alphaTab.synth.IExternalMediaHandler` interface has to be implemented and provided to alphaTab.
37+
38+
#### YouTube
39+
40+
YouTube is a great counterpart to alphaTab to provide audio and video for your music sheet. Until now we decided to NOT ship a direct YouTube integration as there are quite some implications to it.
41+
Some key reasons behind this is:
42+
43+
1. alphaTab is a cross-platform toolkit. While integrating a YouTube player in the web is easy, SDKs for Android or Desktop platforms are way more complex. We want to keep this on the integrator side.
44+
2. Looking at data protection laws like GDPR, users have to accept the load and integration of such external media. We want to be sure devs take proper care asking users for consent where required.
45+
3. The integration of the YouTube player requires further considerations in your user interface (sizing, where to place it, how to configure the YouTube player).
46+
47+
Nevertheless we want to give you some guidance on how to link alphaTab to YouTube. The following steps show how to use the [YouTube Player API Reference for iframe Embeds](https://developers.google.com/youtube/iframe_api_reference) together with alphaTab.
48+

‎docs/guides/media-sync-editor.mdx‎

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,8 @@
1+
---
2+
title: Media Sync Editor
3+
---
4+
5+
This guide is about the Media Sync Editor we provide as part of our [Playground](/docs/playground/playground.mdx).
6+
7+
> [!WARNING]
8+
> This guide is still incomplete and is currently being extended.

‎docs/playground/playground.mdx‎

Lines changed: 1 addition & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -8,11 +8,4 @@ import { AlphaTabPlayground } from "@site/src/components/AlphaTabPlayground";
88
<body className="playground" />
99
</head>
1010

11-
<AlphaTabPlayground
12-
settings={{
13-
core: {
14-
file: "/files/canon-full.gp",
15-
tracks: [0,1],
16-
},
17-
}}
18-
/>
11+
<AlphaTabPlayground />

‎src/components/AlphaTabPlayground/helpers.ts‎

Lines changed: 57 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -7,6 +7,13 @@ export function timePositionToX(pixelPerMilliseconds: number,
77
return timePosition * zoomedPixelPerMilliseconds + leftPadding;
88
}
99

10+
export function xToTimePosition(pixelPerMilliseconds: number,
11+
x: number, zoom: number, leftPadding: number): number {
12+
13+
const zoomedPixelPerMilliseconds = pixelPerMilliseconds * zoom;
14+
return (x - leftPadding) / zoomedPixelPerMilliseconds;
15+
}
16+
1017
type UndoStack = {
1118
undo: SyncPointInfo[];
1219
redo: SyncPointInfo[];
@@ -60,4 +67,53 @@ export const useSyncPointInfoUndo = () => {
6067
});
6168
}
6269
}
63-
};
70+
};
71+
72+
export type HTMLMediaElementLikeEvents = 'timeupdate' | 'durationchange' | 'seeked' | 'play' | 'pause' | 'ended' | 'volumechange' | 'ratechange' | 'loadedmetadata';
73+
74+
export interface HTMLMediaElementLike {
75+
currentTime: number;
76+
volume: number;
77+
playbackRate: number;
78+
readonly duration: number;
79+
addEventListener(eventType: HTMLMediaElementLikeEvents, handler: () => void): void;
80+
removeEventListener(eventType: HTMLMediaElementLikeEvents, handler: () => void): void;
81+
82+
play(): void;
83+
pause(): void;
84+
}
85+
86+
export function extractYouTubeVideoId(src: string | undefined) {
87+
try {
88+
if (!src) {
89+
return undefined;
90+
}
91+
const url = new URL(src);
92+
const host = url.host.toLowerCase();
93+
if (host.endsWith('youtube.com') && url.searchParams.has('v')) {
94+
return url.searchParams.get('v')!;
95+
}
96+
if (host.endsWith('youtube-nocookie.com')) {
97+
return url.pathname.split('/')[2];
98+
}
99+
if (host.endsWith('youtu.be')) {
100+
return url.pathname.split('/')[1];
101+
}
102+
} catch (e) {
103+
return undefined;
104+
}
105+
}
106+
107+
108+
export enum MediaType {
109+
Synth = 0,
110+
Audio = 1,
111+
YouTube = 2
112+
}
113+
114+
export interface MediaTypeState {
115+
type: MediaType;
116+
audioFile?: Uint8Array
117+
youtubeUrl?: string;
118+
youtubeVideoDuration?: number;
119+
}

‎src/components/AlphaTabPlayground/index.tsx‎

Lines changed: 162 additions & 11 deletions
Original file line numberDiff line numberDiff line change
@@ -1,7 +1,7 @@
11
'use client';
22

33
import * as alphaTab from '@coderline/alphatab';
4-
import React, { useState } from 'react';
4+
import React, { useCallback, useEffect, useRef, useState } from 'react';
55
import { useAlphaTab, useAlphaTabEvent } from '@site/src/hooks';
66
import styles from './styles.module.scss';
77
import { FontAwesomeIcon } from '@fortawesome/react-fontawesome';
@@ -12,25 +12,26 @@ import { PlaygroundSettings } from './playground-settings';
1212
import { Tooltip } from 'react-tooltip';
1313
import { PlaygroundTrackSelector } from './track-selector';
1414
import { MediaSyncEditor } from './media-sync-editor';
15+
import { type HTMLMediaElementLike, MediaType, type MediaTypeState } from './helpers';
16+
import { YouTubePlayer } from './youtube-player';
1517

16-
interface AlphaTabPlaygroundProps {
17-
settings?: alphaTab.json.SettingsJson;
18-
}
19-
20-
export const AlphaTabPlayground: React.FC<AlphaTabPlaygroundProps> = ({ settings }) => {
18+
export const AlphaTabPlayground: React.FC = () => {
2119
const viewPortRef = React.createRef<HTMLDivElement>();
2220
const [isLoading, setLoading] = useState(true);
2321
const [sidePanel, setSidePanel] = useState(SidePanel.None);
2422
const [bottomPanel, setBottomPanel] = useState(BottomPanel.None);
23+
const [mediaType, setMediaType] = useState<MediaTypeState>({
24+
type: MediaType.Synth
25+
});
26+
const youtubePlayer = useRef<HTMLMediaElementLike | null>(null);
2527

2628
const [api, element] = useAlphaTab(s => {
2729
s.core.engine = 'svg';
30+
s.core.file = '/files/canon-full.gp';
31+
s.core.tracks = [0, 1];
2832
s.player.scrollElement = viewPortRef.current!;
2933
s.player.scrollOffsetY = -10;
30-
s.player.playerMode = alphaTab.PlayerMode.EnabledAutomatic;
31-
if (settings) {
32-
s.fillFromJson(settings);
33-
}
34+
s.player.playerMode = alphaTab.PlayerMode.EnabledSynthesizer;
3435
});
3536

3637
useAlphaTabEvent(api, 'renderFinished', () => {
@@ -56,6 +57,144 @@ export const AlphaTabPlayground: React.FC<AlphaTabPlaygroundProps> = ({ settings
5657
}
5758
};
5859

60+
const youtubePlayerUnsubscribe = useRef<() => void>(null);
61+
const setYoutubePlayer = useCallback(
62+
(newPlayer: HTMLMediaElementLike) => {
63+
if (youtubePlayerUnsubscribe.current) {
64+
youtubePlayerUnsubscribe.current();
65+
youtubePlayerUnsubscribe.current = null;
66+
}
67+
68+
if (newPlayer && api) {
69+
youtubePlayer.current = newPlayer;
70+
71+
const onLoadedMetadata = () => {
72+
setMediaType(t => ({
73+
...t,
74+
youtubeVideoDuration: newPlayer.duration * 1000
75+
}));
76+
};
77+
const onTimeUpdate = () => {
78+
if (api!.actualPlayerMode === alphaTab.PlayerMode.EnabledExternalMedia) {
79+
(api!.player!.output as alphaTab.synth.IExternalMediaSynthOutput).updatePosition(
80+
newPlayer.currentTime * 1000
81+
);
82+
}
83+
};
84+
85+
const onPlay = () => {
86+
api.play();
87+
};
88+
const onPause = () => {
89+
api.pause();
90+
};
91+
92+
const onEnded = () => {
93+
api.pause();
94+
};
95+
96+
const onVolumeChange = () => {
97+
api.masterVolume = newPlayer.volume;
98+
};
99+
100+
const onRateChange = () => {
101+
api.playbackSpeed = newPlayer.playbackRate;
102+
};
103+
104+
newPlayer.addEventListener('loadedmetadata', onLoadedMetadata);
105+
newPlayer.addEventListener('timeupdate', onTimeUpdate);
106+
newPlayer.addEventListener('seeked', onTimeUpdate);
107+
newPlayer.addEventListener('play', onPlay);
108+
newPlayer.addEventListener('pause', onPause);
109+
newPlayer.addEventListener('ended', onEnded);
110+
newPlayer.addEventListener('volumechange', onVolumeChange);
111+
newPlayer.addEventListener('ratechange', onRateChange);
112+
newPlayer.addEventListener('volumechange', onVolumeChange);
113+
newPlayer.addEventListener('ratechange', onRateChange);
114+
115+
youtubePlayerUnsubscribe.current = () => {
116+
newPlayer.removeEventListener('loadedmetadata', onLoadedMetadata);
117+
newPlayer.removeEventListener('timeupdate', onTimeUpdate);
118+
newPlayer.removeEventListener('seeked', onTimeUpdate);
119+
newPlayer.removeEventListener('play', onPlay);
120+
newPlayer.removeEventListener('pause', onPause);
121+
newPlayer.removeEventListener('ended', onEnded);
122+
newPlayer.removeEventListener('volumechange', onVolumeChange);
123+
newPlayer.removeEventListener('ratechange', onRateChange);
124+
newPlayer.removeEventListener('volumechange', onVolumeChange);
125+
newPlayer.removeEventListener('ratechange', onRateChange);
126+
};
127+
}
128+
},
129+
[api]
130+
);
131+
132+
useEffect(() => {
133+
if (!api) {
134+
return;
135+
}
136+
137+
api.pause();
138+
switch (mediaType.type) {
139+
case MediaType.Synth:
140+
api.settings.player.playerMode = alphaTab.PlayerMode.EnabledSynthesizer;
141+
api.updateSettings();
142+
break;
143+
144+
case MediaType.Audio:
145+
api.settings.player.playerMode = alphaTab.PlayerMode.EnabledBackingTrack;
146+
api.updateSettings();
147+
148+
break;
149+
150+
case MediaType.YouTube:
151+
api.settings.player.playerMode = alphaTab.PlayerMode.EnabledExternalMedia;
152+
api.updateSettings();
153+
154+
const handler: alphaTab.synth.IExternalMediaHandler = {
155+
get backingTrackDuration() {
156+
const duration = youtubePlayer.current?.duration ?? 0;
157+
return Number.isFinite(duration) ? duration * 1000 : 0;
158+
},
159+
get playbackRate() {
160+
return youtubePlayer.current?.duration ?? 1;
161+
},
162+
set playbackRate(value) {
163+
if (youtubePlayer.current) {
164+
youtubePlayer.current.playbackRate = value;
165+
}
166+
},
167+
get masterVolume() {
168+
return youtubePlayer.current?.volume ?? 1;
169+
},
170+
set masterVolume(value) {
171+
if (youtubePlayer.current) {
172+
youtubePlayer.current.volume = value;
173+
}
174+
},
175+
seekTo(time) {
176+
if (youtubePlayer.current) {
177+
youtubePlayer.current.currentTime = time / 1000;
178+
}
179+
},
180+
play() {
181+
if (youtubePlayer.current) {
182+
youtubePlayer.current.play();
183+
}
184+
},
185+
pause() {
186+
if (youtubePlayer.current) {
187+
youtubePlayer.current.pause();
188+
}
189+
}
190+
};
191+
192+
(api.player!.output as alphaTab.synth.IExternalMediaSynthOutput).handler = handler;
193+
194+
break;
195+
}
196+
}, [api, mediaType]);
197+
59198
return (
60199
<>
61200
<div className={styles['at-wrap']} onDragOver={onDragOver} onDrop={onDrop}>
@@ -87,11 +226,23 @@ export const AlphaTabPlayground: React.FC<AlphaTabPlaygroundProps> = ({ settings
87226
<div className={styles['at-viewport']} ref={viewPortRef}>
88227
<div ref={element} />
89228
</div>
229+
230+
{mediaType.type === MediaType.YouTube && (
231+
<div className={styles.video}>
232+
<YouTubePlayer ref={setYoutubePlayer} src={mediaType.youtubeUrl!} />
233+
</div>
234+
)}
90235
</div>
91236

92237
<div className={styles['at-footer']}>
93238
{api && api?.score && bottomPanel === BottomPanel.MediaSyncEditor && (
94-
<MediaSyncEditor api={api} score={api!.score} />
239+
<MediaSyncEditor
240+
api={api}
241+
score={api!.score}
242+
mediaType={mediaType}
243+
onMediaTypeChange={t => setMediaType(t)}
244+
youtubePlayer={youtubePlayer.current ?? undefined}
245+
/>
95246
)}
96247
{api && (
97248
<PlayerControlsGroup

0 commit comments

Comments
 (0)