mtmd : (WIP) add ultravox audio input #13623
Conversation
|
Ok somehow it works magically, the code is still nowhere near finish Tested using first 6 seconds from https://www.youtube.com/watch?v=vP4iY1TtS3s 
|
| #define MINIAUDIO_IMPLEMENTATION | ||
| #define MA_NO_ENCODING | ||
| #define MA_NO_DEVICE_IO | ||
| #define MA_NO_RESOURCE_MANAGER | ||
| #define MA_NO_NODE_GRAPH | ||
| #define MA_NO_ENGINE | ||
| #define MA_NO_GENERATION | ||
| #define MA_API static | ||
| #include "miniaudio.h" |
There was a problem hiding this comment.
@ggerganov I initially use dr_wav.h here, but I struggled to write myself a resampling algorithm to downsample/upsample audio to 16KHz. I ended up using miniaudio.h here which provide decoding wav, mp3, flac, etc while also come with resampling built-in.
However, the caveat is that this single-header library is 3MB of code, and most of the components are disabled upon compilation as you see here.
What do you think about keeping this lib? I think the other components can be useful for TTS, as it allow us to play the generated audio without an external command.
There was a problem hiding this comment.
It's ok to use it - we also switched to miniaudio.h for the whisper.cpp examples: ggml-org/whisper.cpp#2759.
Just make sure it will be used only in the examples/tools and that libmtmd will not depend on it to function.
Supersede #12745
TODO writing more in details:
miniaudio.h, why it's compiled in as separated lib (but static link)?imagetomediaorbuffermtmd_API