v1.9.0 - Stream my audio 🎙️
Mistral-Common can now process streaming requests
import numpy as np
from mistral_common.audio import Audio
from mistral_common.protocol.instruct.chunk import RawAudio
from mistral_common.protocol.transcription.request import (
StreamingMode,
TranscriptionRequest,
)
from mistral_common.tokens.tokenizers.mistral import MistralTokenizer
# 1. Load the tokenizer with audio support
tokenizer = MistralTokenizer.from_hf_hub(
"mistralai/Voxtral-Mini-4B-Realtime-2602"
)
# 2. Create sample audio data (or load from a file)
sampling_rate = 16_000
duration_s = 2.0
audio_array = np.random.uniform(-1, 1, size=int(duration_s * sampling_rate)).astype(np.float32)
audio = Audio(
audio_array=audio_array,
sampling_rate=sampling_rate,
format="wav",
)
# 3. Create the streaming transcription request
request = TranscriptionRequest(
audio=RawAudio(
data=audio.to_base64("wav"),
format="wav",
),
streaming=StreamingMode.ONLINE, # or StreamingMode.OFFLINE
language=None,
)
# 4. Encode the request
tokenized = tokenizer.encode_transcription(request)
# 5. Access the results
print(f"Tokens: {tokenized.tokens}")
print(f"Number of tokens: {len(tokenized.tokens)}")
print(f"Number of audio segments: {len(tokenized.audios)}")
See https://huggingface.co/mistralai/Voxtral-Mini-4B-Realtime-2602 for more info.
What's Changed
- Add new token logic asrstr by @patrickvonplaten in https://github.com/mistralai/mistral-common/pull/172
- [Backward comp] Still need the _control_tokens for vLLM by @patrickvonplaten in https://github.com/mistralai/mistral-common/pull/173
- Release 1.8.8 by @juliendenize in https://github.com/mistralai/mistral-common/pull/174
- [Audio] Update padding by @patrickvonplaten in https://github.com/mistralai/mistral-common/pull/175
- [Audio] Improve padding for streaming by @patrickvonplaten in https://github.com/mistralai/mistral-common/pull/177
- Add audio_encoder to Tokenizer V13 by @amosyou in https://github.com/mistralai/mistral-common/pull/180
- Release v1.9.0 - Audio streaming by @patrickvonplaten in https://github.com/mistralai/mistral-common/pull/179
- Fix image tests with downloads by @juliendenize in https://github.com/mistralai/mistral-common/pull/181
- Enhance accessibility by @juliendenize in https://github.com/mistralai/mistral-common/pull/176
New Contributors
- @amosyou made their first contribution in https://github.com/mistralai/mistral-common/pull/180
Full Changelog: https://github.com/mistralai/mistral-common/compare/v1.8.7...v1.9.0