modelslab-audio-generation

ModelsLab 音频生成

使用 AI 生成高品质音频，包括语音、音乐、声音转换、音效和配音。

适用场景

• 将文本转换为自然流畅的语音（TTS）
• 将语音转录为文本
• 转换声音特征（语音到语音）
• 根据文本提示词生成音乐
• 创建音效
• 将音频配制成不同语言
• 延长或修补歌曲
• 构建语音助手或有声读物

可用 API（v7）

语音端点

• 文本转语音：POST https://modelslab.com/api/v7/voice/text-to-speech
• 语音转文本：POST https://modelslab.com/api/v7/voice/speech-to-text
• 语音到语音：POST https://modelslab.com/api/v7/voice/speech-to-speech
• 音乐生成：POST https://modelslab.com/api/v7/voice/music-gen
• 声音生成：POST https://modelslab.com/api/v7/voice/sound-generation
• 创建配音：POST https://modelslab.com/api/v7/voice/create-dubbing
• 歌曲延长：POST https://modelslab.com/api/v7/voice/song-extender
• 歌曲修补：POST https://modelslab.com/api/v7/voice/song-inpaint
• 获取结果：POST https://modelslab.com/api/v7/voice/fetch/{id}

注意：v6 端点（/api/v6/voice/text_to_speech 等）仍然可用，但 v7 是当前版本。v7 中的参数名称有所变化（例如，text 现在为 prompt，audio 现在为 init_audio）。

发现音频模型

# Search audio/voice models
modelslab models search --feature audio_gen

# Search by provider
modelslab models search --search "eleven"

# Get model details
modelslab models detail --id eleven_multilingual_v2

音频模型 ID

model_id	名称	适用场景
eleven_multilingual_v2	ElevenLabs Multilingual v2	文本转语音
eleven_english_sts_v2	ElevenLabs Voice Changer	语音到语音
scribe_v1	ElevenLabs Scribe	语音转文本
eleven_sound_effect	ElevenLabs Sound Effects	声音生成
music_v1	ElevenLabs Music	音乐生成
inworld-tts-1	Inworld TTS	文本转语音

文本转语音

import requests
import time

def text_to_speech(text, api_key, voice_id="21m00Tcm4TlvDq8ikWAM", model_id="eleven_multilingual_v2"):
    """Convert text to speech.

    Args:
        text: The text to convert to speech
        api_key: Your ModelsLab API key
        voice_id: ElevenLabs voice ID (see Available Voices below)
        model_id: TTS model to use
    """
    response = requests.post(
        "https://modelslab.com/api/v7/voice/text-to-speech",
        json={
            "key": api_key,
            "prompt": text,             # v7 uses "prompt" not "text"
            "voice_id": voice_id,
            "model_id": model_id
        }
    )

    data = response.json()

    if data["status"] == "success":
        return data["output"][0]
    elif data["status"] == "processing":
        return poll_audio_result(data["id"], api_key)
    else:
        raise Exception(f"Error: {data.get('message', 'Unknown error')}")

# Usage
audio_url = text_to_speech(
    "Hello! Welcome to ModelsLab. This is a test of our text-to-speech API.",
    "your_api_key"
)
print(f"Audio URL: {audio_url}")

语音转文本（转录）

def speech_to_text(audio_url, api_key, model_id="scribe_v1"):
    """Transcribe speech from audio to text.

    Args:
        audio_url: URL of audio file (must be publicly accessible)
        model_id: STT model to use
    """
    response = requests.post(
        "https://modelslab.com/api/v7/voice/speech-to-text",
        json={
            "key": api_key,
            "init_audio": audio_url,    # v7 uses "init_audio" not "audio"
            "model_id": model_id
        }
    )

    data = response.json()

    if data["status"] == "success":
        return data["output"][0]
    elif data["status"] == "processing":
        return poll_audio_result(data["id"], api_key)
    else:
        raise Exception(data.get("message"))

# Transcribe audio
result = speech_to_text(
    "https://example.com/speech.mp3",
    "your_api_key"
)
print(f"Transcription: {result}")

语音到语音（声音转换）

def speech_to_speech(audio_url, voice_id, api_key, model_id="eleven_english_sts_v2"):
    """Convert voice characteristics in audio.

    Args:
        audio_url: URL of the source audio
        voice_id: Target ElevenLabs voice ID
        model_id: Voice conversion model
    """
    response = requests.post(
        "https://modelslab.com/api/v7/voice/speech-to-speech",
        json={
            "key": api_key,
            "init_audio": audio_url,
            "voice_id": voice_id,
            "model_id": model_id
        }
    )

    data = response.json()
    if data["status"] == "success":
        return data["output"][0]
    elif data["status"] == "processing":
        return poll_audio_result(data["id"], api_key)

音效生成

def generate_sound_effect(description, api_key, model_id="eleven_sound_effect"):
    """Generate a sound effect from a text description.

    Args:
        description: What sound to generate
        model_id: Sound effects model
    """
    response = requests.post(
        "https://modelslab.com/api/v7/voice/sound-generation",
        json={
            "key": api_key,
            "prompt": description,
            "model_id": model_id
        }
    )

    data = response.json()
    if data["status"] == "success":
        return data["output"][0]
    elif data["status"] == "processing":
        return poll_audio_result(data["id"], api_key)

# Generate door slam sound
sfx_url = generate_sound_effect(
    "Heavy wooden door slamming shut",
    "your_api_key"
)

音乐生成

def generate_music(prompt, api_key, model_id="music_v1"):
    """Generate music from a text description.

    Args:
        prompt: Description of music style/mood
        model_id: Music generation model
    """
    response = requests.post(
        "https://modelslab.com/api/v7/voice/music-gen",
        json={
            "key": api_key,
            "prompt": prompt,
            "model_id": model_id
        }
    )

    data = response.json()
    if data["status"] == "success":
        return data["output"][0]
    elif data["status"] == "processing":
        return poll_audio_result(data["id"], api_key)

# Generate background music
music_url = generate_music(
    "Upbeat electronic music with a driving beat, perfect for a tech startup video",
    "your_api_key"
)
print(f"Music: {music_url}")

轮询异步结果

def poll_audio_result(request_id, api_key, timeout=300):
    """Poll for async audio generation results."""
    start_time = time.time()

    while time.time() - start_time < timeout:
        fetch = requests.post(
            f"https://modelslab.com/api/v7/voice/fetch/{request_id}",
            json={"key": api_key}
        )
        result = fetch.json()

        if result["status"] == "success":
            return result["output"][0]
        elif result["status"] == "failed":
            raise Exception(result.get("message", "Generation failed"))

        time.sleep(5)

    raise Exception("Timeout waiting for audio generation")

可用的 ElevenLabs 声音 ID

声音 ID	名称	风格
21m00Tcm4TlvDq8ikWAM	Rachel	中性、平静
AZnzlk1XvdvUeBnXmlld	Domi	自信
EXAVITQu4vr4xnSDxMaL	Bella	柔和、温暖
ErXwobaYiN019PkySvjV	Antoni	全面均衡
MF3mGyEYCl7XYWbV9V6O	Elli	年轻、清晰
TxGEqnHWrfWFTfGW9XjX	Josh	低沉、温暖
VR6AewLTigWG4xSOukaG	Arnold	有力
pNInz6obpgDQGcFmaJgB	Adam	低沉、叙事感
yoZ06aMxZJJ28mfd3POQ	Sam	富有活力

关键参数

文本转语音

参数	类型	必填	描述
prompt	string	是	要转换为语音的文本
voice_id	string	是	ElevenLabs 声音标识符
model_id	string	是	TTS 模型（例如，eleven_multilingual_v2）
temperature	float	否	声音变化程度
webhook	string	否	异步通知 URL

语音转文本

参数	类型	必填	描述
init_audio	string	是	要转录的音频 URL
model_id	string	是	STT 模型（例如，scribe_v1）

声音生成

参数	类型	必填	描述
prompt	string	是	音效描述
model_id	string	是	SFX 模型（例如，eleven_sound_effect）

v6 到 v7 的参数变化

v6 参数	v7 参数	说明
text	prompt	TTS 文本输入
audio	init_audio	STT/STS 音频输入
target_audio	init_audio	声音转换的源音频
（非必填）	model_id	现在所有端点都必填

最佳实践

1. 使用正确的语音 ID

TTS 需要有效的 ElevenLabs 语音 ID（不能使用“alloy”等通用名称）。

2. 确保音频可访问

用于语音转文本的音频 URL 必须无需重定向或身份验证即可公开访问。

3. 对长时间运行的操作使用 Webhook

payload = {
    "key": api_key,
    "prompt": "...",
    "model_id": "eleven_multilingual_v2",
    "webhook": "https://yourserver.com/webhook/audio",
    "track_id": "audio_001"
}

错误处理

try:
    audio = text_to_speech(text, api_key)
    print(f"Audio generated: {audio}")
except Exception as e:
    print(f"Audio generation failed: {e}")

资源

• 音频 API 文档：https://docs.modelslab.com/voice-cloning/overview
• 模型浏览器：https://modelslab.com/models
• 模型选择指南：https://docs.modelslab.com/guides/model-selection
• 获取 API 密钥：https://modelslab.com/dashboard

相关技能

• modelslab-model-discovery - 查找并筛选模型
• modelslab-video-generation - 为视频添加音频
• modelslab-chat-generation - 与 LLM 模型聊天
• modelslab-webhooks - 处理异步音频生成