# Source: https://docs.livekit.io/agents/multimodality/audio.md

LiveKit docs › Multimodality › Speech & audio

---

# Agent speech and audio

> Speech and audio capabilities for LiveKit agents.

## Overview

Speech capabilities are a core feature of LiveKit agents, enabling them to interact with users through voice. This guide covers the various speech features and functionalities available for agents.

LiveKit Agents provide a unified interface for controlling agents using both the STT-LLM-TTS pipeline and realtime models.

To learn more and see usage examples, see the following topics:

- **[Text-to-speech (TTS)](https://docs.livekit.io/agents/models/tts.md)**: TTS is a synthesis process that converts text into audio, giving AI agents a "voice."

- **[Speech-to-speech](https://docs.livekit.io/agents/models/realtime.md)**: Multimodal, realtime APIs can understand speech input and generate speech output directly.

## Instant connect

The instant connect feature reduces perceived connection time by capturing microphone input before the agent connection is established. This pre-connect audio buffer sends speech as context to the agent, avoiding awkward gaps between a user's connection and their ability to interact with an agent.

Microphone capture begins locally while the agent is connecting. Once the connection is established, the speech and metadata is sent over a byte stream with the topic `lk.agent.pre-connect-audio-buffer`. If no agent connects before timeout, the buffer is discarded.

You can enable this feature using `withPreconnectAudio`:

**JavaScript**:

In the Javascript SDK, this functionality is exposed via `TrackPublishOptions`.

```typescript
await room.localParticipant.setMicrophoneEnabled(!enabled, undefined, {
  preConnectBuffer: true,
});

```

---

**Swift**:

```swift
try await room.withPreConnectAudio(timeout: 10) {
  try await room.connect(url: serverURL, token: token)
} onError: { err in
  print("Pre-connect audio send failed:", err)
}

```

---

**Android**:

```kotlin
try {
  room.withPreconnectAudio {
      // Audio is being captured automatically
      // Perform other async setup
      val (url, token) = tokenService.fetchConnectionDetails()
      room.connect(
          url = url,
          token = token,
      )
      room.localParticipant.setMicrophoneEnabled(true)
  }
} catch (e: Throwable) {
  Log.e(TAG, "Error!")
}

```

---

**Flutter**:

```dart
try {
  await room.withPreConnectAudio(() async {
    // Audio is being captured automatically, perform other async setup
    // Get connection details from token service etc.
    final connectionDetails = await tokenService.fetchConnectionDetails();
    await room.connect(
      connectionDetails.serverUrl,
      connectionDetails.participantToken,
    );
    // Mic already enabled
  });
} catch (error) {
  print("Error: $error");
}

```

## Preemptive speech generation

**Preemptive generation** allows the agent to begin generating a response before the user's end of turn is committed. The response is based on partial transcription or early signals from user input, helping reduce perceived response delay and improving conversational flow.

When enabled, the agent starts generating a response as soon as the final transcript is available. If the chat context or tools change in the `on_user_turn_completed` [node](https://docs.livekit.io/agents/build/nodes.md#on_user_turn_completed), the preemptive response is canceled and replaced with a new one based on the final transcript.

This feature reduces latency when the following are true:

- [STT node](https://docs.livekit.io/agents/build/nodes.md#stt_node) returns the final transcript faster than [VAD](https://docs.livekit.io/agents/logic/turns/vad.md) emits the `end_of_speech` event.
- [Turn detection model](https://docs.livekit.io/agents/logic/turns/turn-detector.md) is enabled.

You can enable this feature for STT-LLM-TTS pipeline agents using the `preemptive_generation` parameter for AgentSession:

**Python**:

```python
session = AgentSession(
   preemptive_generation=True,
   ... # STT, LLM, TTS, etc.
)

```

---

**Node.js**:

```typescript
const session = new voice.AgentSession({
    // ... llm, stt, etc.
    voiceOptions: {
      preemptiveGeneration: true,
    },  
});

```

> ℹ️ **Note**
> 
> Preemptive generation doesn't guarantee reduced latency. Use [Agent observability](https://docs.livekit.io/deploy/observability/insights.md) to validate and fine tune agent performance.

## Initiating speech

By default, the agent waits for user input before responding—the Agents framework automatically handles response generation.

In some cases, though, the agent might need to initiate the conversation. For example, it might greet the user at the start of a session or check in after a period of silence.

### session.say

To have the agent speak a predefined message, use `session.say()`. This triggers the configured TTS to synthesize speech and play it back to the user.

You can also optionally provide pre-synthesized audio for playback. This skips the TTS step and reduces response time.

> 💡 **Realtime models and TTS**
> 
> The `say` method requires a TTS plugin. If you're using a realtime model, you need to add a TTS plugin to your session or use the [`generate_reply()`](#manually-interrupt-and-generate-responses) method instead.

**Python**:

```python
await session.say(
   "Hello. How can I help you today?",
   allow_interruptions=False,
)

```

---

**Node.js**:

```typescript
await session.say(
  'Hello. How can I help you today?',
  {
    allowInterruptions: false,
  }
);

```

#### Parameters

You can call `session.say()` with the following options:

- `text` only: Synthesizes speech using TTS, which is added to the transcript and chat context (unless `add_to_chat_ctx=False`).
- `audio` only: Plays audio, which is not added to the transcript or chat context.
- `text` + `audio`: Plays the provided audio and the `text` is used for the transcript and chat context.

- **`text`** _(str | AsyncIterable[str])_ (optional): Text for TTS playback, added to the transcript and by default to the chat context.

- **`audio`** _(AsyncIterable[rtc.AudioFrame])_ (optional): Pre-synthesized audio to play. If used without `text`, nothing is added to the transcript or chat context.

- **`allow_interruptions`** _(boolean)_ (optional) - Default: `True`: If `True`, allow the user to interrupt the agent while speaking.

- **`add_to_chat_ctx`** _(boolean)_ (optional) - Default: `True`: If `True`, add the text to the agent's chat context after playback. Has no effect if `text` is not provided.

#### Returns

Returns a [`SpeechHandle`](#speechhandle) object.

#### Events

This method triggers a [`speech_created`](https://docs.livekit.io/reference/other/events.md#speech_created) event.

### generate_reply

To make conversations more dynamic, use `session.generate_reply()` to prompt the LLM to generate a response.

There are two ways to use `generate_reply`:

1. give the agent instructions to generate a response

**Python**:

```python
session.generate_reply(
   instructions="greet the user and ask where they are from",
)

```

---

**Node.js**:

```typescript
 session.generateReply({
 instructions: 'greet the user and ask where they are from',
 });

```
2. provide the user's input via text

**Python**:

```python
session.generate_reply(
   user_input="how is the weather today?",
)

```

---

**Node.js**:

```typescript
 session.generateReply({
 userInput: 'how is the weather today?',
 });

```

> ℹ️ **Impact to chat history**
> 
> When using `generate_reply` with `instructions`, the agent uses the instructions to generate a response, which is added to the chat history. The instructions themselves are not recorded in the history.
> 
> In contrast, `user_input` is directly added to the chat history.

#### Parameters

- **`user_input`** _(string)_ (optional): The user input to respond to.

 - **`instructions`** _(string)_ (optional): Instructions for the agent to use for the reply.

 - **`allow_interruptions`** _(boolean)_ (optional): If `True`, allow the user to interrupt the agent while speaking. (default `True`)

#### Returns

Returns a [`SpeechHandle`](#speechhandle) object.

#### Events

This method triggers a [`speech_created`](https://docs.livekit.io/reference/other/events.md#speech_created) event.

## Controlling agent speech

You can control agent speech using the `SpeechHandle` object returned by the `say()` and `generate_reply()` methods, and allowing user interruptions.

### SpeechHandle

The `say()` and `generate_reply()` methods return a `SpeechHandle` object, which lets you track the state of the agent's speech. This can be useful for coordinating follow-up actions—for example, notifying the user before ending the call.

**Python**:

```python
# The following is a shortcut for:
# handle = session.say("Goodbye for now.", allow_interruptions=False)
# await handle.wait_for_playout()
await session.say("Goodbye for now.", allow_interruptions=False)

```

---

**Node.js**:

```typescript
// The following is a shortcut for:
// const handle = session.say('Goodbye for now.', { allowInterruptions: false });
// await handle.waitForPlayout();
await session.say('Goodbye for now.', { allowInterruptions: false });

```

You can wait for the agent to finish speaking before continuing:

**Python**:

```python
handle = session.generate_reply(instructions="Tell the user we're about to run some slow operations.")

# perform an operation that takes time
...

await handle # finally wait for the speech

```

---

**Node.js**:

```typescript
const handle = session.generateReply({ 
  instructions: "Tell the user we're about to run some slow operations." 
});

// perform an operation that takes time
...

await handle.waitForPlayout(); // finally wait for the speech

```

The following example makes a web request for the user, and cancels the request when the user interrupts:

**Python**:

```python
async with aiohttp.ClientSession() as client_session:
    web_request = client_session.get('https://api.example.com/data')
    handle = await session.generate_reply(instructions="Tell the user we're processing their request.")
    if handle.interrupted:
        # if the user interrupts, cancel the web_request too
        web_request.cancel()

```

---

**Node.js**:

```typescript
import { Task } from '@livekit/agents';

const webRequestTask = Task.from(async (controller) => {
  const response = await fetch('https://api.example.com/data', { 
    signal: controller.signal 
  });
  return response.json();
});

const handle = session.generateReply({
  instructions: "Tell the user we're processing their request.",
});

await handle.waitForPlayout();

if (handle.interrupted) {
  // if the user interrupts, cancel the web_request too
  webRequestTask.cancel();
}

```

`SpeechHandle` has an API similar to `ayncio.Future`, allowing you to add a callback:

**Python**:

```python
handle = session.say("Hello world")
handle.add_done_callback(lambda _: print("speech done"))

```

---

**Node.js**:

```typescript
const handle = session.say('Hello world');
handle.then(() => console.log('speech done'));

```

### Getting the current speech handle

Available in:
- [ ] Node.js
- [x] Python

The agent session's active speech handle, if any, is available with the `current_speech` property. If no speech is active, this property returns `None`. Otherwise, it returns the active `SpeechHandle`.

Use the active speech handle to coordinate with the speaking state. For instance, you can ensure that a hang up occurs only after the current speech has finished, rather than mid-speech:

```python
# to hang up the call as part of a function call
@function_tool
async def end_call(self, ctx: RunContext):
   """Use this tool when the user has signaled they wish to end the current call. The session ends automatically after invoking this tool."""
   await ctx.wait_for_playout() # let the agent finish speaking


   # call API to delete_room
   ...

```

### Interruptions

By default, the agent stops speaking when it detects that the user has started speaking. You can customize this behavior. To learn more, see [Interruptions](https://docs.livekit.io/agents/build/turns.md#interruptions) in the Turn detection topic.

## Customizing pronunciation

Most TTS providers allow you to customize pronunciation of words using Speech Synthesis Markup Language (SSML). The following example uses the [tts_node](https://docs.livekit.io/agents/build/nodes.md#tts_node) to add custom pronunciation rules:

** Filename: `agent.py`**

```python
async def tts_node(
    self,
    text: AsyncIterable[str],
    model_settings: ModelSettings
) -> AsyncIterable[rtc.AudioFrame]:
    # Pronunciation replacements for common technical terms and abbreviations.
    # Support for custom pronunciations depends on the TTS provider.
    pronunciations = {
        "API": "A P I",
        "REST": "rest",
        "SQL": "sequel",
        "kubectl": "kube control",
        "AWS": "A W S",
        "UI": "U I",
        "URL": "U R L",
        "npm": "N P M",
        "LiveKit": "Live Kit",
        "async": "a sink",
        "nginx": "engine x",
    }
    
    async def adjust_pronunciation(input_text: AsyncIterable[str]) -> AsyncIterable[str]:
        async for chunk in input_text:
            modified_chunk = chunk
            
            # Apply pronunciation rules
            for term, pronunciation in pronunciations.items():
                # Use word boundaries to avoid partial replacements
                modified_chunk = re.sub(
                    rf'\b{term}\b',
                    pronunciation,
                    modified_chunk,
                    flags=re.IGNORECASE
                )
            
            yield modified_chunk
    
    # Process with modified text through base TTS implementation
    async for frame in Agent.default.tts_node(
        self,
        adjust_pronunciation(text),
        model_settings
    ):
        yield frame

```

** Filename: `Required imports`**

```python
import re
from livekit import rtc
from livekit.agents.voice import ModelSettings
from livekit.agents import tts
from typing import AsyncIterable

```

** Filename: `agent.ts`**

```typescript
async ttsNode(
  text: ReadableStream<string>,
  modelSettings: voice.ModelSettings,
): Promise<ReadableStream<AudioFrame> | null> {
  // Pronunciation replacements for common technical terms and abbreviations.
  // Support for custom pronunciations depends on the TTS provider.
  const pronunciations = {
    API: 'A P I',
    REST: 'rest',
    SQL: 'sequel',
    kubectl: 'kube control',
    AWS: 'A W S',
    UI: 'U I',
    URL: 'U R L',
    npm: 'N P M',
    LiveKit: 'Live Kit',
    async: 'a sink',
    nginx: 'engine x',
  };

  const adjustPronunciation = (inputText: ReadableStream<string>): ReadableStream<string> => {
    return new ReadableStream({
      async start(controller) {
        const reader = inputText.getReader();
        try {
          while (true) {
            const { done, value: chunk } = await reader.read();
            if (done) break;

            let modifiedChunk = chunk;

            // Apply pronunciation rules
            for (const [term, pronunciation] of Object.entries(pronunciations)) {
              // Use word boundaries to avoid partial replacements
              const regex = new RegExp(`\\b${term}\\b`, 'gi');
              modifiedChunk = modifiedChunk.replace(regex, pronunciation);
            }

            controller.enqueue(modifiedChunk);
          }
        } finally {
          reader.releaseLock();
          controller.close();
        }
      },
    });
  };

  // Process with modified text through base TTS implementation
  return voice.Agent.default.ttsNode(this, adjustPronunciation(text), modelSettings);
}

```

** Filename: `Required imports`**

```typescript
import type { AudioFrame } from '@livekit/rtc-node';
import { ReadableStream } from 'stream/web';
import { voice } from '@livekit/agents';

```

The following table lists the SSML tags supported by most TTS providers:

| SSML Tag | Description |
| `phoneme` | Used for phonetic pronunciation using a standard phonetic alphabet. These tags provide a phonetic pronunciation for the enclosed text. |
| `say as` | Specifies how to interpret the enclosed text. For example, use `character` to speak each character individually, or `date` to specify a calendar date. |
| `lexicon` | A custom dictionary that defines the pronunciation of certain words using phonetic notation or text-to-pronunciation mappings. |
| `emphasis` | Speak text with an emphasis. |
| `break` | Add a manual pause. |
| `prosody` | Controls pitch, speaking rate, and volume of speech output. |

## Adjusting speech volume

To adjust the volume of the agent's speech, add a processor to the `tts_node` or the `realtime_audio_output_node`.  Alternative, you can also [adjust the volume of playback](https://docs.livekit.io/transport/media/subscribe.md#volume) in the frontend SDK.

The following example agent has an adjustable volume between 0 and 100, and offers a [tool call](https://docs.livekit.io/agents/build/tools.md) to change it.

** Filename: `agent.py`**

```python
class Assistant(Agent):
    def __init__(self) -> None:
        self.volume: int = 50
        super().__init__(
            instructions=f"You are a helpful voice AI assistant. Your starting volume level is {self.volume}."
        )

    @function_tool()
    async def set_volume(self, volume: int):
        """Set the volume of the audio output.

        Args:
            volume (int): The volume level to set. Must be between 0 and 100.
        """
        self.volume = volume

    # Audio node used by STT-LLM-TTS pipeline models
    async def tts_node(self, text: AsyncIterable[str], model_settings: ModelSettings):
        return self._adjust_volume_in_stream(
            Agent.default.tts_node(self, text, model_settings)
        )

    # Audio node used by realtime models
    async def realtime_audio_output_node(
        self, audio: AsyncIterable[rtc.AudioFrame], model_settings: ModelSettings
    ) -> AsyncIterable[rtc.AudioFrame]:
        return self._adjust_volume_in_stream(
            Agent.default.realtime_audio_output_node(self, audio, model_settings)
        )

    async def _adjust_volume_in_stream(
        self, audio: AsyncIterable[rtc.AudioFrame]
    ) -> AsyncIterable[rtc.AudioFrame]:
        stream: utils.audio.AudioByteStream | None = None
        async for frame in audio:
            if stream is None:
                stream = utils.audio.AudioByteStream(
                    sample_rate=frame.sample_rate,
                    num_channels=frame.num_channels,
                    samples_per_channel=frame.sample_rate // 10,  # 100ms
                )
            for f in stream.push(frame.data):
                yield self._adjust_volume_in_frame(f)

        if stream is not None:
            for f in stream.flush():
                yield self._adjust_volume_in_frame(f)

    def _adjust_volume_in_frame(self, frame: rtc.AudioFrame) -> rtc.AudioFrame:
        audio_data = np.frombuffer(frame.data, dtype=np.int16)
        audio_float = audio_data.astype(np.float32) / np.iinfo(np.int16).max
        audio_float = audio_float * max(0, min(self.volume, 100)) / 100.0
        processed = (audio_float * np.iinfo(np.int16).max).astype(np.int16)

        return rtc.AudioFrame(
            data=processed.tobytes(),
            sample_rate=frame.sample_rate,
            num_channels=frame.num_channels,
            samples_per_channel=len(processed) // frame.num_channels,
        )

```

** Filename: `Required imports`**

```python
import numpy as np
from typing import AsyncIterable
from livekit.agents import Agent, function_tool, utils
from livekit.plugins import rtc

```

** Filename: `agent.ts`**

```typescript
class Assistant extends voice.Agent {
  private volume = 50;

  constructor(initialVolume: number) {
    super({
      instructions: `You are a helpful voice AI assistant. Your starting volume level is ${initialVolume}.`,
      tools: {
        setVolume: llm.tool({
          description: 'Set the volume of the audio output.',
          parameters: z.object({
            volume: z
              .number()
              .min(0)
              .max(100)
              .describe('The volume level to set. Must be between 0 and 100.'),
          }),
          execute: async ({ volume }) => {
            this.volume = volume;
            return `Volume set to ${volume}`;
          },
        }),
      },
    });
    this.volume = initialVolume;
  }

  // Audio node used by STT-LLM-TTS pipeline models
  async ttsNode(
    text: ReadableStream<string>,
    modelSettings: voice.ModelSettings,
  ): Promise<ReadableStream<AudioFrame> | null> {
    const baseStream = await voice.Agent.default.ttsNode(this, text, modelSettings);
    if (!baseStream) return null;
    return this.adjustVolumeInStream(baseStream);
  }

  // Audio node used by realtime models
  async realtimeAudioOutputNode(
    audio: ReadableStream<AudioFrame>,
    modelSettings: voice.ModelSettings,
  ): Promise<ReadableStream<AudioFrame> | null> {
    const baseStream = await voice.Agent.default.realtimeAudioOutputNode(
      this,
      audio,
      modelSettings,
    );
    if (!baseStream) return null;
    return this.adjustVolumeInStream(baseStream);
  }

  private adjustVolumeInStream(
    audioStream: ReadableStream<AudioFrame>,
  ): ReadableStream<AudioFrame> {
    return new ReadableStream({
      start: async (controller) => {
        const reader = audioStream.getReader();
        try {
          while (true) {
            const { done, value: frame } = await reader.read();
            if (done) break;

            const adjustedFrame = this.adjustVolumeInFrame(frame);
            controller.enqueue(adjustedFrame);
          }
        } finally {
          reader.releaseLock();
          controller.close();
        }
      },
    });
  }

  private adjustVolumeInFrame(frame: AudioFrame): AudioFrame {
    const audioData = new Int16Array(frame.data);
    const volumeMultiplier = Math.max(0, Math.min(this.volume, 100)) / 100.0;

    const processedData = new Int16Array(audioData.length);
    for (let i = 0; i < audioData.length; i++) {
      const floatSample = audioData[i]! / 32767.0;
      const adjustedSample = floatSample * volumeMultiplier;
      processedData[i] = Math.round(adjustedSample * 32767.0);
    }

    return new AudioFrame(processedData, frame.sampleRate, frame.channels, frame.samplesPerChannel);
  }
}

```

** Filename: `Required imports`**

```typescript
import { voice } from '@livekit/agents';
import { AudioFrame } from '@livekit/rtc-node';
import { ReadableStream } from 'stream/web';

```

## Adding background audio

To add more realism to your agent, or add additional sound effects, publish background audio. This audio is played on a separate audio track. The `BackgroundAudioPlayer` class supports on-demand playback of custom audio as well as automatic ambient and thinking sounds synchronized to the agent lifecycle.

For a complete example, see the following recipes:

- **[Background audio](https://github.com/livekit/agents/blob/main/examples/voice_agents/background_audio.py)**: A voice AI agent with background audio for thinking states and ambiance.

- **[Background audio example in Node.js](https://github.com/livekit/agents-js/blob/main/examples/src/background_audio.ts)**: A voice AI agent with background audio for ambiance.

### Create the player

The `BackgroundAudioPlayer` class manages audio playback to a room. It can also play ambient and thinking sounds automatically during the lifecycle of the agent session, if desired.

- **`ambient_sound`** _(AudioSource | AudioConfig | list[AudioConfig])_ (optional): Ambient sound plays on a loop in the background during the agent session. See [Supported audio sources](#audio-sources) and [Multiple audio clips](#multiple-audio-clips) for more details.

- **`thinking_sound`** _(AudioSource | AudioConfig | list[AudioConfig])_ (optional): Thinking sound plays while the agent is in the "thinking" state. See [Supported audio sources](#audio-sources) and [Multiple audio clips](#multiple-audio-clips) for more details. This parameter is currently [only supported](#background-audio-limitations) in Python.

Create the player within your entrypoint function:

**Python**:

```python
from livekit.agents import BackgroundAudioPlayer, AudioConfig, BuiltinAudioClip

# An audio player with automated ambient and thinking sounds
background_audio = BackgroundAudioPlayer(
    ambient_sound=AudioConfig(BuiltinAudioClip.OFFICE_AMBIENCE, volume=0.8),
    thinking_sound=[
        AudioConfig(BuiltinAudioClip.KEYBOARD_TYPING, volume=0.8),
        AudioConfig(BuiltinAudioClip.KEYBOARD_TYPING2, volume=0.7),
    ],
)

# An audio player with a custom ambient sound played on a loop
background_audio = BackgroundAudioPlayer(
    ambient_sound="/path/to/my-custom-sound.mp3",
)

# An audio player for on-demand playback only
background_audio = BackgroundAudioPlayer()

```

---

**Node.js**:

```typescript
import { voice } from '@livekit/agents';

const backgroundAudio = new voice.BackgroundAudioPlayer({
    ambientSound: { 
            source: voice.BuiltinAudioClip.OFFICE_AMBIENCE,
            volume: 0.8,
    },
    // Thinking sounds are not yet supported in Node.js
});

# An audio player with a custom ambient sound played on a loop
backgroundAudio = new voice.BackgroundAudioPlayer({
    ambientSound: "/path/to/my-custom-sound.mp3",
})

# An audio player for on-demand playback only
backgroundAudio = new voice.BackgroundAudioPlayer()

```

### Start and stop the player

Call the `start` method after room connection and after starting the agent session. Ambient sounds, if any, begin playback immediately.

- `room`: The room to publish the audio to.
- `agent_session`: The agent session to publish the audio to.

**Python**:

```python
await background_audio.start(room=ctx.room, agent_session=session)

```

---

**Node.js**:

```typescript
await backgroundAudio.start({ room: ctx.room, agentSession: session });

```

To stop and clean up the player, call the `aclose` (or `close` in Node.js) method. You must create a new player instance if you want to start again.

**Python**:

```python
await background_audio.aclose()

```

---

**Node.js**:

```typescript
await backgroundAudio.close();

```

### Play audio on-demand

You can play audio at any time, after starting the player, with the `play` method.

- **`audio`** _(AudioSource | AudioConfig | list[AudioConfig])_: The audio source or a probabilistic list of sources to play. To learn more, see [Supported audio sources](#audio-sources) and [Multiple audio clips](#multiple-audio-clips).

- **`loop`** _(boolean)_ (optional) - Default: `False`: Set to `True` to continuously loop playback.

For example, if you created `background_audio` in the [previous example](#publishing-background-audio), you can play an audio file like this:

**Python**:

```python
background_audio.play("/path/to/my-custom-sound.mp3")

```

---

**Node.js**:

```typescript
backgroundAudio.play("/path/to/my-custom-sound.mp3");

```

The `play` method returns a `PlayHandle` which you can use to await or cancel the playback.

The following example uses the handle to await playback completion:

**Python**:

```python
# Wait for playback to complete
await background_audio.play("/path/to/my-custom-sound.mp3")

```

---

**Node.js**:

```typescript
const handle = await backgroundAudio.play("/path/to/my-custom-sound.mp3");

```

The next example shows the handle's `stop` method, which stops playback early:

**Python**:

```python
handle = background_audio.play("/path/to/my-custom-sound.mp3")
await(asyncio.sleep(1))
handle.stop() # Stop playback early

```

---

**Node.js**:

```typescript
const handle = backgroundAudio.play("/path/to/my-custom-sound.mp3");
await new Promise(resolve => setTimeout(resolve, 1000));
handle.stop(); // Stop playback early

```

### Multiple audio clips

You can pass a list of audio sources to any of `play`, `ambient_sound`, or `thinking_sound`. The player selects a single entry in the list based on the `probability` parameter. This is useful to avoid repetitive sound effects. To allow for the possibility of no audio at all, ensure the sum of the probabilities is less than 1.

`AudioConfig` has the following properties:

- **`source`** _(AudioSource)_: The audio source to play. See [Supported audio sources](#audio-sources) for more details.

- **`volume`** _(float)_ (optional) - Default: `1`: The volume at which to play the given audio.

- **`probability`** _(float)_ (optional) - Default: `1`: The relative probability of selecting this audio source from the list.

**Python**:

```python
# Play the KEYBOARD_TYPING sound with an 80% probability and the KEYBOARD_TYPING2 sound with a 20% probability
background_audio.play([
    AudioConfig(BuiltinAudioClip.KEYBOARD_TYPING, volume=0.8, probability=0.8),
    AudioConfig(BuiltinAudioClip.KEYBOARD_TYPING2, volume=0.7, probability=0.2),
])

```

---

**Node.js**:

```typescript
// Play the KEYBOARD_TYPING sound with an 80% probability and the KEYBOARD_TYPING2 sound with a 20% probability
backgroundAudio.play([
    { source: voice.BuiltinAudioClip.KEYBOARD_TYPING, volume: 0.8, probability: 0.8 },
    { source: voice.BuiltinAudioClip.KEYBOARD_TYPING2, volume: 0.7, probability: 0.2 },
])

```

### Supported audio sources

The following audio sources are supported:

#### Local audio file

Pass a string path to any local audio file. The player decodes files with FFmpeg via [PyAV](https://github.com/PyAV-Org/PyAV) and supports all common audio formats including MP3, WAV, AAC, FLAC, OGG, Opus, WebM, and MP4.

> 💡 **WAV files**
> 
> The player uses an optimized custom decoder to load WAV data directly to audio frames, without the overhead of FFmpeg. For small files, WAV is the highest-efficiency option.

#### Built-in audio clips

The following built-in audio clips are available by default for common sound effects:

- `BuiltinAudioClip.OFFICE_AMBIENCE`: Chatter and general background noise of a busy office.
- `BuiltinAudioClip.KEYBOARD_TYPING`: The sound of an operator typing on a keyboard, close to their microphone.
- `BuiltinAudioClip.KEYBOARD_TYPING2`: A shorter version of `KEYBOARD_TYPING`.

#### Raw audio frames

Pass an `AsyncIterator[rtc.AudioFrame]` to play raw audio frames from any source.

#### Limitations

Thinking sounds are not yet supported in Node.js.

## Additional resources

To learn more, see the following resources.

- **[Voice AI quickstart](https://docs.livekit.io/agents/start/voice-ai.md)**: Use the quickstart as a starting base for adding audio code.

- **[Speech related event](https://docs.livekit.io/agents/build/events.md#speech_created)**: Learn more about the `speech_created` event, triggered when new agent speech is created.

- **[LiveKit SDK](https://docs.livekit.io/transport/media/publish.md#publishing-audio-tracks)**: Learn how to use the LiveKit SDK to play audio tracks.

- **[Background audio](https://github.com/livekit/agents/blob/main/examples/voice_agents/background_audio.py)**: A voice AI agent with background audio for thinking states and ambiance.

- **[Background audio example in Node.js](https://github.com/livekit/agents-js/blob/main/examples/src/background_audio.ts)**: A voice AI agent with background audio for ambiance.

- **[Text-to-speech (TTS)](https://docs.livekit.io/agents/models/tts.md)**: TTS models for pipeline agents.

- **[Speech-to-speech](https://docs.livekit.io/agents/models/realtime.md)**: Realtime models that understand speech input and generate speech output directly.

---

This document was rendered at 2026-02-03T03:24:55.092Z.
For the latest version of this document, see [https://docs.livekit.io/agents/multimodality/audio.md](https://docs.livekit.io/agents/multimodality/audio.md).

To explore all LiveKit documentation, see [llms.txt](https://docs.livekit.io/llms.txt).