# Pipecat

> ## Documentation Index

---

# Source: https://docs.pipecat.ai/deployment/pipecat-cloud/fundamentals/accounts-and-organizations.md

> ## Documentation Index
> Fetch the complete documentation index at: https://docs.pipecat.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Accounts and Organizations

> Managing your account and collaborating on agents as part of a team

Pipecat Cloud accounts are issued a personal workspace where you can deploy agents independently.
User workspaces are unique and isolated from one another, making them unsuitable for collaboration.

## Authentication

Before you can issue any commands via the CLI, you must authenticate with your Pipecat Cloud account.

```bash  theme={null}
pipecat cloud auth login
```

This command will open a browser window and prompt you to authenticate with your Pipecat Cloud account.

You can verify a login was successful by running the `whoami` command:

```bash  theme={null}
pipecat cloud auth whoami
```

You can sign out of your account by running the following:

```bash  theme={null}
pipecat cloud auth logout
```

**Credentials**

Access tokens are stored locally on your device at `~/.config/pipecatcloud/pipecatcloud.toml`. You can override the default location by setting the `PIPECAT_CONFIG_PATH` environment variable.
If you need to revoke access to your account, please sign out of the Pipecat Cloud dashboard to invalidate your access token.

## Organizations

Working as part of a team requires creating an organization. Organizations are multi-tenant workspaces that enable collaboration on deployments.

* Billing and integrations are managed at the organization level.
* Agents and secrets are shared across the organization, with varying roles and privileges.
* Organizations have dedicated access tokens for initializing agent instances.

<Info>
  Deployments cannot be shared or moved between organizations or user
  accounts. If you plan to collaborate with others, we recommend creating
  a new organization.
</Info>

### Creating organizations

Organization creation must be done via the [Pipecat Cloud dashboard<Icon icon="arrow-up-right-from-square" iconType="solid" />](https://pipecat.daily.co/).

### Inviting accounts to your organization

You can invite a user to your organization by their email address using the [Pipecat Cloud dashboard<Icon icon="arrow-up-right-from-square" iconType="solid" />](https://pipecat.daily.co/).

### Working with organizations

You can get a list of available organizations that your account is a member of by running the following:

```bash  theme={null}
pipecat cloud organizations list
```

You can switch to a different organization by running the following:

```bash  theme={null}
pipecat cloud organizations select
```

Switching organizations will target all CLI commands to the selected organization. For example, `pipecat cloud deploy` will issue the deployment request to your select organization.

Alternatively, you issue commands with the optional `--org` flag, for example:

```bash  theme={null}
pipecat cloud deploy my-agent image-name --org my-organization
```

## API keys

Pipecat Cloud requires using an API key for some CLI commands or interfacing via the REST API.

API keys are issued at the user workspace and organization level.

You can create up to 5 private and 5 public API keys for different agents / purposes.

### Private API keys

Used to authenticate with private REST API endpoints. Private API keys
are typically used for administrative tasks and should not be shared or used by end-user applications.

Examples of where you may want to use private API keys include:

* Managing organizations and users via the REST API
* Server-to-server deployment scripts
* GitHub actions

You should not share your private API key with anyone.

### Public API keys

Public API keys are suitable for CLI usage for commands that could be invoked by end-user actions on the platform.

Examples of where you may want to use public API keys include:

* [Starting an agent instance](./active-sessions) (via the CLI or REST API)
* Server-to-server end-user applications
* Client-side while developing / iterating

Public API keys allow starting new agent processes, so while they are generally safe to share, you should still be cautious to avoid
malicious use (i.e. triggering your account rate limit.) We recommend keeping your public API key server-side and not sharing it in public repositories.

### Managing API keys

You can create, cycle, revoke or delete API keys via the [Pipecat Cloud dashboard <Icon icon="arrow-up-right-from-square" iconType="solid" />](https://pipecat.daily.co) and CLI.

```bash  theme={null}
# Create a new API key
pipecat cloud organizations keys create

# List available keys
pipecat cloud organizations keys list
```

When using the CLI, you can specify a default API key to use for all commands that require it:

```bash  theme={null}
pipecat cloud organizations keys use
```

Selecting a key will modify your local configuration file and use it for all subsequent commands.

Please be aware that if the key is cycled or revoked, you will need to run the `keys use` command again to update your local configuration file with a new key.

---

# Source: https://docs.pipecat.ai/deployment/pipecat-cloud/fundamentals/active-sessions.md

> ## Documentation Index
> Fetch the complete documentation index at: https://docs.pipecat.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Starting Sessions

> Serving user requests with active sessions.

Once you have made a successful deployment, you can start on-demand sessions to begin interacting with your agent.

An active session is a running instance of your agent code that connects with you or your end-users.
Sessions are created in response to a request and scaled up or down based on your deployment configuration (see [Scaling](./scaling)).

<Info>
  You must have an active billing account to start sessions. Usage is billed for
  the duration of each session, please refer to
  [pricing](https://www.daily.co/pricing/pipecat-cloud) for more information.
</Info>

Starting active sessions can be achieved programmatically via the REST API and Python library or directly via the CLI.

## Running an agent

Requests to start an agent requires passing a [public API key](./accounts-and-organizations#api-keys). Keys are used to ensure that only authorized users or apps can start sessions and can be cycled or revoked at any time.

### Development pattern

A typical application will start a session in response to a user request (via a web UI, mobile app, phone call or other client). Pipecat Cloud takes care of your session infrastructure and scaling, so you can focus on building your application logic.

Developers should:

1. Provide a secure endpoint to receive requests that keeps your API key secret

2. Pass any required data to the agent when starting a session (such as pipeline configuration, transport session data, etc.)

3. Handle any errors that may occur during the start process

### Using REST

The Pipecat Cloud REST API provides an endpoint to start a session with your agent over HTTP:

<CodeGroup>
  ```bash cURL theme={null}
  curl --location --request POST 'https://api.pipecat.daily.co/v1/public/{agent_name}/start' \
    --header 'Authorization: Bearer PUBLIC_API_KEY' \
    --header 'Content-Type: application/json' \
    --data-raw '{
      "createDailyRoom": true,
      "body": {"custom": "data"}
    }'
  ```

  ```python Python theme={null}
  endpoint = "https://api.pipecat.daily.co/v1/public/{service}/start"
  agent_name = "my-first-agent"
  api_key = "pk_..."

  async with aiohttp.ClientSession() as session:
      response = await session.post(
          f"{endpoint.format(service=agent_name)}",
          headers={"Authorization": f"Bearer {api_key}"},
          json={
              "createDailyRoom": True,  # Creates a Daily room
              "body": {"custom": "data"}  # Data to pass to your agent
          }
      )
  ```
</CodeGroup>

### Using Python

The Pipecat Cloud Python library (`pip install pipecatcloud`) provides helper methods to start a session with your agent:

```python  theme={null}
import asyncio

from pipecatcloud.exception import AgentStartError
from pipecatcloud.session import Session, SessionParams


async def main():
    try:
        # Create session object
        session = Session(
            agent_name="my-first-agent",
            api_key=API_KEY,  # Replace with your actual API key
            params=SessionParams(
                use_daily=True,  # Optional: Creates a Daily room
                daily_room_properties={"start_video_off": False},
                data={"key": "value"},
            ),
        )

        # Start the session
        response = await session.start()

        # Get Daily room URL
        daily_url = f"{response['dailyRoom']}?t={response['dailyToken']}"
        print(f"Join Daily room: {daily_url}")

    except AgentStartError as e:
        print(f"Error starting agent: {e}")
    except Exception as e:
        print(f"Unexpected error: {e}")


# Run the async function
if __name__ == "__main__":
    asyncio.run(main())
```

### Using the CLI

To start an agent via the command line interface:

```shell  theme={null}
pipecat cloud agent start my-first-agent --api-key pk_...
```

You can select and use a default API key for the agent start command:

```shell  theme={null}
pipecat cloud organizations keys use my-default-key
```

See the [CLI reference](../cli-reference/agent#start) for more information.

## Passing data

Most agents will require some initial data, such pipeline configuration, transport session credentials, etc.

For sensitive data, we recommend using [secrets](./secrets) to store and access your data securely.

You pass a data blob through to the start request (please note that the data must be JSON-serializable):

```bash  theme={null}
pipecat cloud agent start my-first-agent --data '{"room": "my-room"}'

# or from a file
pipecat cloud agent start my-first-agent --data-file data.json
```

Data passed to your session can be accessed in your agent code via the `bot()` method, which receives session arguments:

<Tabs>
  <Tab title="Standard Session">
    ```python  theme={null}
    # bot.py
    from pipecat.runner.types import PipecatRunnerArguments

    async def bot(args: PipecatRunnerArguments):
        # Access the session ID
        logger.info(f"Session started with ID: {args.session_id}")
        
        # Access the custom data passed to the session
        logger.info(f"Session data: {args.body}")
        
        # ...
    ```
  </Tab>

  <Tab title="Daily Voice Session">
    ```python  theme={null}
    # bot.py for voice agent
    from loguru import logger
    from pipecat.runner.types import DailyRunnerArguments

    async def bot(args: DailyRunnerArguments):
        """Main bot entry point compatible with the FastAPI route handler.

        Args:
            room_url: The Daily room URL
            token: The Daily room token
            body: The configuration object from the request body
        """
        logger.info(f"Bot process initialized {args.room_url} {args.token}")
        logger.info(f"Custom data: {args.body}")
        logger.info(f"Session ID: {args.session_id}")

        try:
            await main(args.room_url, args.token)
            logger.info("Bot process completed")
        except Exception as e:
            logger.exception(f"Error in bot process: {str(e)}")
            raise
    ```
  </Tab>

  <Tab title="WebSocket Session">
    ```python  theme={null}
    # bot.py for WebSocket agent
    from loguru import logger
    from pipecat.runner.types import WebSocketRunnerArguments

    async def bot(args: WebSocketRunnerArguments):
        """Main bot entry point for WebSocket connections.

        Args:
            ws: The WebSocket connection
        """
        logger.info("WebSocket bot process initialized")
        logger.info(f"Custom data: {args.body}")
        logger.info(f"Session ID: {args.session_id}")

        try:
            await main(args.websocket)
            logger.info("WebSocket bot process completed")
        except Exception as e:
            logger.exception(f"Error in WebSocket bot process: {str(e)}")
            raise
    ```
  </Tab>
</Tabs>

For more advanced data requirements, your application should run its own data fetching logic before a session request is made.

If you require your agent to fetch it's own data, be mindful of any blocking requests that could slow down the start process.

## Agent capacity

Deployments have an upper limit for the number of sessions that can be started concurrently.
This helps developers implement cost control and workload management and can be adjusted per deployment.

See [Scaling](./scaling) for more information.

<Warning>
  The default instance pool capacity is **50 active sessions per
  deployment**. Please contact us at [help@daily.co](mailto:help@daily.co) or via
  [Discord](https://discord.gg/dailyco) if you require more capacity.
</Warning>

Developers are responsible for handling capacity errors when starting sessions. When your instance pool is at capacity, the start request will fail with a `429` status and code.

Here is a simple example of how to handle a capacity error:

<CodeGroup>
  ```bash REST theme={null}
  curl --location --request POST 'https://api.pipecat.daily.co/v1/public/{agent_name}/start' \
  --header 'Authorization: Bearer PUBLIC_API_KEY' \
  --header 'Content-Type: application/json' \
  --data-raw '{}'

  >> {
  >>	"status": 429,
  >>	"code": "429",
  >>	"message": "Rate limit exceeded. Please try again later"
  >> }
  ```

  ```python Python theme={null}
  from pipecatcloud import Session, SessionParams
  from pipecatcloud.exception import AgentStartError

  try:
      session = Session(
          agent_name="my-first-agent",
          api_key="pk_...",
          params=SessionParams(
              use_daily=False,
              data={}
          )
      )
  except AgentStartError as e:
      # This will catch capacity errors and other start failures
      print(f"Error starting agent: {e}")
  except Exception as e:
      print(f"Unexpected error: {e}")
  ```
</CodeGroup>

## Logging and monitoring

Session observability can be accessed via the [Dashboard <Icon icon="arrow-up-right-from-square" iconType="solid" />](https://pipecat.daily.co) or from the CLI.

### Active sessions

To view active sessions via the CLI:

```shell  theme={null}
pipecat cloud agent sessions [my-first-agent]
```

This command will list all active sessions for the specified agent, alongside a session ID.

### Session logs

To view individual session logs, use the following command:

```shell  theme={null}
pipecat cloud agent logs [my-first-agent] --session [session-id]
```

For more information, see [Logging and Observability](./logging).

## Stopping sessions

You can issue a termination request to stop an active session. This will immediately stop the instance so be aware of disrupting any ongoing user interactions.

```shell  theme={null}
pipecat cloud agent stop my-first-agent --session [session-id]
```

Deleting an agent deployment will block any new sessions from starting, but will not stop any active sessions. You must stop each session individually.

## Usage

You are charged for the duration of each session, see [pricing](https://www.daily.co/pricing/pipecat-cloud).

The exact duration of session is calculated based on:

* Time spent instantiating the agent
* Time spent running the agent
* Time spent tearing down the agent

You can monitor your usage on your [Dashboard <Icon icon="arrow-up-right-from-square" iconType="solid" />](https://pipecat.daily.co).

---

# Source: https://docs.pipecat.ai/deployment/pipecat-cloud/rest-reference/endpoint/agent-create.md

> ## Documentation Index
> Fetch the complete documentation index at: https://docs.pipecat.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Create an Agent

> Create a new agent with a container image and configuration settings.



## OpenAPI

````yaml POST /agents
openapi: 3.0.0
info:
  title: Pipecat Cloud
  version: 1.0.0
  description: Private API for Pipecat Cloud agent management
servers:
  - url: https://api.pipecat.daily.co/v1
    description: API server
security:
  - PrivateKeyAuth: []
paths:
  /agents:
    post:
      summary: Create a new agent
      operationId: createService
      requestBody:
        required: true
        content:
          application/json:
            schema:
              $ref: '#/components/schemas/CreateServiceRequest'
            example:
              serviceName: voice-starter
              image: your-dockername/voice-starter:0.1
              region: us-west
              nodeType: arm
              imagePullSecretSet: dockerhub-credentials
              secretSet: voice-starter-secrets
              autoScaling:
                minAgents: 1
                maxAgents: 20
              krispViva:
                audioFilter: tel
              agentProfile: agent-1x
              enableManagedKeys: false
      responses:
        '200':
          description: Agent created successfully
          content:
            application/json:
              schema:
                $ref: '#/components/schemas/ServiceDetailsResponse'
              example:
                name: voice-starter
                region: us-west
                ready: false
                createdAt: '2025-04-19T01:20:27.564Z'
                updatedAt: '2025-04-19T01:20:27.572Z'
                activeDeploymentId: 13c0be89-5ae8-4b0b-ad22-79565e11de3b
                activeDeploymentReady: false
                autoScaling:
                  maxReplicas: 20
                  minReplicas: 1
                activeSessionCount: 0
                deployment:
                  id: 13c0be89-5ae8-4b0b-ad22-79565e11de3b
                  manifest:
                    apiVersion: pipecatcloud.daily.co/v1
                    kind: PCService
                    metadata:
                      name: voice-starter
                      namespace: tiny-ferret-maroon-123
                    spec:
                      dailyNodeType: arm
                      deploymentMode: keda
                      clusterLocal: true
                      image: your-dockername/voice-starter:0.1
                      autoScaling:
                        minReplicas: 1
                        maxReplicas: 20
                      envFromSecretNames:
                        - voice-starter-secrets
                      krispModels:
                        enabled: false
                      krispViva:
                        audioFilters: true
                        version: '20251010'
                        modelVars:
                          KRISP_VIVA_MODEL_PATH: audio_filters/krisp-viva-tel-v2.kef
                      integratedKeysProxy:
                        enabled: false
                      resources:
                        cpu: 500m
                        memory: 1Gi
                  serviceId: b59a68ee-61c8-4d99-9ceb-e99a3953bdac
                  createdAt: '2025-04-19T01:20:27.569Z'
                  updatedAt: '2025-04-19T01:20:27.569Z'
                agentProfile: agent-1x
                krispViva:
                  audioFilter: tel
        '400':
          description: Invalid request or agent already exists
          content:
            application/json:
              schema:
                $ref: '#/components/schemas/ErrorResponse'
              examples:
                serviceExists:
                  summary: Service already exists
                  value:
                    error: Service already exists
                    code: GENERIC_BAD_REQUEST
                secretRegionMismatch:
                  summary: Secret set region mismatch
                  value:
                    error: >-
                      Secret set 'my-secrets' is in region 'us-east' but service
                      will be created in region 'us-west'. Secrets must be in
                      the same region as the service.
                    code: GENERIC_BAD_REQUEST
                imagePullSecretRegionMismatch:
                  summary: Image pull secret region mismatch
                  value:
                    error: >-
                      Image pull secret set 'dockerhub-creds' is in region
                      'us-east' but service will be created in region 'us-west'.
                      Image pull secrets must be in the same region as the
                      service.
                    code: GENERIC_BAD_REQUEST
        '404':
          description: Secret set or image pull secret set not found
          content:
            application/json:
              schema:
                $ref: '#/components/schemas/ErrorResponse'
              example:
                error: Secret set not found
                code: NOT_FOUND
        '500':
          description: Internal server error
          content:
            application/json:
              schema:
                $ref: '#/components/schemas/ErrorResponse'
              example:
                error: Internal server error
                code: INTERNAL_SERVER_ERROR
      security:
        - PrivateKeyAuth: []
components:
  schemas:
    CreateServiceRequest:
      type: object
      required:
        - serviceName
        - image
      properties:
        serviceName:
          type: string
          description: >-
            Name of the agent to create.


            Must start with a lowercase letter or number, can include hyphens,
            and must end with a lowercase letter or number. No uppercase letters
            or special characters allowed.
          example: my-voice-agent
          pattern: ^[a-z0-9]([-a-z0-9]*[a-z0-9])$
        image:
          type: string
          description: The container image to use for the agent
          example: your-username/my-agent:latest
        region:
          type: string
          description: >-
            The region where the agent will be deployed. If not specified,
            defaults to `us-west`. All secrets and image pull secrets referenced
            by this agent must be in the same region.
          default: us-west
          example: us-west
        nodeType:
          type: string
          description: >-
            The type of node to run the agent on. Only `arm` is supported at
            this time.
          default: arm
          example: arm
        imagePullSecretSet:
          type: string
          description: The name of the image pull secret set to use
          example: dockerhub-credentials
        secretSet:
          type: string
          description: The name of the secret set to use
          example: my-agent-secrets
        autoScaling:
          type: object
          description: Auto-scaling configuration for the agent
          properties:
            minAgents:
              type: integer
              description: Minimum number of agents
              example: 1
            maxAgents:
              type: integer
              description: 'Maximum number of agents (Default: 50)'
              example: 20
        enableKrisp:
          type: boolean
          deprecated: true
          description: >-
            Whether to enable Krisp noise cancellation. **Deprecated:** Use
            `krispViva` instead for the latest Krisp VIVA models.
          default: false
          example: false
        krispViva:
          type: object
          description: >-
            Krisp VIVA noise cancellation configuration. [Learn
            more](/deployment/pipecat-cloud/guides/krisp-viva).
          properties:
            audioFilter:
              type: string
              nullable: true
              description: >-
                The Krisp VIVA audio filter model to use:


                - `tel`: Optimized for telephony, cellular, landline, mobile,
                desktop, and browser (up to 16kHz)

                - `pro`: Optimized for mobile, desktop, and browser with WebRTC
                (up to 32kHz)

                - `null`: Disables Krisp VIVA


                By default, Pipecat processes input audio at 16kHz, making `tel`
                appropriate for most use cases.
              enum:
                - tel
                - pro
                - null
              example: tel
        agentProfile:
          type: string
          description: >-
            The agent profile to use for resource allocation. Valid values are:


            - `agent-1x`: 0.5 vCPU and 1 GB of memory. Best for voice agents.

            - `agent-2x`: 1 vCPU and 2 GB of memory. Well suited for voice and
            video agents or voice agents requiring extra processing.

            - `agent-3x`: 1.5 vCPU and 3 GB of memory. Best for voice and video
            agents requiring extra processing or multiple video inputs.
          enum:
            - agent-1x
            - agent-2x
            - agent-3x
          default: agent-1x
          example: agent-1x
        enableManagedKeys:
          type: boolean
          description: >-
            Pipecat Cloud provides API keys for some Pipecat services that you
            can use directly in your applications. [Learn
            more](/deployment/pipecat-cloud/guides/managed-api-keys).
          default: false
          example: false
    ServiceDetailsResponse:
      type: object
      properties:
        name:
          type: string
          description: Name of the agent
        region:
          type: string
          description: The region where the agent is deployed
        ready:
          type: boolean
          description: Whether the agent is ready to accept sessions
        createdAt:
          type: string
          format: date-time
          description: Creation timestamp of the agent
        updatedAt:
          type: string
          format: date-time
          description: Last update timestamp
        activeDeploymentId:
          type: string
          description: ID of the active deployment
        activeDeploymentReady:
          type: boolean
          description: Whether the active deployment is ready
        autoScaling:
          type: object
          properties:
            maxReplicas:
              type: integer
              description: Maximum number of replicas
            minReplicas:
              type: integer
              description: Minimum number of replicas
        activeSessionCount:
          type: integer
          description: Number of active sessions
        deployment:
          type: object
          nullable: true
          description: Details of the current deployment
          properties:
            id:
              type: string
              description: Unique identifier for the deployment
            manifest:
              type: object
              description: Kubernetes manifest for the deployment
              properties:
                apiVersion:
                  type: string
                  description: API version of the manifest
                kind:
                  type: string
                  description: Kind of Kubernetes resource
                metadata:
                  type: object
                  description: Metadata for the resource
                  properties:
                    name:
                      type: string
                      description: Name of the agent
                    namespace:
                      type: string
                      description: Kubernetes namespace
                spec:
                  type: object
                  description: Specification for the agent deployment
                  properties:
                    dailyNodeType:
                      type: string
                      description: Type of node to run on
                    clusterLocal:
                      type: boolean
                      description: Whether the service is cluster-local
                    image:
                      type: string
                      description: Container image used
                    autoScaling:
                      type: object
                      description: Auto-scaling configuration
                      properties:
                        minReplicas:
                          type: integer
                          description: Minimum number of replicas
                        maxReplicas:
                          type: integer
                          description: Maximum number of replicas
                    envFromSecretNames:
                      type: array
                      description: Names of secrets to use as environment variables
                      items:
                        type: string
                    krispModels:
                      type: object
                      description: Krisp noise cancellation configuration (deprecated)
                      properties:
                        enabled:
                          type: boolean
                          description: Whether Krisp is enabled
                    krispViva:
                      type: object
                      description: >-
                        Krisp VIVA noise cancellation configuration in the
                        deployment manifest
                      properties:
                        audioFilters:
                          type: boolean
                          description: Whether Krisp VIVA audio filters are enabled
                        version:
                          type: string
                          description: Version of the Krisp VIVA models
                        modelVars:
                          type: object
                          description: >-
                            Environment variables for Krisp VIVA model
                            configuration
                          properties:
                            KRISP_VIVA_MODEL_PATH:
                              type: string
                              description: Path to the Krisp VIVA model file
                    enableManagedKeys:
                      type: object
                      description: Managed API keys configuration
                      properties:
                        enabled:
                          type: boolean
                          description: Whether managed keys are enabled
            serviceId:
              type: string
              description: ID of the service this deployment belongs to
            createdAt:
              type: string
              format: date-time
              description: Creation timestamp of the deployment
            updatedAt:
              type: string
              format: date-time
              description: Last update timestamp of the deployment
        agentProfile:
          type: string
          description: The agent profile used for resource allocation
          nullable: true
        krispViva:
          type: object
          nullable: true
          description: Krisp VIVA noise cancellation status
          properties:
            audioFilter:
              type: string
              nullable: true
              description: >-
                The currently configured Krisp VIVA audio filter model (tel,
                pro, or null if disabled)
    ErrorResponse:
      type: object
      properties:
        error:
          type: string
          description: Error message
        code:
          type: string
          description: Error code
  securitySchemes:
    PrivateKeyAuth:
      type: http
      scheme: bearer
      description: >-
        Authentication requires a Pipecat Cloud Private API token.


        Generate a Private API key from your Dashboard (Settings > API Keys >
        Private > Create key) and include it as a Bearer token in the
        Authorization header.

````

---

# Source: https://docs.pipecat.ai/deployment/pipecat-cloud/rest-reference/endpoint/agent-delete.md

> ## Documentation Index
> Fetch the complete documentation index at: https://docs.pipecat.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Delete an Agent

> Permanently delete an agent and its associated resources.



## OpenAPI

````yaml DELETE /agents/{agentName}
openapi: 3.0.0
info:
  title: Pipecat Cloud
  version: 1.0.0
  description: Private API for Pipecat Cloud agent management
servers:
  - url: https://api.pipecat.daily.co/v1
    description: API server
security:
  - PrivateKeyAuth: []
paths:
  /agents/{agentName}:
    delete:
      summary: Delete an agent
      description: Permanently removes an agent and its associated Kubernetes resources.
      operationId: deleteService
      parameters:
        - name: agentName
          in: path
          required: true
          description: Name of the agent to delete
          schema:
            type: string
      responses:
        '200':
          description: Agent successfully deleted
          content:
            application/json:
              schema:
                type: object
                properties:
                  status:
                    type: string
                    enum:
                      - OK
                    description: Status of the deletion operation
              example:
                status: OK
        '404':
          description: Agent not found
          content:
            application/json:
              schema:
                $ref: '#/components/schemas/ErrorResponse'
              example:
                error: Not found
                code: NOT_FOUND
        '500':
          description: Internal server error
          content:
            application/json:
              schema:
                $ref: '#/components/schemas/ErrorResponse'
              example:
                error: Internal server error
                code: INTERNAL_SERVER_ERROR
      security:
        - PrivateKeyAuth: []
components:
  schemas:
    ErrorResponse:
      type: object
      properties:
        error:
          type: string
          description: Error message
        code:
          type: string
          description: Error code
  securitySchemes:
    PrivateKeyAuth:
      type: http
      scheme: bearer
      description: >-
        Authentication requires a Pipecat Cloud Private API token.


        Generate a Private API key from your Dashboard (Settings > API Keys >
        Private > Create key) and include it as a Bearer token in the
        Authorization header.

````

---

# Source: https://docs.pipecat.ai/deployment/pipecat-cloud/rest-reference/endpoint/agent-get-logs.md

> ## Documentation Index
> Fetch the complete documentation index at: https://docs.pipecat.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Retrieve Agent Logs

> Get execution logs for an agent with filtering and pagination options.



## OpenAPI

````yaml GET /agents/{agentName}/logs
openapi: 3.0.0
info:
  title: Pipecat Cloud
  version: 1.0.0
  description: Private API for Pipecat Cloud agent management
servers:
  - url: https://api.pipecat.daily.co/v1
    description: API server
security:
  - PrivateKeyAuth: []
paths:
  /agents/{agentName}/logs:
    get:
      summary: Get agent logs
      description: >-
        Retrieves logs for the specified agent with optional filtering and
        pagination.
      operationId: getLogs
      parameters:
        - name: agentName
          in: path
          required: true
          description: Name of the agent to retrieve logs for
          schema:
            type: string
        - name: limit
          in: query
          required: false
          description: Maximum number of log entries to return
          schema:
            type: integer
            default: 10
        - name: offset
          in: query
          required: false
          description: Number of log entries to skip
          schema:
            type: integer
            default: 0
        - name: deploymentId
          in: query
          required: false
          description: Filter logs to a specific deployment ID
          schema:
            type: string
        - name: sessionId
          in: query
          required: false
          description: Filter logs to a specific session ID
          schema:
            type: string
        - name: query
          in: query
          required: false
          description: Free-text search query to filter logs
          schema:
            type: string
        - name: order
          in: query
          required: false
          description: Sort order for logs
          schema:
            type: string
            enum:
              - asc
              - desc
            default: desc
        - name: startTime
          in: query
          required: false
          description: Filter logs to those after this Unix timestamp
          schema:
            type: integer
        - name: endTime
          in: query
          required: false
          description: Filter logs to those before this Unix timestamp
          schema:
            type: integer
      responses:
        '200':
          description: Agent logs retrieved successfully
          content:
            application/json:
              schema:
                type: object
                properties:
                  total:
                    type: object
                    properties:
                      value:
                        type: integer
                        description: Total number of log entries matching the query
                      relation:
                        type: string
                        enum:
                          - eq
                          - gte
                        description: >-
                          Relation type for the total count (eq = exact, gte =
                          greater than or equal)
                  logs:
                    type: array
                    description: List of log entries
                    items:
                      type: object
                      properties:
                        log:
                          type: string
                          description: Log message content
                        timestamp:
                          type: string
                          format: date-time
                          description: Timestamp when the log was generated
                        deploymentId:
                          type: string
                          description: ID of the deployment that generated the log
                          nullable: true
              example:
                total:
                  value: 79
                  relation: eq
                logs:
                  - log: 'INFO:     Finished server process [1]'
                    timestamp: '2025-04-18T20:34:17.153744602Z'
                  - log: 'INFO:     Application shutdown complete.'
                    timestamp: '2025-04-18T20:34:17.153706059Z'
                  - log: 'INFO:     Waiting for application shutdown.'
                    timestamp: '2025-04-18T20:34:17.153611029Z'
                  - log: >-
                      INFO:     Uvicorn running on http://0.0.0.0:8080 (Press
                      CTRL+C to quit)
                    timestamp: '2025-04-18T20:32:42.246099974Z'
                  - log: 'INFO:     Application startup complete.'
                    timestamp: '2025-04-18T20:32:42.245760327Z'
        '400':
          description: Invalid query parameters
          content:
            application/json:
              schema:
                $ref: '#/components/schemas/ErrorResponse'
              example:
                error: Invalid timestamp format
                code: GENERIC_BAD_REQUEST
        '404':
          description: Agent not found
          content:
            application/json:
              schema:
                $ref: '#/components/schemas/ErrorResponse'
              example:
                error: Not found
                code: NOT_FOUND
        '500':
          description: Internal server error
          content:
            application/json:
              schema:
                $ref: '#/components/schemas/ErrorResponse'
              example:
                error: Internal server error
                code: INTERNAL_SERVER_ERROR
      security:
        - PrivateKeyAuth: []
components:
  schemas:
    ErrorResponse:
      type: object
      properties:
        error:
          type: string
          description: Error message
        code:
          type: string
          description: Error code
  securitySchemes:
    PrivateKeyAuth:
      type: http
      scheme: bearer
      description: >-
        Authentication requires a Pipecat Cloud Private API token.


        Generate a Private API key from your Dashboard (Settings > API Keys >
        Private > Create key) and include it as a Bearer token in the
        Authorization header.

````

---

# Source: https://docs.pipecat.ai/deployment/pipecat-cloud/rest-reference/endpoint/agent-get-sessions.md

> ## Documentation Index
> Fetch the complete documentation index at: https://docs.pipecat.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Retrieve Agent Sessions

> Get sessions for an agent with filtering and pagination options.



## OpenAPI

````yaml GET /agents/{agentName}/sessions
openapi: 3.0.0
info:
  title: Pipecat Cloud
  version: 1.0.0
  description: Private API for Pipecat Cloud agent management
servers:
  - url: https://api.pipecat.daily.co/v1
    description: API server
security:
  - PrivateKeyAuth: []
paths:
  /agents/{agentName}/sessions:
    get:
      summary: Get agent sessions
      description: >-
        Retrieves sessions for the specified agent with optional filtering and
        pagination.
      operationId: getSessions
      parameters:
        - name: agentName
          in: path
          required: true
          description: Name of the agent to retrieve sessions for
          schema:
            type: string
        - name: limit
          in: query
          required: false
          description: Maximum number of sessions to return
          schema:
            type: integer
            default: 10
        - name: offset
          in: query
          required: false
          description: Number of sessions to skip
          schema:
            type: integer
            default: 0
        - name: deploymentId
          in: query
          required: false
          description: Filter sessions to a specific deployment ID
          schema:
            type: string
        - name: startTime
          in: query
          required: false
          description: Filter sessions to those after this Unix timestamp
          schema:
            type: integer
        - name: endTime
          in: query
          required: false
          description: Filter sessions to those before this Unix timestamp
          schema:
            type: integer
        - name: status
          in: query
          required: false
          description: Filter sessions to those with this status
          schema:
            type: string
            enum:
              - active
              - ended
              - all
            default: all
        - name: startType
          in: query
          required: false
          description: Filter sessions to those with this start type
          schema:
            type: string
            enum:
              - cold
              - warm
              - all
            default: all
        - name: startTimeMin
          in: query
          required: false
          description: >-
            Filter sessions to those whose start times are greater than this
            value in seconds
          schema:
            type: integer
            default: 0
        - name: startTimeMax
          in: query
          required: false
          description: >-
            Filter sessions to those whose start times are less than this value
            in seconds
          schema:
            type: integer
            default: 0
      responses:
        '200':
          description: Agent sessions retrieved successfully
          content:
            application/json:
              schema:
                type: object
                properties:
                  total_count:
                    type: integer
                    description: Total number of sessions matching the parameters
                  sessions:
                    type: array
                    description: List of sessions
                    items:
                      type: object
                      properties:
                        sessionId:
                          type: string
                          description: ID of the session
                        serviceId:
                          type: string
                          description: ID of the service for this session
                        organizationId:
                          type: string
                          description: >-
                            ID of the organization that this session is
                            associated with
                        deploymentId:
                          type: string
                          description: >-
                            ID of the deployment that this session is associated
                            with
                        endedAt:
                          type: string
                          format: date-time
                          description: Timestamp when the log was generated
                        botStartSeconds:
                          type: integer
                          description: >-
                            Number of seconds between bot start request and bot
                            start
                          nullable: true
                        coldStart:
                          type: boolean
                          description: >-
                            Whether the bot start was a cold start (see
                            https://docs.pipecat.ai/deployment/pipecat-cloud/fundamentals/scaling#cold-starts)
                          nullable: true
                        completionStatus:
                          type: string
                          description: >-
                            The completion status of the session. This can be an
                            http return code or a string like `HTTP_COMPLETED`,
                            `HTTP_ERROR`, `WS_CONNECTION_CLOSED`, among others
                          nullable: true
                        createdAt:
                          type: string
                          format: date-time
                          description: Timestamp when the session was created
                        updatedAt:
                          type: string
                          format: date-time
                          description: Timestamp when the session was last updated
              example:
                total_count: 330
                sessions:
                  - sessionId: 800d444a-e67e-4686-afb3-b0de509cccb3
                    serviceId: b900ae4a-f710-4ef1-95d8-c5f5b2789da9
                    organizationId: cc71b52d-4271-46c9-87be-cac600fd821a
                    deploymentId: 81f4da55-106d-401f-b069-a1f1992f871a
                    endedAt: '2025-10-22T08:39:26.000Z'
                    botStartSeconds: 0
                    coldStart: false
                    completionStatus: HTTP_COMPLETED
                    createdAt: '2025-10-22T08:39:26.000Z'
                    updatedAt: '2025-10-22T08:39:26.000Z'
                  - sessionId: 803349cc-2ae0-437a-9a72-408817299e43
                    serviceId: b900ae4a-f710-4ef1-95d8-c5f5b2789da9
                    organizationId: cc71b52d-4271-46c9-87be-cac600fd821a
                    deploymentId: 81f4da55-106d-401f-b069-a1f1992f871a
                    endedAt: '2025-10-22T08:38:09.000Z'
                    botStartSeconds: 2
                    coldStart: true
                    completionStatus: HTTP_COMPLETED
                    createdAt: '2025-10-22T08:38:09.000Z'
                    updatedAt: '2025-10-22T08:38:09.000Z'
                  - sessionId: 44003e83-b679-46ef-88ec-18e9ea9dd81d
                    serviceId: b900ae4a-f710-4ef1-95d8-c5f5b2789da9
                    organizationId: cc71b52d-4271-46c9-87be-cac600fd821a
                    deploymentId: f76bae42-5ce8-4ae3-8a75-0349bfd7ec63
                    endedAt: '2025-10-21T08:17:47.000Z'
                    botStartSeconds: 0
                    coldStart: false
                    completionStatus: HTTP_ERROR
                    createdAt: '2025-10-21T08:17:22.000Z'
                    updatedAt: '2025-10-21T08:17:38.000Z'
        '400':
          description: Invalid query parameters
          content:
            application/json:
              schema:
                $ref: '#/components/schemas/ErrorResponse'
              example:
                error: Invalid parameters
                err:
                  issues:
                    - received: bad_value
                      code: invalid_enum_value
                      options:
                        - active
                        - ended
                        - all
                      path:
                        - status
                      message: >-
                        Invalid enum value. Expected 'active' | 'ended' | 'all',
                        received 'bad_value'
                  name: ZodError
                code: '400'
        '404':
          description: Agent not found
          content:
            application/json:
              schema:
                $ref: '#/components/schemas/ErrorResponse'
              example:
                error: >-
                  API endpoint not found / agent deployment not found /
                  Organization not found
                code: '404'
        '500':
          description: Internal server error
          content:
            application/json:
              schema:
                $ref: '#/components/schemas/ErrorResponse'
              example:
                error: >-
                  Internal server error. Please check logs for more information
                  or contact support.
                code: '500'
      security:
        - PrivateKeyAuth: []
components:
  schemas:
    ErrorResponse:
      type: object
      properties:
        error:
          type: string
          description: Error message
        code:
          type: string
          description: Error code
  securitySchemes:
    PrivateKeyAuth:
      type: http
      scheme: bearer
      description: >-
        Authentication requires a Pipecat Cloud Private API token.


        Generate a Private API key from your Dashboard (Settings > API Keys >
        Private > Create key) and include it as a Bearer token in the
        Authorization header.

````

---

# Source: https://docs.pipecat.ai/deployment/pipecat-cloud/fundamentals/agent-images.md

> ## Documentation Index
> Fetch the complete documentation index at: https://docs.pipecat.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Agent Images

> How to containerize your agent project

Pipecat Cloud agents are designed to be run from containerized images. This allows you to run the agent in a controlled environment, with all the dependencies and configurations needed.

Your project defines the environment that your agent will run using Docker and built using a Dockerfile in the root directory of the project.

For example, your Dockerfile might look like this:

<Tabs>
  <Tab title="Dockerfile using uv">
    ```dockerfile  theme={null}
    FROM dailyco/pipecat-base:latest

    # Enable bytecode compilation
    ENV UV_COMPILE_BYTECODE=1

    # Copy from the cache instead of linking since it's a mounted volume
    ENV UV_LINK_MODE=copy

    # Uncomment this if you wish to print a summary of the features available in the base image.
    # ENV PCC_LOG_FEATURES_SUMMARY=true

    # Install the project's dependencies using the lockfile and settings
    RUN --mount=type=cache,target=/root/.cache/uv \
        --mount=type=bind,source=uv.lock,target=uv.lock \
        --mount=type=bind,source=pyproject.toml,target=pyproject.toml \
        uv sync --locked --no-install-project --no-dev

    # Copy the application code
    COPY ./bot.py bot.py
    ```
  </Tab>

  <Tab title="Dockerfile using pip">
    ```dockerfile  theme={null}
    FROM dailyco/pipecat-base:latest

    COPY ./requirements.txt requirements.txt

    # Uncomment this if you wish to print a summary of the features available in the base image.
    # ENV PCC_LOG_FEATURES_SUMMARY=true

    RUN pip install --no-cache-dir --upgrade -r requirements.txt

    COPY ./bot.py bot.py
    ```
  </Tab>
</Tabs>

<Frame>
  <img src="https://mintcdn.com/daily/2bYrACcmgvvzC075/deployment/pipecat-cloud/images/agent-anatomy.png?fit=max&auto=format&n=2bYrACcmgvvzC075&q=85&s=bc88f08e1595415dcaad3ab0f8142364" data-og-width="1220" width="1220" data-og-height="746" height="746" data-path="deployment/pipecat-cloud/images/agent-anatomy.png" data-optimize="true" data-opv="3" srcset="https://mintcdn.com/daily/2bYrACcmgvvzC075/deployment/pipecat-cloud/images/agent-anatomy.png?w=280&fit=max&auto=format&n=2bYrACcmgvvzC075&q=85&s=aa0b73e2953a018524c71cf2229300fc 280w, https://mintcdn.com/daily/2bYrACcmgvvzC075/deployment/pipecat-cloud/images/agent-anatomy.png?w=560&fit=max&auto=format&n=2bYrACcmgvvzC075&q=85&s=a31591a55f14651292113d9c06eacd98 560w, https://mintcdn.com/daily/2bYrACcmgvvzC075/deployment/pipecat-cloud/images/agent-anatomy.png?w=840&fit=max&auto=format&n=2bYrACcmgvvzC075&q=85&s=a54bbc8c1ed70ac2a7cd4b27c36bfbb7 840w, https://mintcdn.com/daily/2bYrACcmgvvzC075/deployment/pipecat-cloud/images/agent-anatomy.png?w=1100&fit=max&auto=format&n=2bYrACcmgvvzC075&q=85&s=eff208e45488762d78e49ac271dd4af0 1100w, https://mintcdn.com/daily/2bYrACcmgvvzC075/deployment/pipecat-cloud/images/agent-anatomy.png?w=1650&fit=max&auto=format&n=2bYrACcmgvvzC075&q=85&s=e63339ba853433a56eea9dc325bebe7c 1650w, https://mintcdn.com/daily/2bYrACcmgvvzC075/deployment/pipecat-cloud/images/agent-anatomy.png?w=2500&fit=max&auto=format&n=2bYrACcmgvvzC075&q=85&s=04f8f236b8226786728223986f0ded64 2500w" />
</Frame>

## Using an official base image

Pipecat Cloud provides a series of base images that we recommend for most use-cases. Base images provide:

* Simplified development and deployment
* Optimizations for performance and security
* Pre-installed system dependencies for most multi-modal agent use-cases

Using a base image reduces complexity in your project but requires you to adhere to a specific project structure.

* Your project must contain a `bot.py` file that defines the agent pipeline
* The `bot.py` must contain a `bot()` method that is the entry point for your agent pipeline
* The `bot()` method must be asynchronous, e.g. `async def bot():`

You do not need to specify a `CMD` as part of your Dockerfile - the base image is configured to run your `bot.py` module.

You can browse available base images in the [Pipecat Cloud Docker Hub <Icon icon="arrow-up-right-from-square" iconType="solid" />](https://hub.docker.com/u/dailyco).

### Reserved paths

The base image uses the `/app` directory for its internal operation. **Avoid copying files to `/app`** in your Dockerfile to prevent conflicts with system files.

<Warning>
  Writing to `/app` may overwrite critical system files and cause your
  deployment to fail. Your agent code should be placed in `bot.py` (required)
  and any additional modules, which the base image handles automatically.
</Warning>

### Reserved HTTP routes

The base image exposes the following HTTP routes for Pipecat Cloud platform integration:

| Route        | Method    | Description                                                                   |
| ------------ | --------- | ----------------------------------------------------------------------------- |
| `/bot`       | POST      | Main entry point called by Pipecat Cloud to start agent sessions              |
| `/ws`        | WebSocket | WebSocket endpoint for real-time communication (e.g., telephony integrations) |
| `/api/offer` | POST      | SmallWebRTC offer handling for peer-to-peer connections                       |
| `/api/offer` | PATCH     | SmallWebRTC ICE candidate handling                                            |
| `/whatsapp`  | POST      | WhatsApp Business webhook endpoint                                            |

<Note>
  These routes are automatically configured based on available features. For
  example, the `/whatsapp` route is only available when WhatsApp environment
  variables are configured.
</Note>

### Reserved environment variables

The base image uses the following environment variables for configuration:

| Variable                   | Description                                                                 |
| -------------------------- | --------------------------------------------------------------------------- |
| `PORT`                     | HTTP server port (default: `8080`)                                          |
| `SHUTDOWN_TIMEOUT`         | Server shutdown timeout in seconds (default: `7200`)                        |
| `PIPECAT_LOG_LEVEL`        | Pipecat logging level: `TRACE`, `DEBUG`, `INFO`, `WARNING`, `ERROR`, `NONE` |
| `PCC_LOG_FEATURES_SUMMARY` | Set to `true` to log available features on startup                          |
| `IMAGE_VERSION`            | Set automatically during build to track image versions                      |
| `ESP32_ENABLED`            | Enable ESP32 mode for SmallWebRTC                                           |
| `ESP32_HOST`               | ESP32 host address                                                          |
| `ICE_CONFIG_URL`           | ICE server configuration endpoint                                           |

For WhatsApp integration, the following environment variables are required:

| Variable                   | Description                                  |
| -------------------------- | -------------------------------------------- |
| `WHATSAPP_TOKEN`           | WhatsApp API access token                    |
| `WHATSAPP_PHONE_NUMBER_ID` | WhatsApp Business phone number ID            |
| `WHATSAPP_APP_SECRET`      | WhatsApp app secret for webhook verification |

### Logging available features

To see which features are available in the base image at startup, set `PCC_LOG_FEATURES_SUMMARY=true`. This outputs a summary like:

```
2025-10-22 22:04:41.411 | INFO     | feature_manager:log_features_summary:287 - ============================================================
2025-10-22 22:04:41.411 | INFO     | feature_manager:log_features_summary:288 - Features available in this base image:
2025-10-22 22:04:41.412 | INFO     | feature_manager:log_features_summary:289 - ============================================================
2025-10-22 22:04:41.412 | INFO     | feature_manager:log_features_summary:299 - ✅ Daily Transport: ENABLED
2025-10-22 22:04:41.412 | INFO     | feature_manager:log_features_summary:299 - ✅ Websocket Transport: ENABLED
2025-10-22 22:04:41.412 | INFO     | feature_manager:log_features_summary:299 - ✅ SmallWebRTC Session Arguments: ENABLED
2025-10-22 22:04:41.412 | INFO     | feature_manager:log_features_summary:299 - ✅ SmallWebRTC Transport: ENABLED
2025-10-22 22:04:41.412 | INFO     | feature_manager:log_features_summary:299 - ✅ SmallWebRTC ICE Candidates: ENABLED
2025-10-22 22:04:41.412 | INFO     | feature_manager:log_features_summary:299 - ✅ WhatsApp Integration: ENABLED
2025-10-22 22:04:41.412 | INFO     | feature_manager:log_features_summary:314 - ============================================================
```

### Available base images

| Name                   | Description                                               |
| ---------------------- | --------------------------------------------------------- |
| `dailyco/pipecat-base` | Multi-modal Pipecat optimized, suitable for most use-case |

The base image supports multiple Python versions. Starting with version 0.1.0, the default Python version is 3.12 (previously 3.10).

**Latest tags:**

* `dailyco/pipecat-base:latest` (Python 3.12, default)
* `dailyco/pipecat-base:latest-py3.10` (Python 3.10)
* `dailyco/pipecat-base:latest-py3.11` (Python 3.11)
* `dailyco/pipecat-base:latest-py3.12` (Python 3.12)
* `dailyco/pipecat-base:latest-py3.13` (Python 3.13)

**Versioned tags:**

* `dailyco/pipecat-base:0.1.0` (Python 3.12, default)
* `dailyco/pipecat-base:0.1.0-py3.10` (Python 3.10)
* `dailyco/pipecat-base:0.1.0-py3.11` (Python 3.11)
* `dailyco/pipecat-base:0.1.0-py3.12` (Python 3.12)
* `dailyco/pipecat-base:0.1.0-py3.13` (Python 3.13)

<Tip>For production use, we recommend pinning to specific versions.</Tip>

## Using a custom image

For more complex use-cases, you can use a custom image.

When doing so, we recommend following best practices to ensure your agent instance runs optimally on the platform.

Our [base image](https://github.com/daily-co/pipecat-cloud-images/tree/main/pipecat-base) is open source and serves as a useful blueprint for configuring your custom agent image.

### Agent image structure

<Warning>
  Custom agent images are for advanced use cases. For most teams,
  we recommend using our base images. If needed, consult the base
  image code as a reference.

  For unsupported use cases, contact us at [help@daily.co](mailto:help@daily.co) or via
  [Discord](https://discord.gg/dailyco).
</Warning>

Pipecat Cloud agent images must adhere to a specific structure to run on the platform. Our base images abstract away much of this complexity, but if you are building a custom image, you must ensure your agent adheres to the following:

* HTTP API that can handle requests from the platform to configure and run agent instances.
* The necessary system level dependencies (such as Python.)

In order to start an instance of your custom agent, you must expose a HTTP `POST /bot` route that will be called by the platform.

We recommend using FastAPI to create this route. Please refer to the base image code for an example of how to do this.

## Building the image

Pipecat Cloud requires all images to be built to target Linux on ARM. This is the most common platform for cloud deployments.

```bash  theme={null}
docker build --platform linux/arm64 -t my-agent:latest .
```

Your agent image should include:

* All dependencies required for your agent to run.
* Assets (such as images or models) available in the container filesystem
* The entry point for your agent (usually a Python script)
* Additional system dependencies (if required)

### Best practices

* Keep your image as small as possible. Use multi-stage builds to reduce the size of the final image.
* Use a `.dockerignore` file to exclude unnecessary files from the image.
* Pipecat Cloud will automatically restart your agent if it crashes. Ensure your agent can handle this gracefully.
* Use [Secrets](./secrets) to securely store sensitive information in your agent image.
* To optimize for fast start-ups, avoid long running or blocking processes during initialization.

---

# Source: https://docs.pipecat.ai/deployment/pipecat-cloud/rest-reference/endpoint/agent-list-all.md

> ## Documentation Index
> Fetch the complete documentation index at: https://docs.pipecat.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# List All Agents

> Retrieve a list of all agents in your organization with their status and configuration.



## OpenAPI

````yaml GET /agents
openapi: 3.0.0
info:
  title: Pipecat Cloud
  version: 1.0.0
  description: Private API for Pipecat Cloud agent management
servers:
  - url: https://api.pipecat.daily.co/v1
    description: API server
security:
  - PrivateKeyAuth: []
paths:
  /agents:
    get:
      summary: List all agents
      operationId: listServices
      parameters:
        - name: includeActiveDeployment
          in: query
          required: false
          description: Whether to include the active deployment details in the response
          schema:
            type: boolean
        - name: region
          in: query
          required: false
          description: >-
            Filter agents by region. If not specified, returns agents from all
            regions.
          schema:
            type: string
            example: us-west
      responses:
        '200':
          description: List of agents retrieved successfully
          content:
            application/json:
              schema:
                type: object
                properties:
                  services:
                    type: array
                    items:
                      $ref: '#/components/schemas/ServiceListItem'
              example:
                services:
                  - id: c359e1ea-64d6-4bcf-a7c6-28d3bd1c8909
                    name: voice-starter
                    region: us-west
                    activeDeploymentId: 4cab03c7-8f53-418d-9c3c-91a1265d179e
                    organizationId: 7c489df3-7d1d-482f-b1ed-51300f630645
                    createdAt: '2025-04-09T15:01:17.414Z'
                    updatedAt: '2025-04-18T20:29:19.181Z'
                    deletedAt: null
                  - id: 4e0d1f30-77a8-4283-a054-14a63b06720f
                    name: voice-starter-krisp
                    region: us-west
                    activeDeploymentId: 62e89096-23c6-4265-984b-f419696c58da
                    organizationId: 7c489df3-7d1d-482f-b1ed-51300f630645
                    createdAt: '2025-04-09T15:59:50.465Z'
                    updatedAt: '2025-04-11T16:54:54.246Z'
                    deletedAt: null
        '500':
          description: Internal server error
          content:
            application/json:
              schema:
                $ref: '#/components/schemas/ErrorResponse'
      security:
        - PrivateKeyAuth: []
components:
  schemas:
    ServiceListItem:
      type: object
      properties:
        id:
          type: string
          description: Unique identifier for the agent
        name:
          type: string
          description: Name of the agent
        region:
          type: string
          description: The region where the agent is deployed
        activeDeploymentId:
          type: string
          description: ID of the active deployment
        organizationId:
          type: string
          description: Organization ID this agent belongs to
        createdAt:
          type: string
          format: date-time
          description: Creation timestamp of the agent
        updatedAt:
          type: string
          format: date-time
          description: Last update timestamp
        deletedAt:
          type: string
          format: date-time
          description: Deletion timestamp, if the agent has been deleted
          nullable: true
        deployment:
          type: object
          description: >-
            Details of the active deployment (only included when
            includeActiveDeployment=true)
          properties:
            id:
              type: string
              description: Deployment ID
            serviceId:
              type: string
              description: Service ID
            manifest:
              type: object
              description: Kubernetes manifest for the deployment
            createdAt:
              type: string
              format: date-time
              description: Creation timestamp of the deployment
            updatedAt:
              type: string
              format: date-time
              description: Last update timestamp of the deployment
    ErrorResponse:
      type: object
      properties:
        error:
          type: string
          description: Error message
        code:
          type: string
          description: Error code
  securitySchemes:
    PrivateKeyAuth:
      type: http
      scheme: bearer
      description: >-
        Authentication requires a Pipecat Cloud Private API token.


        Generate a Private API key from your Dashboard (Settings > API Keys >
        Private > Create key) and include it as a Bearer token in the
        Authorization header.

````

---

# Source: https://docs.pipecat.ai/deployment/pipecat-cloud/rest-reference/endpoint/agent-list-one.md

> ## Documentation Index
> Fetch the complete documentation index at: https://docs.pipecat.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Get Agent Details

> Retrieve detailed information about a specific agent including its deployment status.



## OpenAPI

````yaml GET /agents/{agentName}
openapi: 3.0.0
info:
  title: Pipecat Cloud
  version: 1.0.0
  description: Private API for Pipecat Cloud agent management
servers:
  - url: https://api.pipecat.daily.co/v1
    description: API server
security:
  - PrivateKeyAuth: []
paths:
  /agents/{agentName}:
    get:
      summary: Get agent details
      operationId: getService
      parameters:
        - name: agentName
          in: path
          required: true
          description: Name of the agent to retrieve
          schema:
            type: string
      responses:
        '200':
          description: Agent details retrieved successfully
          content:
            application/json:
              schema:
                $ref: '#/components/schemas/ServiceDetailsResponse'
              example:
                name: voice-starter
                region: us-west
                ready: true
                createdAt: '2025-04-09T15:01:17.414Z'
                updatedAt: '2025-04-18T20:29:19.181Z'
                activeDeploymentId: 4cab03c7-8f53-418d-9c3c-91a1265d179e
                activeDeploymentReady: true
                autoScaling:
                  maxReplicas: 10
                  minReplicas: 1
                activeSessionCount: 0
                deployment:
                  id: 4cab03c7-8f53-418d-9c3c-91a1265d179e
                  manifest:
                    apiVersion: pipecatcloud.daily.co/v1
                    kind: PCService
                    metadata:
                      name: voice-starter
                      namespace: tiny-ferret-maroon-123
                    spec:
                      dailyNodeType: arm
                      clusterLocal: true
                      image: your-dockername/voice-starter:0.1
                      autoScaling:
                        minReplicas: 1
                        maxReplicas: 10
                      envFromSecretNames:
                        - voice-starter-secrets
                      krispModels:
                        enabled: false
                      krispViva:
                        audioFilters: true
                        version: '20251010'
                      enableManagedKeys:
                        enabled: false
                  serviceId: c359e1ea-64d6-4bcf-a7c6-28d3bd1c8909
                  createdAt: '2025-04-18T20:29:19.177Z'
                  updatedAt: '2025-04-18T20:29:19.177Z'
                agentProfile: agent-1x
        '404':
          description: Agent not found
          content:
            application/json:
              schema:
                $ref: '#/components/schemas/ErrorResponse'
        '500':
          description: Internal server error
          content:
            application/json:
              schema:
                $ref: '#/components/schemas/ErrorResponse'
      security:
        - PrivateKeyAuth: []
components:
  schemas:
    ServiceDetailsResponse:
      type: object
      properties:
        name:
          type: string
          description: Name of the agent
        region:
          type: string
          description: The region where the agent is deployed
        ready:
          type: boolean
          description: Whether the agent is ready to accept sessions
        createdAt:
          type: string
          format: date-time
          description: Creation timestamp of the agent
        updatedAt:
          type: string
          format: date-time
          description: Last update timestamp
        activeDeploymentId:
          type: string
          description: ID of the active deployment
        activeDeploymentReady:
          type: boolean
          description: Whether the active deployment is ready
        autoScaling:
          type: object
          properties:
            maxReplicas:
              type: integer
              description: Maximum number of replicas
            minReplicas:
              type: integer
              description: Minimum number of replicas
        activeSessionCount:
          type: integer
          description: Number of active sessions
        deployment:
          type: object
          description: Details of the current deployment
          properties:
            id:
              type: string
              description: Deployment ID
            manifest:
              type: object
              description: Kubernetes manifest for the deployment
              properties:
                apiVersion:
                  type: string
                  description: API version of the manifest
                kind:
                  type: string
                  description: Kind of Kubernetes resource
                metadata:
                  type: object
                  description: Metadata for the resource
                  properties:
                    name:
                      type: string
                      description: Name of the agent
                    namespace:
                      type: string
                      description: Kubernetes namespace
                spec:
                  type: object
                  description: Specification for the agent deployment
                  properties:
                    dailyNodeType:
                      type: string
                      description: Type of node to run on
                    clusterLocal:
                      type: boolean
                      description: Whether the service is cluster-local
                    image:
                      type: string
                      description: Container image used
                    autoScaling:
                      type: object
                      description: Auto-scaling configuration
                      properties:
                        minReplicas:
                          type: integer
                          description: Minimum number of replicas
                        maxReplicas:
                          type: integer
                          description: Maximum number of replicas
                    envFromSecretNames:
                      type: array
                      description: Names of secrets to use as environment variables
                      items:
                        type: string
                    krispModels:
                      type: object
                      description: Krisp noise cancellation configuration
                      properties:
                        enabled:
                          type: boolean
                          description: Whether Krisp is enabled
                    krispViva:
                      type: object
                      description: Krisp VIVA noise cancellation configuration
                      properties:
                        audioFilters:
                          type: boolean
                          description: Whether Krisp VIVA audio filters are enabled
                        version:
                          type: string
                          description: Version of the Krisp VIVA models
                    enableManagedKeys:
                      type: object
                      description: Managed API keys configuration
                      properties:
                        enabled:
                          type: boolean
                          description: Whether managed keys are enabled
            serviceId:
              type: string
              description: Service ID
            createdAt:
              type: string
              format: date-time
              description: Creation timestamp of the deployment
            updatedAt:
              type: string
              format: date-time
              description: Last update timestamp of the deployment
        agentProfile:
          type: string
          description: The agent profile used for resource allocation
          nullable: true
    ErrorResponse:
      type: object
      properties:
        error:
          type: string
          description: Error message
        code:
          type: string
          description: Error code
  securitySchemes:
    PrivateKeyAuth:
      type: http
      scheme: bearer
      description: >-
        Authentication requires a Pipecat Cloud Private API token.


        Generate a Private API key from your Dashboard (Settings > API Keys >
        Private > Create key) and include it as a Bearer token in the
        Authorization header.

````

---

# Source: https://docs.pipecat.ai/deployment/pipecat-cloud/rest-reference/endpoint/agent-update.md

> ## Documentation Index
> Fetch the complete documentation index at: https://docs.pipecat.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Update an Agent

> Update an existing agent's configuration or deploy a new version.



## OpenAPI

````yaml POST /agents/{agentName}
openapi: 3.0.0
info:
  title: Pipecat Cloud
  version: 1.0.0
  description: Private API for Pipecat Cloud agent management
servers:
  - url: https://api.pipecat.daily.co/v1
    description: API server
security:
  - PrivateKeyAuth: []
paths:
  /agents/{agentName}:
    post:
      summary: Update an agent deployment
      operationId: updateService
      parameters:
        - name: agentName
          in: path
          required: true
          description: Name of the agent to update
          schema:
            type: string
      requestBody:
        required: true
        content:
          application/json:
            schema:
              $ref: '#/components/schemas/UpdateServiceRequest'
            example:
              image: your-dockername/voice-starter:0.1
              nodeType: arm
              imagePullSecretSet: dockerhub-credentials
              secretSet: voice-starter-secrets
              autoScaling:
                minAgents: 1
                maxAgents: 10
              krispViva:
                audioFilter: tel
              agentProfile: agent-1x
              enableManagedKeys: false
      responses:
        '200':
          description: Agent deployment updated successfully
          content:
            application/json:
              schema:
                $ref: '#/components/schemas/ServiceDetailsResponse'
              example:
                name: voice-starter
                region: us-west
                ready: true
                createdAt: '2025-04-19T01:20:27.564Z'
                updatedAt: '2025-04-19T01:25:47.229Z'
                activeDeploymentId: 19db578d-808c-420a-a047-87edde4410c1
                activeDeploymentReady: false
                autoScaling:
                  maxReplicas: 10
                  minReplicas: 1
                activeSessionCount: 0
                agentProfile: agent-1x
                deployment:
                  id: 19db578d-808c-420a-a047-87edde4410c1
                  manifest:
                    apiVersion: pipecatcloud.daily.co/v1
                    kind: PCService
                    metadata:
                      name: voice-starter
                      namespace: tiny-ferret-maroon-123
                    spec:
                      dailyNodeType: arm
                      deploymentMode: keda
                      clusterLocal: true
                      image: your-dockername/voice-starter:0.1
                      autoScaling:
                        minReplicas: 1
                        maxReplicas: 10
                      envFromSecretNames:
                        - voice-starter-secrets
                      krispModels:
                        enabled: false
                      krispViva:
                        audioFilters: true
                        version: '20251010'
                        modelVars:
                          KRISP_VIVA_MODEL_PATH: audio_filters/krisp-viva-tel-v2.kef
                      integratedKeysProxy:
                        enabled: false
                      resources:
                        cpu: 500m
                        memory: 1Gi
                  serviceId: b59a68ee-61c8-4d99-9ceb-e99a3953bdac
                  createdAt: '2025-04-19T01:25:47.225Z'
                  updatedAt: '2025-04-19T01:25:47.225Z'
                krispViva:
                  audioFilter: tel
        '400':
          description: Invalid request
          content:
            application/json:
              schema:
                $ref: '#/components/schemas/ErrorResponse'
              examples:
                invalidParameters:
                  summary: Invalid parameters
                  value:
                    error: Invalid parameters
                    code: GENERIC_BAD_REQUEST
                secretRegionMismatch:
                  summary: Secret set region mismatch
                  value:
                    error: >-
                      Secret set 'my-secrets' is in region 'us-east' but service
                      is in region 'us-west'. Secrets must be in the same region
                      as the service.
                    code: GENERIC_BAD_REQUEST
                imagePullSecretRegionMismatch:
                  summary: Image pull secret region mismatch
                  value:
                    error: >-
                      Image pull secret set 'dockerhub-creds' is in region
                      'us-east' but service is in region 'us-west'. Image pull
                      secrets must be in the same region as the service.
                    code: GENERIC_BAD_REQUEST
        '404':
          description: Agent or referenced secret not found
          content:
            application/json:
              schema:
                $ref: '#/components/schemas/ErrorResponse'
              example:
                error: Service not found
                code: NOT_FOUND
        '500':
          description: Internal server error
          content:
            application/json:
              schema:
                $ref: '#/components/schemas/ErrorResponse'
              example:
                error: Internal server error
                code: INTERNAL_SERVER_ERROR
      security:
        - PrivateKeyAuth: []
components:
  schemas:
    UpdateServiceRequest:
      type: object
      properties:
        image:
          type: string
          description: The container image to use for the agent
          example: your-username/my-agent:latest
        nodeType:
          type: string
          description: >-
            The type of node to run the agent on. Only `arm` is supported at
            this time.
          default: arm
          example: arm
        imagePullSecretSet:
          type: string
          description: The name of the image pull secret set to use
          example: dockerhub-credentials
        secretSet:
          type: string
          description: The name of the secret set to use
          example: my-agent-secrets
        autoScaling:
          type: object
          description: Auto-scaling configuration for the agent
          properties:
            minAgents:
              type: integer
              description: Minimum number of agents
              example: 1
            maxAgents:
              type: integer
              description: 'Maximum number of agents (Default: 50)'
              example: 10
        enableKrisp:
          type: boolean
          deprecated: true
          description: >-
            Whether to enable Krisp noise cancellation. **Deprecated:** Use
            `krispViva` instead for the latest Krisp VIVA models.
          example: false
        krispViva:
          type: object
          description: >-
            Krisp VIVA noise cancellation configuration. [Learn
            more](/deployment/pipecat-cloud/guides/krisp-viva).
          properties:
            audioFilter:
              type: string
              nullable: true
              description: >-
                The Krisp VIVA audio filter model to use:


                - `tel`: Optimized for telephony, cellular, landline, mobile,
                desktop, and browser (up to 16kHz)

                - `pro`: Optimized for mobile, desktop, and browser with WebRTC
                (up to 32kHz)

                - `null`: Disables Krisp VIVA


                By default, Pipecat processes input audio at 16kHz, making `tel`
                appropriate for most use cases.
              enum:
                - tel
                - pro
                - null
              example: tel
        agentProfile:
          type: string
          description: >-
            The agent profile to use for resource allocation. Valid values are:


            - `agent-1x`: 0.5 vCPU and 1 GB of memory. Best for voice agents.

            - `agent-2x`: 1 vCPU and 2 GB of memory. Well suited for voice and
            video agents or voice agents requiring extra processing.

            - `agent-3x`: 1.5 vCPU and 3 GB of memory. Best for voice and video
            agents requiring extra processing or multiple video inputs.
          enum:
            - agent-1x
            - agent-2x
            - agent-3x
          default: agent-1x
          example: agent-1x
        enableManagedKeys:
          type: boolean
          description: >-
            Pipecat Cloud provides API keys for some Pipecat services that you
            can use directly in your applications. [Learn
            more](/deployment/pipecat-cloud/guides/managed-api-keys).
          default: false
          example: false
    ServiceDetailsResponse:
      type: object
      properties:
        name:
          type: string
          description: Name of the agent
        region:
          type: string
          description: The region where the agent is deployed
        ready:
          type: boolean
          description: Whether the agent is ready to accept sessions
        createdAt:
          type: string
          format: date-time
          description: Creation timestamp of the agent
        updatedAt:
          type: string
          format: date-time
          description: Last update timestamp
        activeDeploymentId:
          type: string
          description: ID of the active deployment
        activeDeploymentReady:
          type: boolean
          description: Whether the active deployment is ready
        autoScaling:
          type: object
          properties:
            maxReplicas:
              type: integer
              description: Maximum number of agent replicas
            minReplicas:
              type: integer
              description: Minimum number of agent replicas
        activeSessionCount:
          type: integer
          description: Number of active sessions
        deployment:
          type: object
          nullable: true
          description: Details of the current deployment
          properties:
            id:
              type: string
              description: Unique identifier for the deployment
            manifest:
              type: object
              description: Kubernetes manifest for the deployment
              properties:
                apiVersion:
                  type: string
                  description: API version of the manifest
                kind:
                  type: string
                  description: Kind of Kubernetes resource
                metadata:
                  type: object
                  description: Metadata for the resource
                  properties:
                    name:
                      type: string
                      description: Name of the agent
                    namespace:
                      type: string
                      description: Kubernetes namespace
                spec:
                  type: object
                  description: Specification for the agent deployment
                  properties:
                    dailyNodeType:
                      type: string
                      description: Type of node to run on
                    clusterLocal:
                      type: boolean
                      description: Whether the service is cluster-local
                    image:
                      type: string
                      description: Container image used
                    autoScaling:
                      type: object
                      description: Auto-scaling configuration
                      properties:
                        minReplicas:
                          type: integer
                          description: Minimum number of replicas
                        maxReplicas:
                          type: integer
                          description: Maximum number of replicas
                    envFromSecretNames:
                      type: array
                      description: Names of secrets to use as environment variables
                      items:
                        type: string
                    krispModels:
                      type: object
                      description: Krisp noise cancellation configuration (deprecated)
                      properties:
                        enabled:
                          type: boolean
                          description: Whether Krisp is enabled
                    krispViva:
                      type: object
                      description: >-
                        Krisp VIVA noise cancellation configuration in the
                        deployment manifest
                      properties:
                        audioFilters:
                          type: boolean
                          description: Whether Krisp VIVA audio filters are enabled
                        version:
                          type: string
                          description: Version of the Krisp VIVA models
                        modelVars:
                          type: object
                          description: >-
                            Environment variables for Krisp VIVA model
                            configuration
                          properties:
                            KRISP_VIVA_MODEL_PATH:
                              type: string
                              description: Path to the Krisp VIVA model file
                    enableManagedKeys:
                      type: object
                      description: Managed API keys configuration
                      properties:
                        enabled:
                          type: boolean
                          description: Whether managed keys are enabled
            serviceId:
              type: string
              description: ID of the service this deployment belongs to
            createdAt:
              type: string
              format: date-time
              description: Creation timestamp of the deployment
            updatedAt:
              type: string
              format: date-time
              description: Last update timestamp of the deployment
        agentProfile:
          type: string
          description: The agent profile used for resource allocation
          nullable: true
        krispViva:
          type: object
          nullable: true
          description: Krisp VIVA noise cancellation status
          properties:
            audioFilter:
              type: string
              nullable: true
              description: >-
                The currently configured Krisp VIVA audio filter model (tel,
                pro, or null if disabled)
    ErrorResponse:
      type: object
      properties:
        error:
          type: string
          description: Error message
        code:
          type: string
          description: Error code
  securitySchemes:
    PrivateKeyAuth:
      type: http
      scheme: bearer
      description: >-
        Authentication requires a Pipecat Cloud Private API token.


        Generate a Private API key from your Dashboard (Settings > API Keys >
        Private > Create key) and include it as a Bearer token in the
        Authorization header.

````

---

# Source: https://docs.pipecat.ai/cli/cloud/agent.md

> ## Documentation Index
> Fetch the complete documentation index at: https://docs.pipecat.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# agent

> Manage agent deployments

The `agent` command provides sub-commands for managing your deployed agents. These commands allow you to view status, start agents, see logs, and manage deployments.

## start

Start a deployed agent instance, creating an active session.

**Usage:**

```shell  theme={null}
pipecat cloud agent start [ARGS] [OPTIONS]
```

**Arguments:**

<ParamField path="agent-name" type="string" required>
  Unique string identifier for the agent deployment. Must not contain spaces.
</ParamField>

**Options:**

<ParamField path="--api-key / -k" type="string">
  Public API key to authenticate the agent deployment. Will default to any key
  set in your config.

  For more information, see [API keys](/deployment/pipecat-cloud/fundamentals/accounts-and-organizations#api-keys).
</ParamField>

<ParamField path="--data / -d" type="string">
  Stringified JSON object to pass to the agent deployment. This data will be
  available to the agent as a `data` parameter in your `bot()` method.

  More information [here](/deployment/pipecat-cloud/fundamentals/active-sessions#running-an-agent).
</ParamField>

<ParamField path="--force / -f" type="boolean" default="false">
  Skip summary confirmation before issuing start request.
</ParamField>

<ParamField path="--use-daily / -D" type="boolean" default="false">
  Create a Daily WebRTC session for the agent.
</ParamField>

<ParamField path="--daily-properties / -p" type="string">
  Stringified JSON object with Daily room properties to customize the WebRTC
  session. Only used when `--use-daily` is set to true.

  See [Daily API
  documentation](https://docs.daily.co/reference/rest-api/rooms/config) for
  available properties.
</ParamField>

<ParamField path="--organization / -o" type="str">
  Organization to start the agent for. If not provided, uses the current
  organization from your configuration.
</ParamField>

## stop

Stop an active agent session and clean up its resources.

**Usage:**

```shell  theme={null}
pcc agent stop [ARGS] [OPTIONS]
```

**Arguments:**

<ParamField path="agent-name" type="string" required>
  Name of the agent. Must not contain spaces.
</ParamField>

**Options:**

<ParamField path="--session-id / -s" type="string" required>
  ID of the session to stop.
</ParamField>

<ParamField path="--organization / -o" type="string">
  Organization which the agent belongs to. If not provided, uses the current
  organization from your configuration.
</ParamField>

<ParamField path="--force / -f" type="boolean" default="false">
  Bypass prompt for confirmation before stopping the session.
</ParamField>

## status

Shows the current status of an agent deployment, including health and conditions.

**Usage:**

```shell  theme={null}
pipecat cloud agent status [ARGS]
```

**Arguments:**

<ParamField path="agent-name" type="string" required>
  Unique string identifier for the agent deployment. Must not contain spaces.
</ParamField>

## deployments

Lists deployment history for an agent, including image versions and timestamps.

**Usage:**

```shell  theme={null}
pipecat cloud agent deployments [ARGS]
```

**Arguments:**

<ParamField path="agent-name" type="string" required>
  Unique string identifier for the agent deployment. Must not contain spaces.
</ParamField>

## logs

Displays combined logs from all agent instances, useful for debugging issues.

**Usage:**

```shell  theme={null}
pipecat cloud agent logs [ARGS] [OPTIONS]
```

**Arguments:**

<ParamField path="agent-name" type="string" required>
  Unique string identifier for the agent deployment. Must not contain spaces.
</ParamField>

**Options:**

<ParamField path="--level / -l" type="string" default="ALL">
  Filter logs by severity: `ALL`, `DEBUG`, `INFO`, `WARNING`, `ERROR`,
  `CRITICAL`.
</ParamField>

<ParamField path="--limit / -n" type="int" default="100">
  Limit the number of log lines to display.
</ParamField>

<ParamField path="--deployment / -d" type="str">
  Filter results for specific agent deployment ID (obtainable from `pipecat
    cloud agent deployments [agent-name]`).
</ParamField>

<ParamField path="--session-id / -s" type="str">
  Filter results for specific session ID (obtainable from `pipecat cloud agent
    sessions [agent-name]`).
</ParamField>

## list

Lists all agents in an organization with their details.

**Usage:**

```shell  theme={null}
pipecat cloud agent list [OPTIONS]
```

**Options:**

<ParamField path="--organization / -o" type="string">
  Organization to list agents for. If not provided, uses the current
  organization from your configuration.
</ParamField>

<ParamField path="--region / -r" type="string">
  Filter agents by region. Only agents deployed in the specified region will be
  shown. If not provided, agents from all regions are listed.
</ParamField>

## sessions

Lists active sessions for a specified agent. When there are no active sessions, it suggests how to start a new session.

When used with the `--id` option, displays detailed information about a specific session including CPU and memory usage with sparkline visualizations and percentile summaries.

**Usage:**

```shell  theme={null}
pipecat cloud agent sessions [ARGS] [OPTIONS]
```

**Arguments:**

<ParamField path="agent-name" type="string" required>
  Name of the agent to list active sessions for.
</ParamField>

**Options:**

<ParamField path="--id / -i" type="string">
  Session ID to view detailed metrics for. When provided, displays CPU and memory usage statistics including sparkline visualizations and percentile summaries (p50, p90, p99).
</ParamField>

<ParamField path="--organization / -o" type="string">
  Organization to list sessions for. If not provided, uses the current
  organization from your configuration.
</ParamField>

## delete

Deletes an agent deployment. This will prevent starting new agents and remove all associated data.

<Warning>This action is irreversible. All data will be lost.</Warning>

**Usage:**

```shell  theme={null}
pipecat cloud agent delete [ARGS] [OPTIONS]
```

**Arguments:**

<ParamField path="agent-name" type="string" required>
  Unique string identifier for the agent deployment. Must not contain spaces.
</ParamField>

**Options:**

<ParamField path="--force / -f" type="string">
  Do not prompt for confirmation before deleting the agent.
</ParamField>

---

# Source: https://docs.pipecat.ai/server/utilities/audio/aic-filter.md

> ## Documentation Index
> Fetch the complete documentation index at: https://docs.pipecat.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# AICFilter

> Speech enhancement using ai-coustics' SDK

## Overview

`AICFilter` is an audio processor that enhances user speech by reducing background noise and improving speech clarity. It inherits from `BaseAudioFilter` and processes audio frames in real-time using ai-coustics' speech enhancement technology.

To use AIC, you need a license key. Get started at [ai-coustics.com](https://ai-coustics.com/pipecat).

<Note>
  This documentation covers **aic-sdk v2.x**. If you're using aic-sdk v1.x, please see the [Migration Guide](#migration-guide-v1-to-v2) section below for upgrading instructions.
</Note>

## Installation

The AIC filter requires additional dependencies:

```bash  theme={null}
pip install "pipecat-ai[aic]"
```

## Constructor Parameters

<ParamField path="license_key" type="str" required>
  ai-coustics license key for authentication. Get your key at [developers.ai-coustics.io](https://developers.ai-coustics.io).
</ParamField>

<ParamField path="model_id" type="Optional[str]" default="None">
  Model identifier to download from CDN. Required if `model_path` is not provided.
  See [artifacts.ai-coustics.io](https://artifacts.ai-coustics.io/) for available models.
  See the [documentation](https://docs.ai-coustics.com/guides/models) for more detailed information about the models.

  Examples: `"quail-vf-l-16khz"`, `"quail-s-16khz"`, `"quail-l-8khz"`
</ParamField>

<ParamField path="model_path" type="Optional[str]" default="None">
  Path to a local `.aicmodel` file. If provided, `model_id` is ignored and no download occurs.
  Useful for offline deployments or custom models.
</ParamField>

<ParamField path="model_download_dir" type="Optional[Path]" default="None">
  Directory for downloading and caching models. Defaults to a cache directory in the user's home folder.
</ParamField>

## Methods

### create\_vad\_analyzer

Creates an `AICVADAnalyzer` that uses the AIC model's built-in voice activity detection.

```python  theme={null}
def create_vad_analyzer(
    *,
    speech_hold_duration: Optional[float] = None,
    minimum_speech_duration: Optional[float] = None,
    sensitivity: Optional[float] = None,
) -> AICVADAnalyzer
```

#### VAD Parameters

<ParamField path="speech_hold_duration" type="Optional[float]" default="None">
  Controls for how long the VAD continues to detect speech after the audio signal no longer contains speech (in seconds).
  Range: `0.0` to `100x model window length`, Default (in SDK): `0.05s`
</ParamField>

<ParamField path="minimum_speech_duration" type="Optional[float]" default="None">
  Controls for how long speech needs to be present in the audio signal before the VAD considers it speech (in seconds).
  Range: `0.0` to `1.0`, Default (in SDK): `0.0s`
</ParamField>

<ParamField path="sensitivity" type="Optional[float]" default="None">
  Controls the sensitivity (energy threshold) of the VAD. This value is used by the VAD as the threshold a speech audio signal's energy has to exceed in order to be considered speech.
  Formula: `Energy threshold = 10 ** (-sensitivity)`
  Range: `1.0` to `15.0`, Default (in SDK): `6.0`
</ParamField>

### get\_vad\_context

Returns the VAD context once the processor is initialized. Can be used to dynamically adjust VAD parameters at runtime.

```python  theme={null}
vad_ctx = aic_filter.get_vad_context()
vad_ctx.set_parameter(VadParameter.Sensitivity, 8.0)
```

## Input Frames

<ParamField path="FilterEnableFrame" type="Frame">
  Specific control frame to toggle filtering on/off

  ```python  theme={null}
  from pipecat.frames.frames import FilterEnableFrame

  # Disable speech enhancement
  await task.queue_frame(FilterEnableFrame(False))

  # Re-enable speech enhancement
  await task.queue_frame(FilterEnableFrame(True))
  ```
</ParamField>

## Usage Examples

### Basic Usage with AIC VAD

The recommended approach is to use `AICFilter` with its built-in VAD analyzer:

```python  theme={null}
from pipecat.audio.filters.aic_filter import AICFilter
from pipecat.processors.aggregators.llm_response_universal import (
    LLMContextAggregatorPair,
    LLMUserAggregatorParams,
)
from pipecat.transports.services.daily import DailyTransport, DailyParams

# Create the AIC filter
aic_filter = AICFilter(
    license_key=os.environ["AIC_SDK_LICENSE"],
    model_id="quail-vf-l-16khz",
)

# Use AIC's integrated VAD
transport = DailyTransport(
    room_url,
    token,
    "Bot",
    DailyParams(
        audio_in_enabled=True,
        audio_out_enabled=True,
        audio_in_filter=aic_filter,
    ),
)

user_aggregator, assistant_aggregator = LLMContextAggregatorPair(
    context,
    user_params=LLMUserAggregatorParams(
        vad_analyzer=aic_filter.create_vad_analyzer(
            speech_hold_duration=0.05,
            minimum_speech_duration=0.0,
            sensitivity=6.0,
        ),
    ),
)
```

### Using a Local Model

For offline deployments or when you want to manage model files yourself:

```python  theme={null}
from pipecat.audio.filters.aic_filter import AICFilter

aic_filter = AICFilter(
    license_key=os.environ["AIC_SDK_LICENSE"],
    model_path="/path/to/your/model.aicmodel",
)
```

### Custom Cache Directory

Specify a custom directory for model downloads:

```python  theme={null}
from pipecat.audio.filters.aic_filter import AICFilter

aic_filter = AICFilter(
    license_key=os.environ["AIC_SDK_LICENSE"],
    model_id="quail-s-16khz",
    model_download_dir="/opt/aic-models",
)
```

### With Other Transports

The AIC filter works with any Pipecat transport:

```python  theme={null}
from pipecat.audio.filters.aic_filter import AICFilter
from pipecat.processors.aggregators.llm_response_universal import (
    LLMContextAggregatorPair,
    LLMUserAggregatorParams,
)
from pipecat.transports.websocket import FastAPIWebsocketTransport, FastAPIWebsocketParams

aic_filter = AICFilter(
    license_key=os.environ["AIC_SDK_LICENSE"],
    model_id="quail-vf-l-16khz",
)

transport = FastAPIWebsocketTransport(
    params=FastAPIWebsocketParams(
        audio_in_enabled=True,
        audio_out_enabled=True,
        audio_in_filter=aic_filter,
    ),
)

user_aggregator, assistant_aggregator = LLMContextAggregatorPair(
    context,
    user_params=LLMUserAggregatorParams(
        vad_analyzer=aic_filter.create_vad_analyzer(
            speech_hold_duration=0.05,
            sensitivity=6.0,
        ),
    ),
)
```

<Info>
  See the [AIC filter example](https://github.com/pipecat-ai/pipecat/blob/main/examples/foundational/07zd-interruptible-aicoustics.py) for a complete working example.
</Info>

## Models

For detailed information about the available models, take a look at the [Models documentation](https://docs.ai-coustics.com/guides/models).

## Audio Flow

```mermaid  theme={null}
graph TD
    A[AudioRawFrame] --> B[AICFilter]
    B --> C[AICVADAnalyzer]
    C --> D[STT]
```

The AIC filter enhances audio before it reaches the VAD and STT stages, improving transcription accuracy in noisy environments.

## Migration Guide (v1 to v2)

<Note>
  For the complete aic-sdk migration guide including all API changes, see the official [Python 1.3 to 2.0 Migration Guide](https://docs.ai-coustics.com/guides/migrations/python-1-3-to-2-0#quick-migration-checklist).
</Note>

### Migration Steps

1. Update Pipecat to the latest version (aic-sdk v2.x is included automatically).
2. Remove deprecated constructor parameters (`model_type`, `enhancement_level`, `voice_gain`, `noise_gate_enable`).
3. Add `model_id` parameter with an appropriate model (e.g., `"quail-vf-l-16khz"`).
4. Update any runtime VAD adjustments to use the new VAD context API.
5. We recommend to use `aic_filter.create_vad_analyzer()` for improved accuracy.

### Breaking Changes

| v1 Parameter        | v2 Replacement                            |
| ------------------- | ----------------------------------------- |
| `model_type`        | `model_id` (string-based model selection) |
| `enhancement_level` | Removed (model-specific behavior)         |
| `voice_gain`        | Removed                                   |
| `noise_gate_enable` | Removed                                   |

## Notes

* Requires ai-coustics license key (get one at [developers.ai-coustics.io](https://developers.ai-coustics.io))
* Models are automatically downloaded and cached on first use
* Supports real-time audio processing with low latency
* Handles PCM\_16 audio format (int16 samples)
* Thread-safe for pipeline processing
* Can be dynamically enabled/disabled via `FilterEnableFrame`
* Integrated VAD provides better accuracy than standalone VAD when using enhancement
* For available models, visit [artifacts.ai-coustics.io](https://artifacts.ai-coustics.io/)

---

# Source: https://docs.pipecat.ai/server/services/llm/anthropic.md

> ## Documentation Index
> Fetch the complete documentation index at: https://docs.pipecat.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Anthropic

> Large Language Model service implementation using Anthropic's Claude API

## Overview

`AnthropicLLMService` provides integration with Anthropic's Claude models, supporting streaming responses, function calling, and prompt caching with specialized context handling for Anthropic's message format and advanced reasoning capabilities.

<CardGroup cols={2}>
  <Card title="Anthropic LLM API Reference" icon="code" href="https://reference-server.pipecat.ai/en/latest/api/pipecat.services.anthropic.llm.html">
    Pipecat's API methods for Anthropic Claude integration
  </Card>

  <Card title="Example Implementation" icon="play" href="https://github.com/pipecat-ai/pipecat/blob/main/examples/foundational/14a-function-calling-anthropic.py">
    Complete example with function calling
  </Card>

  <Card title="Anthropic Documentation" icon="book" href="https://docs.anthropic.com/en/api/messages">
    Official Anthropic API documentation and features
  </Card>

  <Card title="Anthropic Console" icon="microphone" href="https://console.anthropic.com/">
    Access Claude models and API keys
  </Card>
</CardGroup>

## Installation

To use Anthropic services, install the required dependency:

```bash  theme={null}
pip install "pipecat-ai[anthropic]"
```

## Prerequisites

### Anthropic Account Setup

Before using Anthropic LLM services, you need:

1. **Anthropic Account**: Sign up at [Anthropic Console](https://console.anthropic.com/)
2. **API Key**: Generate an API key from your console dashboard
3. **Model Selection**: Choose from available Claude models (Claude 3.5 Sonnet, Claude 3 Opus, etc.)

### Required Environment Variables

* `ANTHROPIC_API_KEY`: Your Anthropic API key for authentication

---

# Source: https://docs.pipecat.ai/client/react-native/api-reference.md

> ## Documentation Index
> Fetch the complete documentation index at: https://docs.pipecat.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# API Reference

> API reference for the Pipecat React Native SDK

<Note>
  The Pipecat React Native SDK leverages the Pipecat JavaScript SDK for seamless integration with React Native applications.
  For detailed information, please reference to the [Javascript SDK docs](/client/js/api-reference/client-constructor).

  **Just ensure you use the appropriate transport layer for React Native.**
</Note>

---

# Source: https://docs.pipecat.ai/server/services/stt/assemblyai.md

> ## Documentation Index
> Fetch the complete documentation index at: https://docs.pipecat.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# AssemblyAI

> Speech-to-text service implementation using AssemblyAI's real-time transcription API

## Overview

`AssemblyAISTTService` provides real-time speech recognition using AssemblyAI's WebSocket API with support for interim results, end-of-turn detection, and configurable audio processing parameters for accurate transcription in conversational AI applications.

<CardGroup cols={2}>
  <Card title="AssemblyAI STT API Reference" icon="code" href="https://reference-server.pipecat.ai/en/latest/api/pipecat.services.assemblyai.stt.html">
    Pipecat's API methods for AssemblyAI STT integration
  </Card>

  <Card title="Example Implementation" icon="play" href="https://github.com/pipecat-ai/pipecat/blob/main/examples/foundational/07o-interruptible-assemblyai.py">
    Complete example with interruption handling
  </Card>

  <Card title="AssemblyAI Documentation" icon="book" href="https://www.assemblyai.com/docs/api-reference/overview">
    Official AssemblyAI documentation and features
  </Card>

  <Card title="AssemblyAI Console" icon="microphone" href="https://www.assemblyai.com/dashboard/signup">
    Access API keys and transcription features
  </Card>
</CardGroup>

## Installation

To use AssemblyAI services, install the required dependency:

```bash  theme={null}
pip install "pipecat-ai[assemblyai]"
```

## Prerequisites

### AssemblyAI Account Setup

Before using AssemblyAI STT services, you need:

1. **AssemblyAI Account**: Sign up at [AssemblyAI Console](https://www.assemblyai.com/dashboard/signup)
2. **API Key**: Generate an API key from your dashboard
3. **Model Selection**: Choose from available transcription models and features

### Required Environment Variables

* `ASSEMBLYAI_API_KEY`: Your AssemblyAI API key for authentication

---

# Source: https://docs.pipecat.ai/server/services/tts/asyncai.md

> ## Documentation Index
> Fetch the complete documentation index at: https://docs.pipecat.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Async

> Text-to-speech services using Async's WebSocket and HTTP APIs

## Overview

Async provides high-quality text-to-speech synthesis with two service implementations: `AsyncAITTSService` (WebSocket-based) for real-time streaming with interruption support, and `AsyncAIHttpTTSService` (HTTP-based) for simpler synthesis. `AsyncAITTSService` is recommended for interactive applications requiring low latency.

<CardGroup cols={2}>
  <Card title="AsyncAI TTS API Reference" icon="code" href="https://reference-server.pipecat.ai/en/latest/api/pipecat.services.asyncai.tts.html">
    Pipecat's API methods for AsyncAI TTS integration
  </Card>

  <Card title="Example Implementation" icon="play" href="https://github.com/pipecat-ai/pipecat/blob/main/examples/foundational/07zc-interruptible-asyncai.py">
    Complete example with WebSocket streaming
  </Card>

  <Card title="Async Documentation" icon="book" href="https://docs.async.ai/">
    Official Async API documentation
  </Card>

  <Card title="Voice Models" icon="microphone" href="https://async.ai/">
    Explore available voice models and features
  </Card>
</CardGroup>

## Installation

To use Async services, install the required dependencies:

```bash  theme={null}
pip install "pipecat-ai[asyncai]"
```

## Prerequisites

### Async Account Setup

Before using Async TTS services, you need:

1. **Async Account**: Sign up at [Async](https://async.ai)
2. **API Key**: Generate an API key from your account dashboard
3. **Voice Selection**: Choose from available voice models

### Required Environment Variables

* `ASYNCAI_API_KEY`: Your Async API key for authentication

---

# Source: https://docs.pipecat.ai/server/utilities/audio/audio-buffer-processor.md

> ## Documentation Index
> Fetch the complete documentation index at: https://docs.pipecat.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# AudioBufferProcessor

> Process and buffer audio frames from conversations with flexible event handling

## Overview

The `AudioBufferProcessor` captures and buffers audio frames from both input (user) and output (bot) sources during conversations. It provides synchronized audio streams with configurable sample rates, supports both mono and stereo output, and offers flexible event handlers for various audio processing workflows.

## Constructor

```python  theme={null}
AudioBufferProcessor(
    sample_rate=None,
    num_channels=1,
    buffer_size=0,
    enable_turn_audio=False,
    **kwargs
)
```

### Parameters

<ParamField path="sample_rate" type="Optional[int]" default="None">
  The desired output sample rate in Hz. If `None`, uses the transport's sample
  rate from the `StartFrame`.
</ParamField>

<ParamField path="num_channels" type="int" default="1">
  Number of output audio channels:

  * `1`: Mono output (user and bot audio are mixed together)
  * `2`: Stereo output (user audio on left channel, bot audio on right channel)
</ParamField>

<ParamField path="buffer_size" type="int" default="0">
  Buffer size in bytes that triggers audio data events:

  * `0`: Events only trigger when recording stops
  * `>0`: Events trigger whenever buffer reaches this size (useful for chunked processing)
</ParamField>

<ParamField path="enable_turn_audio" type="bool" default="False">
  Whether to enable per-turn audio event handlers (`on_user_turn_audio_data` and
  `on_bot_turn_audio_data`).
</ParamField>

## Properties

### sample\_rate

```python  theme={null}
@property
def sample_rate(self) -> int
```

The current sample rate of the audio processor in Hz.

### num\_channels

```python  theme={null}
@property
def num_channels(self) -> int
```

The number of channels in the audio output (1 for mono, 2 for stereo).

## Methods

### start\_recording()

```python  theme={null}
async def start_recording()
```

Start recording audio from both user and bot sources. Initializes recording state and resets audio buffers.

### stop\_recording()

```python  theme={null}
async def stop_recording()
```

Stop recording and trigger final audio data handlers with any remaining buffered audio.

### has\_audio()

```python  theme={null}
def has_audio() -> bool
```

Check if both user and bot audio buffers contain data.

**Returns:** `True` if both buffers contain audio data.

## Event Handlers

The processor supports multiple event handlers for different audio processing workflows. Register handlers using the `@processor.event_handler()` decorator.

### on\_audio\_data

Triggered when `buffer_size` is reached or recording stops, providing merged audio.

```python  theme={null}
@audiobuffer.event_handler("on_audio_data")
async def on_audio_data(buffer, audio: bytes, sample_rate: int, num_channels: int):
    # Handle merged audio data
    pass
```

**Parameters:**

* `buffer`: The AudioBufferProcessor instance
* `audio`: Merged audio data (format depends on `num_channels` setting)
* `sample_rate`: Sample rate in Hz
* `num_channels`: Number of channels (1 or 2)

### on\_track\_audio\_data

Triggered alongside `on_audio_data`, providing separate user and bot audio tracks.

```python  theme={null}
@audiobuffer.event_handler("on_track_audio_data")
async def on_track_audio_data(buffer, user_audio: bytes, bot_audio: bytes,
                             sample_rate: int, num_channels: int):
    # Handle separate audio tracks
    pass
```

**Parameters:**

* `buffer`: The AudioBufferProcessor instance
* `user_audio`: Raw user audio bytes (always mono)
* `bot_audio`: Raw bot audio bytes (always mono)
* `sample_rate`: Sample rate in Hz
* `num_channels`: Always 1 for individual tracks

### on\_user\_turn\_audio\_data

Triggered when a user speaking turn ends. Requires `enable_turn_audio=True`.

```python  theme={null}
@audiobuffer.event_handler("on_user_turn_audio_data")
async def on_user_turn_audio_data(buffer, audio: bytes, sample_rate: int, num_channels: int):
    # Handle user turn audio
    pass
```

**Parameters:**

* `buffer`: The AudioBufferProcessor instance
* `audio`: Audio data from the user's speaking turn
* `sample_rate`: Sample rate in Hz
* `num_channels`: Always 1 (mono)

### on\_bot\_turn\_audio\_data

Triggered when a bot speaking turn ends. Requires `enable_turn_audio=True`.

```python  theme={null}
@audiobuffer.event_handler("on_bot_turn_audio_data")
async def on_bot_turn_audio_data(buffer, audio: bytes, sample_rate: int, num_channels: int):
    # Handle bot turn audio
    pass
```

**Parameters:**

* `buffer`: The AudioBufferProcessor instance
* `audio`: Audio data from the bot's speaking turn
* `sample_rate`: Sample rate in Hz
* `num_channels`: Always 1 (mono)

## Audio Processing Features

* **Automatic resampling**: Converts incoming audio to the specified sample rate
* **Buffer synchronization**: Aligns user and bot audio streams temporally
* **Silence insertion**: Fills gaps in non-continuous audio streams to maintain timing
* **Turn tracking**: Monitors speaking turns when `enable_turn_audio=True`

## Integration Notes

### STT Audio Passthrough

If using an STT service in your pipeline, enable audio passthrough to make audio available to the AudioBufferProcessor:

```python  theme={null}
stt = DeepgramSTTService(
    api_key=os.getenv("DEEPGRAM_API_KEY"),
    audio_passthrough=True,
)
```

<Note>`audio_passthrough` is enabled by default.</Note>

### Pipeline Placement

Add the AudioBufferProcessor after `transport.output()` to capture both user and bot audio:

```python  theme={null}
pipeline = Pipeline([
    transport.input(),
    # ... other processors ...
    transport.output(),
    audiobuffer,  # Place after audio output
    # ... remaining processors ...
])
```

---

# Source: https://docs.pipecat.ai/cli/cloud/auth.md

> ## Documentation Index
> Fetch the complete documentation index at: https://docs.pipecat.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# auth

> Authentication and authorization commands

The `auth` command group manages authentication with Pipecat Cloud, allowing you to login, logout, and check your current account identity.

## login

Begins an authorization flow to authenticate with Pipecat Cloud.

**Usage:**

```shell  theme={null}
pipecat cloud auth login [OPTIONS]
```

**Options:**

<ParamField path="--headless / -h" type="boolean" default="false">
  Skip opening a browser window for authentication and print the URL instead.
  Useful for remote or containerized environments.
</ParamField>

This command initiates the authentication process by:

1. Generating a unique authentication URL
2. Opening your default web browser (if available)
3. Waiting for you to complete the sign-in process in the browser
4. Retrieving and storing your access token locally

If the browser doesn't open automatically, or if you use the `--headless` option, you can copy and paste the displayed URL into your browser manually.

On successful login, the CLI will store the access token in the local configuration file (defaults to `~/.config/pipecatcloud/pipecatcloud.toml`). This token will be used for all subsequent requests to the Pipecat Cloud API.

You can override the default location by setting the `PIPECAT_CONFIG_PATH` environment variable.

## logout

Signs out of the current Pipecat Cloud account by removing the access token from your local configuration file.

**Usage:**

```shell  theme={null}
pipecat cloud auth logout
```

## whoami

Displays information about the currently authenticated user, including user ID, active organization, and Daily API key.

**Usage:**

```shell  theme={null}
pipecat cloud auth whoami
```

## Configuration

CLI configuration is stored in `~/.config/pipecatcloud/pipecatcloud.toml`. You can override this location by setting the `PIPECAT_CONFIG_PATH` environment variable.

The configuration stores your access token, active organization, and default API keys.

**View current configuration:**

```shell  theme={null}
pipecat cloud --config
```

<Accordion title="Example configuration file">
  <Warning>We do not recommend manually editing the configuration file. Should the file become malformed, please run `pipecat cloud auth logout` to reset.</Warning>

  ```toml  theme={null}
    token = "..."
    org = "user-namespace"

    [another-user-org]
    default_public_key = "pk_..."
    default_public_key_name = "Test Key"

    [futher-user-org]
    default_public_key = "pk_..."
    default_public_key_name = "Pipecat Cloud Public Key"
  ```
</Accordion>

***

<Card title="More about user accounts" icon="people-group" href="/deployment/pipecat-cloud/fundamentals/accounts-and-organizations">
  Managing your account and collaborating on agents as part of a team
</Card>

---

# Source: https://docs.pipecat.ai/deployment/pipecat-cloud/guides/container-registries/aws-ecr.md

> ## Documentation Index
> Fetch the complete documentation index at: https://docs.pipecat.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# AWS ECR

> Deploy Pipecat Cloud agents using Amazon Elastic Container Registry

If you're using AWS ECR with private repositories for Pipecat Cloud deployments, you'll need to configure image pull secrets to authenticate with your registry. **ECR tokens expire every 12 hours**, so you'll also need to set up automatic token refresh.

## Authenticate Docker

First, authenticate Docker with ECR:

```bash  theme={null}
aws ecr get-login-password --region <region> | docker login --username AWS --password-stdin <aws_account_id>.dkr.ecr.<region>.amazonaws.com
```

## Configure Image Pull Secrets

Use the Pipecat Cloud REST API to store your ECR registry credentials:

```bash  theme={null}
# Get ECR login token
ECR_TOKEN=$(aws ecr get-login-password --region <region>)

# Create image pull secret using REST API
curl --request PUT \
  --url https://api.pipecat.daily.co/v1/secrets/my-ecr-credentials \
  --header 'Authorization: Bearer <your-private-api-token>' \
  --header 'Content-Type: application/json' \
  --data '{
    "secretValue": "'"$(echo -n "AWS:$ECR_TOKEN" | base64)"'",
    "host": "https://<accountid>.dkr.ecr.<region>.amazonaws.com",
    "isImagePullSecret": true
  }'
```

Replace:

* `<region>` with your AWS region (e.g., `us-east-1`, `us-west-2`)
* `<aws_account_id>` with your AWS account ID
* `<your-private-api-token>` with your Pipecat Cloud private API token

## Configure Your Deployment

Create a `pcc-deploy.toml` file with your ECR image configuration:

```toml  theme={null}
agent_name = "my-ecr-agent"
image = "<aws_account_id>.dkr.ecr.<region>.amazonaws.com/your-repo:tag"
secret_set = "my-agent-secrets"
image_credentials = "my-ecr-credentials"

[scaling]
    min_instances = 0
```

## Build and Push to ECR

Build and push your agent image to ECR using the Pipecat Cloud CLI:

```bash  theme={null}
# Build and push using your pcc-deploy.toml configuration
pipecat cloud docker build-push
```

This command automatically builds for the correct platform (`linux/arm64`) and pushes to your configured ECR repository.

<Note>
  Pipecat Cloud may pull your image on deploy and again during scale-outs, so
  credentials must be valid whenever new pods start.
</Note>

## Deploy Your Agent

Deploy using your configured `pcc-deploy.toml`:

```bash  theme={null}
pipecat cloud deploy
```

## Automatic Token Refresh (Required)

**ECR passwords expire every 12 hours**, so set up a scheduled job to refresh the token:

```bash  theme={null}
#!/bin/bash
# refresh-ecr-token.sh

# Get fresh ECR token
ECR_TOKEN=$(aws ecr get-login-password --region <region>)

# Update the existing image pull secret
curl --request PUT \
  --url https://api.pipecat.daily.co/v1/secrets/my-ecr-credentials \
  --header 'Authorization: Bearer <your-private-api-token>' \
  --header 'Content-Type: application/json' \
  --data '{
    "secretValue": "'"$(echo -n "AWS:$ECR_TOKEN" | base64)"'",
    "host": "https://<accountid>.dkr.ecr.<region>.amazonaws.com",
    "isImagePullSecret": true
  }'

```

Schedule this script to run every 6-8 hours using cron or your preferred scheduler.

## Operational Considerations

**Critical operational tips:**

* **Image pulls can happen during scale-outs**, not just at initial deploy—keep the secret valid continuously
* **If you see agents failing to become ready with no logs**, check that your ECR credentials aren't expired
* **Consider setting up monitoring alerts** for ECR token expiration
* **Test your refresh script** to ensure it works before relying on it in production

<Note>
  Ensure your AWS credentials have the necessary permissions to access ECR,
  including `ecr:GetAuthorizationToken` and `ecr:BatchGetImage` policies.
</Note>

---

# Source: https://docs.pipecat.ai/server/services/tts/aws.md

# Source: https://docs.pipecat.ai/server/services/stt/aws.md

# Source: https://docs.pipecat.ai/server/services/s2s/aws.md

# Source: https://docs.pipecat.ai/server/services/llm/aws.md

> ## Documentation Index
> Fetch the complete documentation index at: https://docs.pipecat.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# AWS Bedrock

> Large Language Model service implementation using Amazon Bedrock API

## Overview

`AWSBedrockLLMService` provides access to Amazon's foundation models including Anthropic Claude and Amazon Nova, with streaming responses, function calling, and multimodal capabilities through Amazon's managed AI service for enterprise-grade LLM deployment.

<CardGroup cols={2}>
  <Card title="AWS Bedrock LLM API Reference" icon="code" href="https://reference-server.pipecat.ai/en/latest/api/pipecat.services.aws.llm.html">
    Pipecat's API methods for AWS Bedrock integration
  </Card>

  <Card title="Example Implementation" icon="play" href="https://github.com/pipecat-ai/pipecat/blob/main/examples/foundational/14r-function-calling-aws.py">
    Complete example with function calling
  </Card>

  <Card title="AWS Bedrock Documentation" icon="book" href="https://docs.aws.amazon.com/bedrock/latest/userguide/what-is-bedrock.html">
    Official AWS Bedrock documentation and features
  </Card>

  <Card title="AWS Console" icon="microphone" href="https://console.aws.amazon.com/bedrock/">
    Access foundation models and manage IAM
  </Card>
</CardGroup>

## Installation

To use AWS Bedrock services, install the required dependencies:

```bash  theme={null}
pip install "pipecat-ai[aws]"
```

## Prerequisites

### AWS Account Setup

Before using AWS Bedrock LLM services, you need:

1. **AWS Account**: Sign up at [AWS Console](https://console.aws.amazon.com/)
2. **IAM User**: Create an IAM user with Amazon Bedrock permissions
3. **Model Access**: Request access to foundation models in your AWS region
4. **Credentials**: Set up AWS access keys and region configuration

### Required Environment Variables

* `AWS_ACCESS_KEY_ID`: Your AWS access key ID
* `AWS_SECRET_ACCESS_KEY`: Your AWS secret access key
* `AWS_SESSION_TOKEN`: Session token (if using temporary credentials)
* `AWS_REGION`: AWS region (defaults to "us-east-1")

---

# Source: https://docs.pipecat.ai/server/services/tts/azure.md

# Source: https://docs.pipecat.ai/server/services/stt/azure.md

# Source: https://docs.pipecat.ai/server/services/llm/azure.md

> ## Documentation Index
> Fetch the complete documentation index at: https://docs.pipecat.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Azure

> Large Language Model service implementation using Azure OpenAI API

## Overview

`AzureLLMService` provides access to Azure OpenAI's language models through an OpenAI-compatible interface. It inherits from `OpenAILLMService` and supports streaming responses, function calling, and context management with enterprise-grade security and compliance.

<CardGroup cols={2}>
  <Card title="Azure LLM API Reference" icon="code" href="https://reference-server.pipecat.ai/en/latest/api/pipecat.services.azure.llm.html">
    Pipecat's API methods for Azure OpenAI integration
  </Card>

  <Card title="Example Implementation" icon="play" href="https://github.com/pipecat-ai/pipecat/blob/main/examples/foundational/14h-function-calling-azure.py">
    Complete example with function calling
  </Card>

  <Card title="Azure OpenAI Documentation" icon="book" href="https://learn.microsoft.com/en-us/azure/ai-services/openai/">
    Official Azure OpenAI documentation and setup
  </Card>

  <Card title="Azure Portal" icon="microphone" href="https://portal.azure.com/">
    Create OpenAI resources and get credentials
  </Card>
</CardGroup>

## Installation

To use Azure OpenAI services, install the required dependency:

```bash  theme={null}
pip install "pipecat-ai[azure]"
```

## Prerequisites

### Azure OpenAI Setup

Before using Azure OpenAI LLM services, you need:

1. **Azure Account**: Sign up at [Azure Portal](https://portal.azure.com/)
2. **OpenAI Resource**: Create an Azure OpenAI resource in your subscription
3. **Model Deployment**: Deploy your chosen model (GPT-4, GPT-3.5-turbo, etc.)
4. **Credentials**: Get your API key, endpoint, and deployment name

### Required Environment Variables

* `AZURE_CHATGPT_API_KEY`: Your Azure OpenAI API key
* `AZURE_CHATGPT_ENDPOINT`: Your Azure OpenAI endpoint URL
* `AZURE_CHATGPT_MODEL`: Your model deployment name

---

# Source: https://docs.pipecat.ai/getting-started/build-your-next-bot.md

> ## Documentation Index
> Fetch the complete documentation index at: https://docs.pipecat.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Build Your Next Bot

> Scaffold a new Pipecat project with the CLI

Ready to build your own bot? The Pipecat CLI scaffolds complete projects tailored to your platform and use case.

<Note>
  **New to Pipecat?** We recommend completing the
  [Quickstart](/getting-started/quickstart) first to understand Pipecat basics
  before scaffolding your own project.
</Note>

## Install the Pipecat CLI

Install the CLI globally using [uv](https://docs.astral.sh/uv/) or [pipx](https://pipx.pypa.io/):

```bash  theme={null}
uv tool install pipecat-ai-cli
# or
pipx install pipecat-ai-cli
```

Verify installation:

```bash  theme={null}
pipecat --version
```

## Create Your Project

Run the interactive setup wizard:

```bash  theme={null}
pipecat init
```

The CLI will guide you through selecting your platform (phone or web/mobile), transport provider, AI services (STT, LLM, TTS), and deployment target.

<Card title="CLI Reference" icon="terminal" href="/cli/init">
  See detailed options and generated project structure
</Card>

## Next Steps

Once `pipecat init` completes, the CLI will provide specific instructions for your generated project, including:

* How to configure your API keys
* Installing dependencies
* Running your bot locally
* Customizing the bot logic and behavior

Follow the instructions displayed by the CLI to get your bot up and running!

## Deploy to Production

Ready to deploy your bot? Choose between managed cloud hosting or self-hosted infrastructure.

<CardGroup cols={2}>
  <Card title="Pipecat Cloud" icon="cloud" href="/deployment/pipecat-cloud/introduction">
    Deploy and manage your agents with the CLI - purpose-built for Pipecat
  </Card>

  <Card title="Self-Hosted Options" icon="server" href="/deployment/overview">
    Deploy to Fly.io, Modal, AWS, or your own infrastructure
  </Card>
</CardGroup>

## Learn More

<CardGroup cols={2}>
  <Card title="Core Concepts" icon="book-open" href="/guides/learn/overview">
    Understand pipelines, processors, and transports
  </Card>

  <Card title="Browse Examples" icon="code" href="https://github.com/pipecat-ai/pipecat-examples">
    30+ production-ready examples for inspiration
  </Card>
</CardGroup>

---

# Source: https://docs.pipecat.ai/client/js/api-reference/callbacks.md

> ## Documentation Index
> Fetch the complete documentation index at: https://docs.pipecat.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Callbacks and events

export const TransportTable = rows => {
  const headers = ["", "DailyTransport", "SmallWebRTCTransport", "RNDailyTransport", "WebSocketTransport", "OpenAIRealTimeWebRTCTransport", "GeminiLiveWebsocketTransport"];
  if (!rows || !rows.rows || !Array.isArray(rows.rows)) {
    return <div>Error: Invalid rows data</div>;
  }
  return <table className="compatibility-table">
      <tr>
        {headers.map(header => <th key={header}>{header}</th>)}
      </tr>
      {rows.rows.map(entry => <tr key={entry.name}>
          <td>
            {entry.name}() {entry.deprecated ? "⚠️ DEPRECATED" : ""}
          </td>
          <td className={entry.dt ? "supported" : "unsupported"}>
            {entry.dt || "❌"}
          </td>
          <td className={entry.swrtc ? "supported" : "unsupported"}>
            {entry.swrtc || "❌"}
          </td>
          <td className={entry.rn ? "supported" : "unsupported"}>
            {entry.rn || "❌"}
          </td>
          <td className={entry.ws ? "supported" : "unsupported"}>
            {entry.ws || "❌"}
          </td>
          <td className={entry.oai ? "supported" : "unsupported"}>
            {entry.oai || "❌"}
          </td>
          <td className={entry.gem ? "supported" : "unsupported"}>
            {entry.gem || "❌"}
          </td>
        </tr>)}
    </table>;
};

export const callbacks = () => {
  return [{
    name: "onConnected",
    dt: "1.0.0",
    swrtc: "1.0.0",
    rn: "1.0.0",
    ws: "1.0.0",
    oai: "1.0.0",
    gem: "1.0.0"
  }, {
    name: "onDisconnected",
    dt: "1.0.0",
    swrtc: "1.0.0",
    rn: "1.0.0",
    ws: "1.0.0",
    oai: "1.0.0",
    gem: "1.0.0"
  }, {
    name: "onTransportStateChanged",
    dt: "1.0.0",
    swrtc: "1.0.0",
    rn: "1.0.0",
    ws: "1.0.0",
    oai: "1.0.0",
    gem: "1.0.0"
  }, {
    name: "onBotReady",
    dt: "1.0.0",
    swrtc: "1.0.0",
    rn: "1.0.0",
    ws: "1.0.0",
    oai: "1.0.0",
    gem: "1.0.0"
  }, {
    name: "onBotConnected",
    dt: "1.0.0"
  }, {
    name: "onBotDisconnected",
    dt: "1.0.0"
  }, {
    name: "onParticipantJoined",
    dt: "1.0.0"
  }, {
    name: "onParticipantLeft",
    dt: "1.0.0"
  }, {
    name: "onServerMessage",
    dt: "1.0.0",
    swrtc: "1.0.0",
    rn: "1.0.0",
    ws: "1.0.0"
  }, {
    name: "onMessageError",
    dt: "1.0.0",
    swrtc: "1.0.0",
    rn: "1.0.0",
    ws: "1.0.0",
    oai: "1.0.0",
    gem: "1.0.0"
  }, {
    name: "onError",
    dt: "1.0.0",
    swrtc: "1.0.0",
    rn: "1.0.0",
    ws: "1.0.0",
    oai: "1.0.0",
    gem: "1.0.0"
  }, {
    name: "onAvailableMicsUpdated",
    dt: "1.0.0",
    swrtc: "1.0.0",
    rn: "1.0.0",
    ws: "1.0.0",
    oai: "1.0.0",
    gem: "1.0.0"
  }, {
    name: "onAvailableCamsUpdated",
    dt: "1.0.0",
    swrtc: "1.0.0",
    rn: "1.0.0",
    oai: "1.0.0"
  }, {
    name: "onAvailableSpeakersUpdated",
    dt: "1.0.0",
    swrtc: "1.0.0",
    rn: "1.0.0",
    ws: "1.0.0",
    oai: "1.0.0",
    gem: "1.0.0"
  }, {
    name: "onMicUpdated",
    dt: "1.0.0",
    swrtc: "1.0.0",
    rn: "1.0.0",
    ws: "1.0.0",
    oai: "1.0.0",
    gem: "1.0.0"
  }, {
    name: "onCamUpdated",
    dt: "1.0.0",
    swrtc: "1.0.0",
    rn: "1.0.0",
    oai: "1.0.0"
  }, {
    name: "onSpeakerUpdated",
    dt: "1.0.0",
    swrtc: "1.0.0",
    rn: "1.0.0",
    ws: "1.0.0",
    oai: "1.0.0",
    gem: "1.0.0"
  }, {
    name: "onDeviceError",
    dt: "1.0.0",
    swrtc: "1.0.0",
    rn: "1.0.0",
    ws: "1.0.0",
    oai: "1.0.0",
    gem: "1.0.0"
  }, {
    name: "onTrackStarted",
    dt: "1.0.0",
    swrtc: "1.0.0",
    rn: "1.0.0",
    ws: "1.0.0",
    oai: "1.0.0",
    gem: "1.0.0"
  }, {
    name: "onTrackStopped",
    dt: "1.0.0",
    swrtc: "1.0.0",
    rn: "1.0.0",
    ws: "1.0.0",
    oai: "1.0.0",
    gem: "1.0.0"
  }, {
    name: "onScreenTrackStarted",
    dt: "1.0.0",
    swrtc: "1.3.0",
    rn: "1.0.0"
  }, {
    name: "onScreenTrackStopped",
    dt: "1.0.0",
    swrtc: "1.3.0",
    rn: "1.0.0"
  }, {
    name: "onLocalAudioLevel",
    dt: "1.0.0",
    swrtc: "1.0.0",
    rn: "1.0.0",
    ws: "1.0.0",
    oai: "1.0.0",
    gem: "1.0.0"
  }, {
    name: "onRemoteAudioLevel",
    dt: "1.0.0",
    swrtc: "1.0.0",
    rn: "1.0.0",
    ws: "1.0.0",
    oai: "1.0.0",
    gem: "1.0.0"
  }, {
    name: "onBotStartedSpeaking",
    dt: "1.0.0",
    swrtc: "1.0.0",
    rn: "1.0.0",
    ws: "1.0.0",
    oai: "1.0.0",
    gem: "1.0.0"
  }, {
    name: "onBotStoppedSpeaking",
    dt: "1.0.0",
    swrtc: "1.0.0",
    rn: "1.0.0",
    ws: "1.0.0",
    oai: "1.0.0",
    gem: "1.0.0"
  }, {
    name: "onUserStartedSpeaking",
    dt: "1.0.0",
    swrtc: "1.0.0",
    rn: "1.0.0",
    ws: "1.0.0",
    oai: "1.0.0",
    gem: "1.0.0"
  }, {
    name: "onUserStoppedSpeaking",
    dt: "1.0.0",
    swrtc: "1.0.0",
    rn: "1.0.0",
    ws: "1.0.0",
    oai: "1.0.0",
    gem: "1.0.0"
  }, {
    name: "onUserTranscript",
    dt: "1.0.0",
    swrtc: "1.0.0",
    rn: "1.0.0",
    ws: "1.0.0",
    oai: "1.0.0",
    gem: "1.0.0"
  }, {
    name: "onBotOutput",
    dt: "1.5.0",
    swrtc: "1.8.0",
    ws: "1.5.0"
  }, {
    name: "onBotTranscript",
    deprecated: true,
    dt: "1.0.0",
    swrtc: "1.0.0",
    rn: "1.0.0",
    ws: "1.0.0",
    oai: "1.0.0"
  }, {
    name: "onBotLlmSearchResponse",
    dt: "1.0.0",
    swrtc: "1.0.0",
    rn: "1.0.0",
    ws: "1.0.0"
  }, {
    name: "onBotLlmText",
    dt: "1.0.0",
    swrtc: "1.0.0",
    rn: "1.0.0",
    ws: "1.0.0"
  }, {
    name: "onBotLlmStarted",
    dt: "1.0.0",
    swrtc: "1.0.0",
    rn: "1.0.0",
    ws: "1.0.0"
  }, {
    name: "onBotLlmStopped",
    dt: "1.0.0",
    swrtc: "1.0.0",
    rn: "1.0.0",
    ws: "1.0.0"
  }, {
    name: "onBotTtsText",
    dt: "1.0.0",
    swrtc: "1.0.0",
    rn: "1.0.0",
    ws: "1.0.0",
    oai: "1.0.0",
    gem: "1.0.0"
  }, {
    name: "onBotTtsStarted",
    dt: "1.0.0",
    swrtc: "1.0.0",
    rn: "1.0.0",
    ws: "1.0.0"
  }, {
    name: "onBotTtsStopped",
    dt: "1.0.0",
    swrtc: "1.0.0",
    rn: "1.0.0",
    ws: "1.0.0"
  }, {
    name: "onMetrics",
    dt: "1.0.0",
    swrtc: "1.0.0",
    rn: "1.0.0",
    ws: "1.0.0"
  }];
};

The Pipecat JavaScript client listens for messages and events from the bot via the transport layer. This allows you to respond to changes in state, errors, and other events. The client implements the RTVI standard for these communications.

## Event Handling Options

You can handle events in two ways:

### 1. Callbacks

Define handlers in the client constructor:

```typescript  theme={null}
const pcClient = new PipecatClient({
  callbacks: {
    onBotReady: () => console.log("Bot ready via callback"),
    // ... other callbacks
  },
});
```

### 2. Event Listeners

Add handlers using the event emitter pattern:

```typescript  theme={null}
pcClient.on(RTVIEvent.BotReady, () => console.log("Bot ready via event"));
```

<Note>
  Events and callbacks provide the same functionality. Choose the pattern that
  best fits your application's architecture.
</Note>

## Callbacks

### State and connectivity

<ParamField path="onConnected">
  Local user successfully established a connection to the transport.
</ParamField>

<ParamField path="onDisconnected">
  Local user disconnected from the transport, either intentionally by calling
  `pcClient.disconnect()` or due to an error.
</ParamField>

<ParamField path="onTransportStateChanged" type="state:TransportState">
  Provides a `TransportState` string representing the connectivity state of the
  local client. See [transports](../transports/transport) for state explanation.
</ParamField>

<ParamField path="onBotStarted" type="botResponse: unknown">
  A call to [`startBot()`](./client-methods#startbot) (i.e. a pre-connection
  REST endpoint) was successful and the bot should now be started or in the
  process of starting. The callback receives any data returned from your
  endpoint.
</ParamField>

<ParamField path="onBotReady" type="botReadyData:BotReadyData">
  The bot has been instantiated, its pipeline is configured, and it is receiving
  user media and interactions. This method is passed a `BotReadyData` object,
  which contains the RTVI `version` number. Since the bot is remote and may be
  using a different version of RTVI than the client, you can use the passed
  `version` string to check for compatibility.
</ParamField>

<ParamField path="onBotConnected">
  Bot connected to the transport and is configuring. Note: bot connectivity does
  not infer that its pipeline is yet ready to run. Please use `onBotReady`
  instead.
</ParamField>

<ParamField path="onBotDisconnected" type="participant: Participant">
  Bot disconnected from the transport. This may occur due to session expiry, a
  pipeline error or for any reason the server deems the session over.
</ParamField>

<ParamField path="onParticipantJoined" type="participant: Participant">
  A non-bot participant joined the session.
</ParamField>

<ParamField path="onParticipantLeft" type="participant: Participant">
  A non-bot participant left the session. Note: excluded local participant.
</ParamField>

### Messages and errors

<ParamField path="onServerMessage" type="data:any">
  Receives custom messages sent from the server to the client. This provides a
  generic channel for server-to-client communication. The data structure is
  flexible and defined by the server implementation.
</ParamField>

<ParamField path="onMessageError" type="message:RTVIMessage">
  Response error when an action fails or an unknown message type is sent from
  the client.
</ParamField>

<ParamField path="onError" type="message:RTVIMessage">
  Error signalled by the bot. This could be due to a malformed config update or
  an unknown action dispatch or the inability to complete a client request. The
  message parameter is of type `error` and matches [the RTVI
  standard](/client/rtvi-standard#error-%F0%9F%A4%96). Its `data` field includes
  a `message` string that describes the error and a `fatal` boolean indicating
  if the error is unrecoverable and resulted in a bot disconnection. If `fatal`
  is true, the client will automatically disconnect.
</ParamField>

### Media and devices

<ParamField path="onAvailableMicsUpdated" type="mics:MediaDeviceInfo[]">
  Lists available local media microphone devices. Triggered when a new device
  becomes available, a device is removed, or in response to
  `pcClient.initDevices()`.
</ParamField>

<ParamField path="onAvailableCamsUpdated" type="cams:MediaDeviceInfo[]">
  Lists available local media camera devices. Triggered when a new device
  becomes available, a device is removed, or in response to
  `pcClient.initDevices()`.
</ParamField>

<ParamField path="onAvailableSpeakersUpdated" type="speakers:MediaDeviceInfo[]">
  Lists available local speaker devices. Triggered when a new device becomes
  available, a device is removed, or in response to `pcClient.initDevices()`.
</ParamField>

<ParamField path="onMicUpdated" type="mic:MediaDeviceInfo">
  User selected a new microphone as their selected/active device.
</ParamField>

<ParamField path="onCamUpdated" type="cam:MediaDeviceInfo">
  User selected a new camera as their selected/active device.
</ParamField>

<ParamField path="onSpeakerUpdated" type="speaker:MediaDeviceInfo">
  User selected a new speaker as their selected/active device.
</ParamField>

<ParamField path="onDeviceError" type="error:DeviceError">
  Error related to media devices, such as camera or microphone issues. This
  could be due to permissions, device unavailability, or other related problems.
  See the [DeviceError](./errors#deviceerror) section for more details about the
  return type.
</ParamField>

<ParamField path="onTrackStarted" type="track: MediaStreamTrack, participant:Participant">
  Media track from a local or remote participant/bot was started and playable.
  Can be either an audio or video track.
</ParamField>

<ParamField path="onTrackStopped" type="track: MediaStreamTrack, participant:Participant">
  Media track from a local or remote participant/bot was stopped and no longer
  playable.
</ParamField>

<ParamField path="onScreenTrackStarted" type="track: MediaStreamTrack, participant:Participant">
  Media track from a local or remote participant's screenshare was started and
  playable. Can be either an audio or video track.
</ParamField>

<ParamField path="onScreenTrackStopped" type="track: MediaStreamTrack, participant:Participant">
  Media track from a local or remote participant's screenshare was stopped and
  no longer playable.
</ParamField>

### Audio and Voice Activity

<ParamField path="onLocalAudioLevel" type="level:number">
  Local audio gain level (0 to 1).
</ParamField>

<ParamField path="onRemoteAudioLevel" type="level: number, participant: Participant">
  Remote audio gain level (0 to 1). Note: if more than one participant is
  connected to the transport, the `participant` property details the associated
  peer/bot.
</ParamField>

<ParamField path="onBotStartedSpeaking">
  The bot started speaking/sending speech audio.
</ParamField>

<ParamField path="onBotStoppedSpeaking">
  The bot stopped speaking/sending speech audio.
</ParamField>

<ParamField path="onUserStartedSpeaking">
  The local user started speaking. This method is more reliable than using audio
  gain and is the result of the bot's VAD (voice activity detection) model. This
  provides a more accurate result in noisy environments.
</ParamField>

<ParamField path="onUserStoppedSpeaking">
  The local user stopped speaking, indicated by the VAD model.
</ParamField>

### Transcription

<ParamField path="onUserTranscript" type="TranscriptData">
  Transcribed local user input (both partial and final).

  Callback receives a `TranscriptData` object:

  <Expandable title="TranscriptData">
    <ParamField path="text" type="string">
      The transcribed text.
    </ParamField>

    <ParamField path="final" type="boolean">
      Indicates if the text is final (true) or partial (false).
    </ParamField>

    <ParamField path="timestamp" type="string">
      The timestamp of the transcription.
    </ParamField>

    <ParamField path="user_id" type="string">
      The ID of the user the transcription is for.
    </ParamField>
  </Expandable>
</ParamField>

<ParamField path="onBotOutput" type="BotOutputData">
  A best-effort stream of the bot's output text, including both spoken and unspoken content. This callback is triggered as the bot aggregates the LLM's response into sentences or other logical text blocks as well as word-by-word during TTS synthesis. The callback receives a `BotOutputData` object:

  <Expandable title="BotOutputData">
    <ParamField path="text" type="string">
      The aggregated output text from the bot.
    </ParamField>

    <ParamField path="spoken" type="boolean">
      Indicates if the text has been spoken by the bot.
    </ParamField>

    <ParamField path="aggregated_by" type="'sentence' | 'word' | string">
      The aggregation type used for this output (e.g., "sentence", "code"). "sentence" and "word" are reserved aggregation types defined by the RTVI standard. Other aggregation types may be defined by custom text aggregators used by the server. "word" aggregated outputs are sent at the time of TTS synthesis for real-time word-level streaming and can be used in lieu of `onBotTtsText` if desired.
    </ParamField>
  </Expandable>
</ParamField>

<ParamField path="onBotTranscript" type="text:BotLLMTextData">
  <Warning>DEPRECATED in favor of `onBotOutput` in Pipecat version 0.0.95 and client-js version 1.5.0</Warning>
  Finalized bot output text generated by the LLM. Sentence aggregated.
</ParamField>

### Service-specific Events

<ParamField path="onBotLlmSearchResponse" type="BotLLMSearchResponseData">
  Bot LLM search response text generated by the LLM service. This is typically
  used for search or retrieval tasks.

  <Warning>
    Search capabilities are currently only supported by Google Gemini. To take
    advantage of this event, your pipeline must include a
    [`GoogleLLMService`](/server/services/llm/gemini) and your pipeline task
    should include the
    [`GoogleRTVIObserver`](/server/frameworks/rtvi/google-rtvi-observer) in lieu
    of the typical `RTVIObserver`.
  </Warning>

  <Expandable title="BotLLMSearchResponseData">
    <ParamField path="search_result" type="string">
      The search result text.
    </ParamField>

    <ParamField path="rendered_content" type="string">
      The rendered content of the search result.
    </ParamField>

    <ParamField path="origins" type="LLMSearchOrigin[]">
      The origins of the search result.

      <Expandable title="LLMSearchOrigin">
        <ParamField path="site_uri" type="string">
          The URI of the site where the search result was found.
        </ParamField>

        <ParamField path="site_title" type="string">
          The title of the site where the search result was found.
        </ParamField>

        <ParamField path="results" type="LLMSearchResult[]">
          The individual search results.

          <Expandable title="LLMSearchResult">
            <ParamField path="text" type="string">
              The text of the search result.
            </ParamField>

            <ParamField path="confidence" type="number[]">
              The confidence scores for the search result.
            </ParamField>
          </Expandable>
        </ParamField>
      </Expandable>
    </ParamField>
  </Expandable>
</ParamField>

<ParamField path="onBotLlmText" type="BotLLMTextData">
  Streamed LLM token response text generated by the LLM service.

  <Expandable title="BotLLMTextData">
    <ParamField path="text" type="string">
      The text of the LLM response.
    </ParamField>
  </Expandable>
</ParamField>

<ParamField path="onBotLlmStarted">LLM service inference started.</ParamField>

<ParamField path="onBotLlmStopped">LLM service inference concluded.</ParamField>

<ParamField path="onBotTtsText" type="BotTTSTextData">
  If your TTS service supports streamed responses over sockets, the text
  parameter contains the words from TTS service as they are spoken. If you are
  using a HTTP based TTS service, the text parameter will contain the full text
  of the TTS response.

  <Expandable title="BotTTSTextData" defaultOpen="true">
    <ParamField path="text" type="string">
      The text of the LLM response.
    </ParamField>
  </Expandable>
</ParamField>

<ParamField path="onBotTtsStarted">TTS service started inference.</ParamField>

<ParamField path="onBotTtsStopped">TTS service inference concluded.</ParamField>

### Other

<ParamField path="onMetrics" default="data:PipecatMetricsData">
  Pipeline mterics data provided by Pipecat. [Learn
  more](/guides/fundamentals/metrics).
</ParamField>

## Events

Each callback described above has a corresponding event that can be listened for using the `.on()` method. This allows you to handle the same functionality using either callbacks or event listeners, depending on your preferred architecture.

Here's the complete reference mapping events to their corresponding callbacks:

### State and connectivity Events

| Event Name              | Callback Name             | Data Type        |
| ----------------------- | ------------------------- | ---------------- |
| `Connected`             | `onConnected`             | -                |
| `Disconnected`          | `onDisconnected`          | -                |
| `TransportStateChanged` | `onTransportStateChanged` | `TransportState` |
| `BotReady`              | `onBotReady`              | `BotReadyData`   |
| `BotConnected`          | `onBotConnected`          | -                |
| `BotDisconnected`       | `onBotDisconnected`       | `Participant`    |
| `ParticipantConnected`  | `onParticipantJoined`     | `Participant`    |
| `ParticipantLeft`       | `onParticipantLeft`       | `Participant`    |

### Message and Error Events

| Event Name      | Callback Name     | Data Type     |
| --------------- | ----------------- | ------------- |
| `ServerMessage` | `onServerMessage` | `any`         |
| `MessageError`  | `onMessageError`  | `RTVIMessage` |
| `Error`         | `onError`         | `RTVIMessage` |

### Media Events

| Event Name             | Callback Name            | Data Type                       |
| ---------------------- | ------------------------ | ------------------------------- |
| `TrackStarted`         | `onTrackStarted`         | `MediaStreamTrack, Participant` |
| `TrackStopped`         | `onTrackStopped`         | `MediaStreamTrack, Participant` |
| `AvailableMicsUpdated` | `onAvailableMicsUpdated` | `MediaDeviceInfo[]`             |
| `AvailableCamsUpdated` | `onAvailableCamsUpdated` | `MediaDeviceInfo[]`             |
| `MicUpdated`           | `onMicUpdated`           | `MediaDeviceInfo`               |
| `CamUpdated`           | `onCamUpdated`           | `MediaDeviceInfo`               |
| `SpeakerUpdated`       | `onSpeakerUpdated`       | `MediaDeviceInfo`               |
| `DeviceError`          | `onDeviceError`          | `DeviceError`                   |

### Audio Activity Events

| Event Name            | Callback Name           | Data Type             |
| --------------------- | ----------------------- | --------------------- |
| `LocalAudioLevel`     | `onLocalAudioLevel`     | `number`              |
| `RemoteAudioLevel`    | `onRemoteAudioLevel`    | `number, Participant` |
| `BotStartedSpeaking`  | `onBotStartedSpeaking`  | -                     |
| `BotStoppedSpeaking`  | `onBotStoppedSpeaking`  | -                     |
| `UserStartedSpeaking` | `onUserStartedSpeaking` | -                     |
| `UserStoppedSpeaking` | `onUserStoppedSpeaking` | -                     |

### Text and Transcription Events

| Event Name          | Callback Name         | Data Type        |
| ------------------- | --------------------- | ---------------- |
| `UserTranscript`    | `onUserTranscript`    | `TranscriptData` |
| `BotOutput`         | `onBotOutput`         | `BotOutputData`  |
| ~~`BotTranscript`~~ | ~~`onBotTranscript`~~ | `BotLLMTextData` |
| `BotLlmText`        | `onBotLlmText`        | `BotLLMTextData` |
| `BotTtsText`        | `onBotTtsText`        | `BotTTSTextData` |

### Service State Events

| Event Name             | Callback Name            | Data Type                  |
| ---------------------- | ------------------------ | -------------------------- |
| `BotLlmSearchResponse` | `onBotLlmSearchResponse` | `BotLLMSearchResponseData` |
| `BotLlmStarted`        | `onBotLlmStarted`        | -                          |
| `BotLlmStopped`        | `onBotLlmStopped`        | -                          |
| `BotTtsStarted`        | `onBotTtsStarted`        | -                          |
| `BotTtsStopped`        | `onBotTtsStopped`        | -                          |

### Other Events

| Event Name | Callback Name | Data Type            |
| ---------- | ------------- | -------------------- |
| `Metrics`  | `onMetrics`   | `PipecatMetricsData` |

## Usage Example

```typescript  theme={null}
import { PipecatClient, RTVIEvent } from "@pipecat-ai/client-js";

// Using callbacks
const pcClient = new PipecatClient({
  callbacks: {
    onBotReady: () => console.log("Bot ready via callback"),
    onUserTranscript: (data) => console.log("Transcript:", data.text),
  },
});

// Alternate approach: Using event listeners
pcClient.on(RTVIEvent.BotReady, () => {
  console.log("Bot ready via event");
});
```

## Transport Compatibility

<TransportTable rows={callbacks()} />

---

# Source: https://docs.pipecat.ai/server/services/tts/camb.md

> ## Documentation Index
> Fetch the complete documentation index at: https://docs.pipecat.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Camb AI

> Text-to-speech service using Camb AI's MARS models for high-quality speech synthesis

## Overview

`CambTTSService` provides high-quality text-to-speech synthesis using Camb AI's MARS model family with streaming capabilities. The service offers multiple model options optimized for different use cases: `mars-flash` for fast inference and `mars-pro` for high-quality output.

<CardGroup cols={2}>
  <Card title="Camb AI TTS API Reference" icon="code" href="https://reference-server.pipecat.ai/en/latest/api/pipecat.services.camb.tts.html">
    Pipecat's API methods for Camb AI TTS integration
  </Card>

  <Card title="Example Implementation" icon="play" href="https://github.com/pipecat-ai/pipecat/blob/main/examples/foundational/07zg-interruptible-camb.py">
    Complete example with interruption handling
  </Card>

  <Card title="Camb AI Documentation" icon="book" href="https://docs.camb.ai/">
    Official Camb AI API documentation
  </Card>

  <Card title="Camb AI Platform" icon="microphone" href="https://studio.camb.ai/">
    Access the Camb AI studio platform
  </Card>
</CardGroup>

## Installation

To use Camb AI services, install the required dependencies:

```bash  theme={null}
pip install "pipecat-ai[camb]"
```

## Prerequisites

### Camb AI Account Setup

Before using Camb AI TTS services, you need:

1. **Camb AI Account**: Sign up at [Camb AI](https://studio.camb.ai/)
2. **API Key**: Generate an API key from your account dashboard
3. **Voice Selection**: Choose voice IDs from the platform

### Required Environment Variables

* `CAMB_API_KEY`: Your Camb AI API key for authentication

## Notes

<Tip>
  Set the `audio_out_sample_rate` in `PipelineParams` to match the model's sample rate (22050 for `mars-flash`, 48000 for `mars-pro`) for optimal quality. See the [example implementation](https://github.com/pipecat-ai/pipecat/blob/main/examples/foundational/07zg-interruptible-camb.py) for usage.
</Tip>

---

# Source: https://docs.pipecat.ai/deployment/pipecat-cloud/guides/capacity-planning.md

> ## Documentation Index
> Fetch the complete documentation index at: https://docs.pipecat.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Capacity Planning

> Optimize your agent deployments to minimize cold starts

## Overview

By default, Pipecat Cloud auto-scales your agent to help minimize the likelihood that your agents experience a cold start. For many applications, you can simply set the `--min-agents` parameter to 1 in order to avoid scaling from zero and let Pipecat Cloud handle the rest.

However, for applications where traffic can fluctuate, you may need to plan for additional warm capacity to ensure your agents are always ready to respond immediately. For those cases, this guide will help you understand how warm capacity works in Pipecat Cloud, when you need to plan to use reserves, and how you can optimize your plan for both performance and cost.

## Agent Types

Before discussing capacity planning, it's important to understand the different types of agent instances in Pipecat Cloud:

* **Active Agents**: Agent instances currently running and handling user sessions.
* **Idle Agents**: When you start an active agent, Pipecat Cloud automatically provisions an additional idle agent to help with scaling. These take approximately 30 seconds to become available, though this time may vary based on system load and image size. You are not charged for these idle agents.
* **Reserved Agents**: Agent instances maintained according to your `--min-agents` deployment setting, ensuring immediate availability regardless of current traffic.

## Understanding Warm Capacity

Your "warm capacity" represents agent instances that are immediately available to handle new sessions without a cold start. Pipecat Cloud automatically manages this based on:

1. Your configured **reserved agents** (`min-agents`)
2. Your current **active sessions**
3. Automatically provisioned **idle agents**

The system ensures you always have the following warm capacity available:

* When **Reserved > Active**: Your warm capacity equals the number of **Reserved Agents**
* When **Active ≥ Reserved**: Your warm capacity equals the number of **Active Agents** (through automatically provisioned idle agents)

To illustrate the point, here are a few scenarios showing how warm capacity works:

| Reserved | Active | Warm Capacity |
| -------- | ------ | ------------- |
| 10       | 1      | 10            |
| 10       | 10     | 10            |
| 1        | 10     | 10            |

This shows how:

* Reserved agents provide a guaranteed minimum warm capacity
* As active sessions increase beyond your reserved count, your warm capacity grows to match through automatically provisioned idle agents

## Agent Cooldown

When an active session ends, the agent instance behavior is governed by a cooldown period:

* The agent instance remains available in your warm capacity pool for a 5-minute cooldown period
* During this time, it can immediately serve another request without any cold start
* After the 5-minute cooldown expires, if the agent instance hasn't been used, it will be terminated
* This cooldown provides a buffer in your warm capacity pool, helping to smooth transitions between traffic peaks

## Planning for Traffic Patterns

To determine the optimal reserved instance count for your deployment, consider:

1. **Peak Concurrent Sessions**: How many simultaneous sessions do you expect during peak periods?

2. **Growth Rate**: How quickly do new sessions start during peak periods?

   * If new sessions start faster than the \~30 second warm-up time for auto-scaled agent instances, you'll need more reserved capacity

3. **Cold Start Tolerance**: How important is immediate response for your use case?

<Tip>
  Completely avoiding cold starts may not be practical for all applications. We
  strongly recommend considering longer or variable start up times when building
  your application. For phone use cases, consider a hold message or for web
  apps, consider a waiting UX or message.
</Tip>

## Cost-Efficient Scaling Strategies

* **Development/Testing**: Use `min-agents: 0` to minimize costs during development
* **Production Voice AI**: Set `min-agents` to cover your baseline traffic to avoid cold starts
* **Time-Based Scaling**: Consider modifying your reserved count for known high-traffic periods
* **Monitoring**: Regularly review your warm capacity utilization to optimize your configuration

<Info>A cold start typically takes around 10 seconds.</Info>

## Calculating Optimal Reserved Agents

For production deployments where immediate response is essential, you can calculate your optimal reserved agent count using a growth rate approach:

```
Optimal Reserved = MAX(Baseline Sessions, CPS × Idle Creation Delay)
```

Where:

* **Baseline Sessions**: Minimum concurrent sessions you typically maintain
* **CPS (Calls Per Second)**: Your expected session growth rate during peak periods
* **Idle Creation Delay**: Time for new idle agents to become available (\~30 seconds)

This formula addresses the fundamental challenge: "Can our warm capacity creation keep pace with our call growth rate?" By reserving capacity based on your growth rate and the idle creation delay, you ensure sufficient capacity is available during the critical period before auto-scaling can catch up.

### Growth Rate Examples

Here's how the formula works with different growth rates:

| Scenario      | Baseline | CPS                | Idle Creation Delay | Calculation       | Optimal Reserved |
| ------------- | -------- | ------------------ | ------------------- | ----------------- | ---------------- |
| High volume   | 10       | 1.0 (1 call/sec)   | 30s                 | MAX(10, 1.0 × 30) | 30               |
| Medium volume | 10       | 0.5 (1 call/2sec)  | 30s                 | MAX(10, 0.5 × 30) | 15               |
| Low volume    | 10       | 0.1 (1 call/10sec) | 30s                 | MAX(10, 0.1 × 30) | 10               |

### Call Center Example

Consider a voice AI call center that:

* Normally handles 10 concurrent calls (baseline)
* During promotions, receives new calls at a rate of 1 call per second
* Has a 30-second idle creation delay

Applying our formula:

```
Optimal Reserved = MAX(10, 1 × 30)
                 = MAX(10, 30)
                 = 30
```

With 30 reserved agents:

* You can handle the growth rate of 1 call per second for the full 30 seconds until new idle agents start becoming available
* This prevents any cold starts during the critical catch-up period
* Auto-scaling will create additional idle agents to handle continued growth

### Understanding the Growth Rate Approach

This approach works because:

1. When your call volume starts increasing, you immediately begin consuming your warm capacity
2. At the same time, each new active call triggers the creation of a new idle agent
3. However, these new idle agents take \~30 seconds to become available
4. Your reserved capacity must be sufficient to handle all calls during this 30-second "catch-up period"

### Trading Cost for Performance

The formula provides a starting point, which you can adjust based on your specific needs:

* **Cost-sensitive**: Use a lower reserved count and accept some cold starts during traffic spikes
* **Performance-sensitive**: Use the calculated reserved count to ensure zero cold starts
* **Hybrid approach**: Monitor your actual traffic patterns and adjust based on real-world performance

<Tip>
  For most production voice AI applications, we recommend using at least the
  calculated optimal reserved agent count during business hours or peak usage
  periods, and then scaling down during off-hours to optimize costs.
</Tip>

## Summary and Next Steps

Proper capacity planning ensures your agents are always ready to respond immediately, providing the best possible user experience while optimizing costs:

1. **Understand your traffic patterns**: Baseline sessions, peak sessions, and growth rate
2. **Calculate your optimal reserved agents**: Use the formula to determine the right level of warm capacity
3. **Monitor and adjust**: Refine your capacity planning based on real-world performance and costs

<CardGroup cols={2}>
  <Card title="Explore Scaling Options" icon="server" href="../fundamentals/scaling">
    Learn more about scaling configuration options and deployment strategies.
  </Card>

  <Card title="Deploy Your Agent" icon="rocket" href="../fundamentals/deploy">
    Apply your capacity planning knowledge to your agent deployment.
  </Card>
</CardGroup>

---

# Source: https://docs.pipecat.ai/server/services/tts/cartesia.md

# Source: https://docs.pipecat.ai/server/services/stt/cartesia.md

> ## Documentation Index
> Fetch the complete documentation index at: https://docs.pipecat.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Cartesia

> Speech-to-text service implementation using Cartesia's real-time transcription API

## Overview

`CartesiaSTTService` provides real-time speech recognition using Cartesia's WebSocket API with the `ink-whisper` model, supporting streaming transcription with both interim and final results for low-latency applications.

<CardGroup cols={2}>
  <Card title="Cartesia STT API Reference" icon="code" href="https://reference-server.pipecat.ai/en/latest/api/pipecat.services.cartesia.stt.html">
    Pipecat's API methods for Cartesia STT integration
  </Card>

  <Card title="Example Implementation" icon="play" href="https://github.com/pipecat-ai/pipecat/blob/main/examples/foundational/13f-cartesia-transcription.py">
    Complete example with transcription logging
  </Card>

  <Card title="Cartesia Documentation" icon="book" href="https://docs.cartesia.ai/api-reference/stt/stt">
    Official Cartesia STT documentation and features
  </Card>

  <Card title="Cartesia Platform" icon="microphone" href="https://cartesia.ai/">
    Access API keys and transcription models
  </Card>
</CardGroup>

## Installation

To use Cartesia services, install the required dependency:

```bash  theme={null}
pip install "pipecat-ai[cartesia]"
```

## Prerequisites

### Cartesia Account Setup

Before using Cartesia STT services, you need:

1. **Cartesia Account**: Sign up at [Cartesia](https://cartesia.ai/)
2. **API Key**: Generate an API key from your account dashboard
3. **Model Access**: Ensure access to the ink-whisper transcription model

### Required Environment Variables

* `CARTESIA_API_KEY`: Your Cartesia API key for authentication

---

# Source: https://docs.pipecat.ai/server/services/llm/cerebras.md

> ## Documentation Index
> Fetch the complete documentation index at: https://docs.pipecat.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Cerebras

> LLM service implementation using Cerebras's API with OpenAI-compatible interface

## Overview

`CerebrasLLMService` provides access to Cerebras's language models through an OpenAI-compatible interface. It inherits from `OpenAILLMService` and supports streaming responses, function calling, and context management with ultra-fast inference speeds.

<CardGroup cols={2}>
  <Card title="Cerebras LLM API Reference" icon="code" href="https://reference-server.pipecat.ai/en/latest/api/pipecat.services.cerebras.llm.html">
    Pipecat's API methods for Cerebras integration
  </Card>

  <Card title="Example Implementation" icon="play" href="https://github.com/pipecat-ai/pipecat/blob/main/examples/foundational/14k-function-calling-cerebras.py">
    Complete example with function calling
  </Card>

  <Card title="Cerebras Documentation" icon="book" href="https://inference-docs.cerebras.ai/api-reference/chat-completions">
    Official Cerebras inference API documentation
  </Card>

  <Card title="Cerebras Platform" icon="microphone" href="https://cloud.cerebras.ai/">
    Access models and manage API keys
  </Card>
</CardGroup>

## Installation

To use Cerebras services, install the required dependency:

```bash  theme={null}
pip install "pipecat-ai[cerebras]"
```

## Prerequisites

### Cerebras Account Setup

Before using Cerebras LLM services, you need:

1. **Cerebras Account**: Sign up at [Cerebras Cloud](https://cloud.cerebras.ai/)
2. **API Key**: Generate an API key from your account dashboard
3. **Model Selection**: Choose from available Cerebras models with ultra-fast inference

### Required Environment Variables

* `CEREBRAS_API_KEY`: Your Cerebras API key for authentication

---

# Source: https://docs.pipecat.ai/deployment/platforms/cerebrium.md

> ## Documentation Index
> Fetch the complete documentation index at: https://docs.pipecat.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Cerebrium

> Deploy Pipecat applications to Cerebrium

[Cerebrium](https://www.cerebrium.ai) is a serverless Infrastructure platform that makes it easy for companies to build, deploy and scale AI applications.
Cerebrium offers both CPUs and GPUs (H100s, A100s etc) with extremely low cold start times allowing us to create highly performant applications in the most cost efficient manner.

### Install the Cerebrium CLI

To get started, let us run the following commands:

1. Run `pip install cerebrium` to install the Python package.
2. Run `cerebrium login` to authenticate yourself.

If you don't have a Cerebrium account, you can create one and get started with \$30 in free credits.

### Create a Cerebrium project

1. Create a new Cerebrium project:

```bash  theme={null}
cerebrium init pipecat-agent
```

2. This will create two key files:

* `main.py` - Your application entrypoint
* `cerebrium.toml` - Configuration for build and environment settings

Update your `cerebrium.toml` with the necessary configuration:

```toml  theme={null}
[cerebrium.hardware]
region = "us-east-1"
provider = "aws"
compute = "CPU"
cpu = 4
memory = 18.0

[cerebrium.dependencies.pip]
torch = ">=2.0.0"
"pipecat-ai[silero, daily, openai, cartesia]" = "latest"
aiohttp = "latest"
torchaudio = "latest"
```

In order for our application to work, we need to copy our API keys from the various platforms. Navigate to the Secrets section in your Cerebrium dashboard to store your API keys:

* `OPENAI_API_KEY` - We use OpenAI For the LLM. You can get your API key from [here](https://platform.openai.com/api-keys)
* `DAILY_TOKEN` - For WebRTC communication. You can get your token from [here](https://dashboard.daily.co/developers)
* `CARTERSIA_API_KEY` - For text-to-speech services. You can get your API key from [here](https://play.cartesia.ai/keys)

We access these secrets in our code as if they are normal ENV vars. For the above, You can swap in any LLM or TTS service you wish to use.

### Agent setup

We create a basic pipeline setup in our `main.py` that combines our LLM, TTS and Daily WebRTC transport layer.

```python  theme={null}
import os
import sys
import time

import aiohttp
from loguru import logger

from pipecat.audio.vad.silero import SileroVADAnalyzer
from pipecat.audio.vad.vad_analyzer import VADParams
from pipecat.frames.frames import EndFrame, LLMMessagesAppendFrame
from pipecat.pipeline.pipeline import Pipeline
from pipecat.pipeline.runner import PipelineRunner
from pipecat.pipeline.task import PipelineParams, PipelineTask
from pipecat.processors.aggregators.llm_context import LLMContext
from pipecat.processors.aggregators.llm_response_universal import LLMContextAggregatorPair
from pipecat.services.cartesia.tts import CartesiaTTSService
from pipecat.services.openai.llm import OpenAILLMService
from pipecat.transports.daily.transport import DailyParams, DailyTransport

logger.remove(0)
logger.add(sys.stderr, level="DEBUG")


async def main(room_url: str, token: str):
    async with aiohttp.ClientSession() as session:
        transport = DailyTransport(
            room_url,
            token,
            "Friendly bot",
            DailyParams(
                audio_in_enabled=True,
                audio_out_enabled=True,
                transcription_enabled=True,
            ),
        )

        messages = [
            {
                "role": "system",
                "content": "You are a helpful AI assistant that can switch between two services to showcase the difference in performance and cost: 'openai_realtime' and 'custom'. Respond to user queries and switch services when asked.",
            },
        ]

        llm = OpenAILLMService(
            name="LLM",
            api_key=os.environ.get("OPENAI_API_KEY"),
            model="gpt-4",
        )

        tts = CartesiaTTSService(
            api_key=os.getenv("CARTESIA_API_KEY"),
            voice_id="79a125e8-cd45-4c13-8a67-188112f4dd22",  # British Lady
        )

        vad_processor = VADProcessor(vad_analyzer=SileroVADAnalyzer(params=VADParams(stop_secs=0.2)))

        custom_context = LLMContext(messages)
        custom_user_aggregator, custom_assistant_aggregator = LLMContextAggregatorPair(context)
        context_aggregator_custom = LLMContextAggregatorPair(custom_context)

        pipeline = Pipeline(
            [
                transport.input(),  # Transport user input
                vad_processor,
                custom_user_aggregator,
                llm,
                tts,
                custom_assistant_aggregator,
                transport.output(),  # Transport bot output
            ]
        )

        task = PipelineTask(
            pipeline,
            params=PipelineParams(
                enable_metrics=True,
                enable_usage_metrics=True,
            ),
        )

        @transport.event_handler("on_first_participant_joined")
        async def on_first_participant_joined(transport, participant):
            transport.capture_participant_transcription(participant["id"])
            time.sleep(1.5)
            await task.queue_frame(
                LLMMessagesAppendFrame(
                    [
                        {
                            "role": "system",
                            "content": "Introduce yourself.",
                        }
                    ],
                    run_llm=True,
                )
            )

        @transport.event_handler("on_participant_left")
        async def on_participant_left(transport, participant, reason):
            await task.queue_frame(EndFrame())

        @transport.event_handler("on_call_state_updated")
        async def on_call_state_updated(transport, state):
            if state == "left":
                await task.queue_frame(EndFrame())

        runner = PipelineRunner()
        await runner.run(task)
        await session.close()
```

First, in our main function, we initialize the daily transport layer to receive/send the audio/video data from the Daily room we will connect to. You can see we pass the room\_url we would like to join as well as a token to authenticate us programmatically joining. We also set our VAD stop seconds which is the amount of time we wait for a pause before our bot will respond - in this example, we set it to 600 milliseconds.

Next we connect to our LLM (OpenAI) as well as our TTS model (Cartesia). By setting 'transcription\_enabled=true' we are using the STT from Daily itself. This is where the Pipecat framework helps convert audio data to text and vice versa.

We then put this all together as a PipelineTask which is what Pipecat runs all together. The make up of a task is completely customizable and has support for Image and Vision use cases. Lastly, we have some event handlers for when a user joins/leaves the room.

### Deploy bot

Deploy your application to Cerebrium:

```bash  theme={null}
cerebrium deploy
```

You will then see that an endpoint is created for your bot at `POST \<BASE_URL\>/main` that you can call with your room\_url and token. Let us test it.

### Test it out

```python  theme={null}
def create_room():
    url = "https://api.daily.co/v1/rooms/"
    headers = {
        "Content-Type": "application/json",
        "Authorization": f"Bearer {os.environ.get('DAILY_TOKEN')}",
    }
    data = {
        "properties": {
            "exp": int(time.time()) + 60 * 5,  ##5 mins
            "eject_at_room_exp": True,
        }
    }

    response = requests.post(url, headers=headers, json=data)
    if response.status_code == 200:
        room_info = response.json()
        token = create_token(room_info["name"])
        if token and "token" in token:
            room_info["token"] = token["token"]
        else:
            logger.error("Failed to create token")
            return {
                "message": "There was an error creating your room",
                "status_code": 500,
            }
        return room_info
    else:
        logger.error(f"Failed to create room: {response.status_code}")
        return {"message": "There was an error creating your room", "status_code": 500}


def create_token(room_name: str):
    url = "https://api.daily.co/v1/meeting-tokens"
    headers = {
        "Content-Type": "application/json",
        "Authorization": f"Bearer {os.environ.get('DAILY_TOKEN')}",
    }
    data = {"properties": {"room_name": room_name, "is_owner": True}}

    response = requests.post(url, headers=headers, json=data)
    if response.status_code == 200:
        token_info = response.json()
        return token_info
    else:
        logger.error(f"Failed to create token: {response.status_code}")
        return None

if __name__ == "__main__":
    room_info = create_room()
    print(f"Join room: {room_info["url"]}")
    asyncio.run(main(room_info["url"], room_info["token"]))

```

## Future Considerations

Since Cerebrium supports both CPU and GPU workloads if you would like to lower the latency of your application then the best would be to get model weights from various providers and run them locally.
You can do this for:

* LLM: Run any OpenSource model using a framework such as vLLM
* TTS: Both PlayHt and Deepgram offer TTS models that can be run locally
* STT: Deepgram offers a local model that can be run locally

If you implement all three models locally, you should have much better performance. We have been able to get \~300ms voice-to-voice responses.

## Examples

* [Fastest voice agent](https://docs.cerebrium.ai/v4/examples/realtime-voice-agents): Local only implementation
* [RAG voice agent](https://www.cerebrium.ai/blog/creating-a-realtime-rag-voice-agent): Create a voice agent that can do RAG using Cerebrium + OpenAI + Pinecone
* [Twilio voice agent](https://docs.cerebrium.ai/v4/examples/twilio-voice-agent): Create a voice agent that can receive phone calls via Twilio
* [OpenAI Realtime API implementation](https://www.cerebrium.ai/blog/an-alternative-to-openai-realtime-api-for-voice-capabilities): Create a voice agent that can receive phone calls via OpenAI Realtime API

---

# Source: https://docs.pipecat.ai/deployment/pipecat-cloud/guides/ci-with-github-actions.md

> ## Documentation Index
> Fetch the complete documentation index at: https://docs.pipecat.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# CI with GitHub Actions

> Automate image builds and deploys for continuous integration

Automate your Pipecat Cloud deployment workflow using GitHub Actions. This guide shows you how to automatically build and deploy your agent whenever you push changes to your main branch.

## Prerequisites

Before setting up your GitHub Actions workflow, ensure you have:

* A Pipecat Cloud account with an API token
* A Docker Hub account (or another container registry)
* An image pull secret configured in Pipecat Cloud
* A secret set configured for your agent

## Configure GitHub Secrets

Add the following secrets to your GitHub repository settings (Settings → Secrets and variables → Actions):

* `DOCKERHUB_USERNAME`: Your Docker Hub username
* `DOCKERHUB_TOKEN`: Your Docker Hub access token
* `PCC_API_KEY`: Your Pipecat Cloud private API token

<Note>
  For Docker Hub access tokens, create a personal access token from your Docker
  Hub account settings rather than using your password.
</Note>

## Basic Workflow Configuration

Create a file at `.github/workflows/deploy.yml` in your repository:

```yml  theme={null}
name: Docker Image CI

on:
  push:
    branches: ["main"]

jobs:
  deploy:
    runs-on: ubuntu-latest

    steps:
      - name: Checkout
        uses: actions/checkout@v4

      - name: Get date
        id: date
        run: echo "date=$(date +'%F')" >> $GITHUB_OUTPUT

      - name: Set up QEMU
        uses: docker/setup-qemu-action@v3

      - name: Login to Docker Hub
        uses: docker/login-action@v3
        with:
          username: ${{ secrets.DOCKERHUB_USERNAME }}
          password: ${{ secrets.DOCKERHUB_TOKEN }}

      - name: Build and Push
        uses: docker/build-push-action@v5
        with:
          platforms: linux/arm64
          context: .
          push: true
          tags: |
            ${{ secrets.DOCKERHUB_USERNAME }}/${{ github.event.repository.name }}:latest
            ${{ secrets.DOCKERHUB_USERNAME }}/${{ github.event.repository.name }}:${{ steps.date.outputs.date }}

      - name: Deploy to Pipecat Cloud
        run: |
          curl -X POST https://api.pipecat.daily.co/v1/agents/${{ github.event.repository.name }} \
            -H "Authorization: Bearer ${{ secrets.PCC_API_KEY }}" \
            -H "Content-Type: application/json" \
            -d '{
              "image": "${{ secrets.DOCKERHUB_USERNAME }}/${{ github.event.repository.name }}:${{ steps.date.outputs.date }}",
              "imagePullSecretSet": "my-image-pull-secret",
              "secretSet": "my-secret-set",
              "krispViva": {
                "audioFilter": "tel"
              },
              "autoScaling": {
                "minAgents": 0,
                "maxAgents": 10
              }
            }'
```

This workflow will:

1. Trigger on every push to the `main` branch
2. Build your Docker image for `linux/arm64` platform
3. Push the image with both `latest` and date-based tags
4. Deploy the new image to Pipecat Cloud

## Image Tagging Strategy

The example workflow creates two tags for each build:

* `latest`: Always points to the most recent build
* Date-based (e.g., `2025-10-30`): Specific version tied to the build date

<Warning>
  Avoid pushing multiple builds to the same image tag. Pipecat Cloud instances
  may cache older versions of an image tag, which can cause running agents to
  use outdated code even after redeployment. Use unique tags (like date-based or
  commit SHA tags) for each build.
</Warning>

If you deploy more frequently than once per day, consider using commit SHA-based tags instead:

```yml  theme={null}
- name: Build and Push
  uses: docker/build-push-action@v5
  with:
    platforms: linux/arm64
    context: .
    push: true
    tags: |
      ${{ secrets.DOCKERHUB_USERNAME }}/${{ github.event.repository.name }}:latest
      ${{ secrets.DOCKERHUB_USERNAME }}/${{ github.event.repository.name }}:${{ github.sha }}
```

Then update the deployment step to use `${{ github.sha }}` as the tag.

## Monorepo Configuration

If your agent code lives in a subdirectory (e.g., `/server`, like code generated from the Pipecat CLI), modify the workflow to set the working directory and build context:

```yml  theme={null}
name: Docker Image CI

on:
  push:
    branches: ["main"]
    paths:
      - "server/**"

jobs:
  deploy:
    runs-on: ubuntu-latest
    defaults:
      run:
        working-directory: ./server

    steps:
      - name: Checkout
        uses: actions/checkout@v4

      - name: Get date
        id: date
        run: echo "date=$(date +'%F')" >> $GITHUB_OUTPUT

      - name: Set up QEMU
        uses: docker/setup-qemu-action@v3

      - name: Login to Docker Hub
        uses: docker/login-action@v3
        with:
          username: ${{ secrets.DOCKERHUB_USERNAME }}
          password: ${{ secrets.DOCKERHUB_TOKEN }}

      - name: Build and Push
        uses: docker/build-push-action@v5
        with:
          platforms: linux/arm64
          context: ./server
          push: true
          tags: |
            ${{ secrets.DOCKERHUB_USERNAME }}/${{ github.event.repository.name }}:latest
            ${{ secrets.DOCKERHUB_USERNAME }}/${{ github.event.repository.name }}:${{ steps.date.outputs.date }}

      - name: Deploy to Pipecat Cloud
        run: |
          curl -X POST https://api.pipecat.daily.co/v1/agents/${{ github.event.repository.name }} \
            -H "Authorization: Bearer ${{ secrets.PCC_API_KEY }}" \
            -H "Content-Type: application/json" \
            -d '{
              "image": "${{ secrets.DOCKERHUB_USERNAME }}/${{ github.event.repository.name }}:${{ steps.date.outputs.date }}",
              "imagePullSecretSet": "my-image-pull-secret",
              "secretSet": "my-secret-set",
              "krispViva": {
                "audioFilter": "tel"
              },
              "autoScaling": {
                "minAgents": 0,
                "maxAgents": 10
              }
            }'
```

The `paths` filter ensures the workflow only runs when files in the `server/` directory change, preventing unnecessary builds.

## Platform Considerations

Pipecat Cloud requires images built for the `linux/arm64` platform. The workflow uses QEMU to enable ARM64 builds on GitHub's x64 runners.

<Note>
  As of October 2025, GitHub still does not provide ARM runners for private
  repositories. The workflow uses QEMU emulation, which can significantly
  increase build times. If you have a public repository or when GitHub makes ARM
  runners available, you can remove the QEMU setup step and use native ARM
  runners for faster builds.
</Note>

To use native ARM runners when available:

```yml  theme={null}
jobs:
  deploy:
    runs-on: ubuntu-24.04-arm # Use ARM runner

    steps:
      - name: Checkout
        uses: actions/checkout@v4

      # Remove the "Set up QEMU" step entirely
      - name: Login to Docker Hub
        uses: docker/login-action@v3
        # ... rest of workflow
```

## Customizing Deployment Configuration

The deployment step in the workflow configures your agent using the Pipecat Cloud REST API. You can customize the following parameters:

* `image`: The Docker image tag to deploy
* `imagePullSecretSet`: Reference to your image pull credentials (replace `my-image-pull-secret` with your actual secret name)
* `secretSet`: Reference to your environment secrets (replace `my-secret-set` with your actual secret set name)
* `krispViva.audioFilter`: The Krisp VIVA audio filter model to use (replace `tel` with your actual filter model)
* `autoScaling.minAgents`: Minimum number of agent instances to maintain
* `autoScaling.maxAgents`: Maximum number of agent instances allowed

<Tip>
  Adjust the `minAgents` and `maxAgents` values based on your expected traffic
  patterns. See our [Capacity Planning guide](./capacity-planning) for guidance
  on optimizing these settings.
</Tip>

## Next Steps

<CardGroup cols={2}>
  <Card title="Secrets Management" icon="key" href="../fundamentals/secrets">
    Learn how to securely manage image pull secrets and environment variables.
  </Card>

  <Card title="Scaling Configuration" icon="chart-line" href="../fundamentals/scaling">
    Configure auto-scaling to handle varying traffic loads.
  </Card>
</CardGroup>

---

# Source: https://docs.pipecat.ai/client/js/api-reference/client-constructor.md

> ## Documentation Index
> Fetch the complete documentation index at: https://docs.pipecat.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# PipecatClient Constructor

> Setting up the PipecatClient

```javascript  theme={null}
import { PipecatClient } from "@pipecat-ai/client-js";
```

`PipecatClient` is the primary component for building the client-side portion of a client-bot interaction. It is designed to work with various transport layers, such as WebRTC, WebSockets, or HTTP, allowing you to pick and choose the communication layer that best suits your application while maintaining a consistent API.

<Note>
  When initializing the `PipecatClient`, you must provide a transport instance to the constructor for your chosen protocol or provider. See [Transport](/client/js/transports) for more information. For the purpose of this guide, we'll demonstrate using the [Daily WebRTC transport](/client/js/transports/daily).
</Note>

## Example

```typescript  theme={null}
import { RTVIEvent, RTVIMessage, PipecatClient } from "@pipecat-ai/client-js";
import { DailyTransport } from "@pipecat-ai/daily-transport";

const PipecatClient = new PipecatClient({
  transport: new DailyTransport(),
  enableMic: true,
  enableCam: false,
  enableScreenShare: false,
  timeout: 15 * 1000,
  callbacks: {
    onConnected: () => {
      console.log("[CALLBACK] User connected");
    },
    onDisconnected: () => {
      console.log("[CALLBACK] User disconnected");
    },
    onTransportStateChanged: (state: string) => {
      console.log("[CALLBACK] State change:", state);
    },
    onBotConnected: () => {
      console.log("[CALLBACK] Bot connected");
    },
    onBotDisconnected: () => {
      console.log("[CALLBACK] Bot disconnected");
    },
    onBotReady: () => {
      console.log("[CALLBACK] Bot ready to chat!");
    },
  },
});
```

***

## API reference

### transport

<ParamField path="transport" type="Class<Transport>" default="undefined" required>
  An instance of the `Transport` type you will use to connect to your bot service (`PipecatClient.connect()`). Transports implement the underlying device management, connectivity, media transmission, and state logic that manage the lifecycle of your session.

  <Note>
    As a best practice, we recommend you construct the transport inline in the client constructor, as opposed to holding a reference to it. Access to the transport is typically unnecessary. For advanced use cases that do require access to the transport, we recommend doing so via the `PipecatClient.transport` property, which provides some additional safeguards.
  </Note>

  ```typescript  theme={null}
  import { PipecatClient } from "@pipecat-ai/client-js";
  import { DailyTransport } from "@pipecat-ai/daily-transport";

  const pcClient = new PipecatClient({
    transport: new DailyTransport(),
  });
  ```
</ParamField>

### callbacks

<ParamField path="callbacks" type="RTVIEventCallbacks">
  Map of callback functions. See [callbacks](./callbacks).
</ParamField>

### Media Initialization

<ParamField path="enableMic" type="boolean" default="true">
  Enable user's local microphone device.
</ParamField>

<ParamField path="enableCam" type="boolean" default="false">
  Enable user's local webcam device. Note: Not all transports support video. Setting this value in that case will have no effect.
</ParamField>

<ParamField path="enableScreenShare" type="boolean" default="false">
  Enable user's local screen share. Note: Not all transports support screen sharing. Setting this value in that case will have no effect.
</ParamField>

---

# Source: https://docs.pipecat.ai/client/js/api-reference/client-methods.md

> ## Documentation Index
> Fetch the complete documentation index at: https://docs.pipecat.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Client Methods

export const TransportTable = rows => {
  const headers = ["", "DailyTransport", "SmallWebRTCTransport", "RNDailyTransport", "WebSocketTransport", "OpenAIRealTimeWebRTCTransport", "GeminiLiveWebsocketTransport"];
  if (!rows || !rows.rows || !Array.isArray(rows.rows)) {
    return <div>Error: Invalid rows data</div>;
  }
  return <table className="compatibility-table">
      <tr>
        {headers.map(header => <th key={header}>{header}</th>)}
      </tr>
      {rows.rows.map(entry => <tr key={entry.name}>
          <td>
            {entry.name}() {entry.deprecated ? "⚠️ DEPRECATED" : ""}
          </td>
          <td className={entry.dt ? "supported" : "unsupported"}>
            {entry.dt || "❌"}
          </td>
          <td className={entry.swrtc ? "supported" : "unsupported"}>
            {entry.swrtc || "❌"}
          </td>
          <td className={entry.rn ? "supported" : "unsupported"}>
            {entry.rn || "❌"}
          </td>
          <td className={entry.ws ? "supported" : "unsupported"}>
            {entry.ws || "❌"}
          </td>
          <td className={entry.oai ? "supported" : "unsupported"}>
            {entry.oai || "❌"}
          </td>
          <td className={entry.gem ? "supported" : "unsupported"}>
            {entry.gem || "❌"}
          </td>
        </tr>)}
    </table>;
};

export const methods = () => {
  return [{
    name: "startBot",
    dt: "1.2.0",
    swrtc: "1.2.0",
    rn: "1.2.0",
    ws: "1.2.0"
  }, {
    name: "connect",
    dt: "1.0.0",
    swrtc: "1.0.0",
    rn: "1.0.0",
    ws: "1.0.0",
    oai: "1.0.0",
    gem: "1.0.0"
  }, {
    name: "startBotAndConnect",
    dt: "1.2.0",
    swrtc: "1.2.0",
    rn: "1.2.0",
    ws: "1.2.0"
  }, {
    name: "disconnect",
    dt: "1.0.0",
    swrtc: "1.0.0",
    rn: "1.0.0",
    ws: "1.0.0",
    oai: "1.0.0",
    gem: "1.0.0"
  }, {
    name: "disconnectBot",
    dt: "1.0.0",
    swrtc: "1.0.0",
    rn: "1.0.0",
    ws: "1.0.0"
  }, {
    name: "sendClientMessage",
    dt: "1.0.0",
    swrtc: "1.0.0",
    rn: "1.0.0",
    ws: "1.0.0"
  }, {
    name: "sendClientRequest",
    dt: "1.0.0",
    swrtc: "1.0.0",
    rn: "1.0.0",
    ws: "1.0.0"
  }, {
    name: "initDevices",
    dt: "1.0.0",
    swrtc: "1.0.0",
    rn: "1.0.0",
    ws: "1.0.0",
    oai: "1.0.0",
    gem: "1.0.0"
  }, {
    name: "getAllMics",
    dt: "1.0.0",
    swrtc: "1.0.0",
    rn: "1.0.0",
    ws: "1.0.0",
    oai: "1.0.0",
    gem: "1.0.0"
  }, {
    name: "getAllCams",
    dt: "1.0.0",
    swrtc: "1.0.0",
    rn: "1.0.0",
    oai: "1.0.0"
  }, {
    name: "getAllSpeakers",
    dt: "1.0.0",
    swrtc: "1.0.0",
    rn: "1.0.0",
    ws: "1.0.0",
    oai: "1.0.0",
    gem: "1.0.0"
  }, {
    name: "selectedMic",
    dt: "1.0.0",
    swrtc: "1.0.0",
    rn: "1.0.0",
    ws: "1.0.0",
    oai: "1.0.0",
    gem: "1.0.0"
  }, {
    name: "selectedCam",
    dt: "1.0.0",
    swrtc: "1.0.0",
    rn: "1.0.0",
    oai: "1.0.0"
  }, {
    name: "selectedSpeaker",
    dt: "1.0.0",
    swrtc: "1.0.0",
    rn: "1.0.0",
    ws: "1.0.0",
    oai: "1.0.0",
    gem: "1.0.0"
  }, {
    name: "updateMic",
    dt: "1.0.0",
    swrtc: "1.0.0",
    rn: "1.0.0",
    ws: "1.0.0",
    oai: "1.0.0",
    gem: "1.0.0"
  }, {
    name: "updateCam",
    dt: "1.0.0",
    swrtc: "1.0.0",
    rn: "1.0.0",
    oai: "1.0.0"
  }, {
    name: "updateSpeaker",
    dt: "1.0.0",
    swrtc: "1.0.0",
    rn: "1.0.0",
    ws: "1.0.0",
    oai: "1.0.0",
    gem: "1.0.0"
  }, {
    name: "enableMic",
    dt: "1.0.0",
    swrtc: "1.0.0",
    rn: "1.0.0",
    ws: "1.0.0",
    oai: "1.0.0",
    gem: "1.0.0"
  }, {
    name: "enableCam",
    dt: "1.0.0",
    swrtc: "1.0.0",
    rn: "1.0.0",
    oai: "1.0.0"
  }, {
    name: "enableScreenShare",
    dt: "1.0.0",
    swrtc: "1.3.0",
    rn: "1.0.0"
  }, {
    name: "isMicEnabled",
    dt: "1.0.0",
    swrtc: "1.0.0",
    rn: "1.0.0",
    ws: "1.0.0",
    oai: "1.0.0",
    gem: "1.0.0"
  }, {
    name: "isCamEnabled",
    dt: "1.0.0",
    swrtc: "1.0.0",
    rn: "1.0.0",
    oai: "1.0.0"
  }, {
    name: "isSharingScreen",
    dt: "1.0.0",
    swrtc: "1.3.0",
    rn: "1.0.0"
  }, {
    name: "tracks",
    dt: "1.0.0",
    swrtc: "1.0.0",
    rn: "1.0.0",
    ws: "1.0.0",
    oai: "1.0.0",
    gem: "1.0.0"
  }, {
    name: "sendText",
    dt: "1.4.0",
    swrtc: "1.5.0",
    ws: "1.4.0",
    oai: "1.3.0",
    gem: "1.3.0"
  }, {
    name: "appendToContext",
    dt: "1.0.0",
    swrtc: "1.0.0",
    rn: "1.0.0",
    ws: "1.0.0",
    oai: "1.0.0",
    gem: "1.0.0"
  }, {
    name: "registerFunctionCallHandler",
    dt: "1.0.0",
    swrtc: "1.0.0",
    rn: "1.0.0",
    ws: "1.0.0",
    oai: "1.0.0"
  }, {
    name: "transport",
    dt: "1.0.0",
    swrtc: "1.0.0",
    rn: "1.0.0",
    ws: "1.0.0",
    oai: "1.0.0",
    gem: "1.0.0"
  }, {
    name: "setLogLevel",
    dt: "1.0.0",
    swrtc: "1.0.0",
    rn: "1.0.0",
    ws: "1.0.0",
    oai: "1.0.0",
    gem: "1.0.0"
  }];
};

The Pipecat JavaScript client provides a comprehensive set of methods for managing bot interactions and media handling. These core methods are documented below.

## Session connectivity

### startBot()

`async startBot(startBotParams: APIEndpoint): Promise<TransportConnectionParams | unknown>`

This method hits your server endpoint to start the bot and optionally obtain the connection parameters needed for `connect()` to connect the `Transport`. It returns a Promise that resolves with the response from the server.

<ParamField path="startBotParams" type="APIEndpoint" required>
  The `APIEndpoint` object should have the following shape:

  <Expandable title="APIEndpoint" defaultOpen="true">
    <ParamField path="endpoint" type="string" required>
      The URL of the endpoint to connect to. This should be a valid REST endpoint.
    </ParamField>

    <ParamField path="headers" type="Headers">
      Optional headers to include in the request to the endpoint. This can be used to pass authentication tokens or other necessary headers.
    </ParamField>

    <ParamField path="requestData" type="object">
      Optional request data to include in the request to the endpoint. This can be used to pass additional data to your server-side endpoint. Oftentimes, this is used to pass the initial prompt or other configuration data to initialize the bot.
    </ParamField>

    <ParamField path="timeout" type="number">
      Optional timeout in milliseconds for the request to the endpoint.
    </ParamField>
  </Expandable>
</ParamField>

During the `startBot()` process, the transport state will transition through the states: "authenticating" and "authenticated".

```javascript  theme={null}
try {
  await pcClient.startBot({
    endpoint: "/api/start", // Your server endpoint to start the bot
    requestData: {
      initial_prompt: "You are a pirate captain",
      llm_provider: "openai"
    }
  });
} catch (error) {
  console.error("Error starting the bot:", error);
}
```

### connect()

`async connect(connectParams): Promise<BotReadyData>`

This method initiates the connection process, optionally passing parameters that your transport class requires to establish a connection or an endpoint to your server for obtaining those parameters.

<ParamField path="connectParams" type="TransportConnectionParams">
  An object containing the `TransportConnectionParams` your Transport expects.

  Check your transport class documentation for the expected shape of `TransportConnectionParams`. For example, the DailyTransport expects a `url` and `token`.

  <Warning>
    In 1.2.0 we deprecated support for passing a `ConnectionEndpoint` object directly to `connect()`. Instead, you should use the `startBot()` or `startBotAndConnect()` methods to fetch connection parameters from your server endpoint and then pass those parameters directly to `connect()`.
  </Warning>
</ParamField>

This method can be try / catched to handle errors at startup:

```typescript  theme={null}
try {
  await pcClient.connect({
    webrtcUrl: "http://my-server/api/offer"
  });
} catch (error) {
  console.error("Error connecting to the bot:", error);
}
```

During the connection process, the transport state will transition through the following states: "connecting", "connected", "ready".

<Note>
  Calling `connect()` asynchronously will resolve when the bot and client signal that they are ready. See [messages and events](./messages). If you want to call `connect()` without `await`, you can use the `onBotReady` callback or `BotReady` event to know when you can interact with the bot.
</Note>

<Warning>
  Attempting to call `connect()` when the transport is already in a 'connected' or 'ready' state will throw an error. You should [disconnect](#disconnect) from a session first before attempting to connect again.
</Warning>

### startBotAndConnect()

`async startBotAndConnect(startBotParams: APIEndpoint): Promise<BotReadyData>`

This method combines the functionality of `startBot()` and `connect()`. It first starts the bot by hitting your server endpoint and then connects the transport passing the response from the endpoint to the transport as connection parameters.

```javascript  theme={null}
try {
  await pcClient.startBotAndConnect({
    endpoint: "/api/start", // Your server endpoint to start the bot
    requestData: {
      initial_prompt: "You are a pirate captain",
      llm_provider: "openai"
    }
  });
} catch (error) {
  console.error("Error starting up:", error);
}
```

It's equivalent to: `pcClient.startBot(...).then((resp) => pcClient.connect(resp))`.

### disconnect()

`async disconnect(): Promise<void>`

Disconnects from the active session. The transport state will transition to "disconnecting" and then "disconnected".

It is common practice for bots to exit and cleanup when the client disconnects.

```typescript  theme={null}
await pcClient.disconnect();
```

### disconnectBot()

`disconnectBot(): void`

Triggers the bot to disconnect from the session, leaving the client connected.

```typescript  theme={null}
await pcClient.disconnectBot();
```

## Messages

Custom messaging between the client and the bot. This is useful for sending data to the bot, triggering specific actions, reacting to server events, or querying the server.

For more, see: [messages and events](./messages).

### sendClientMessage()

`sendClientMessage(msgType: string, data?: unknown): void`

Sends a custom message to the bot and does not expect a response. This is useful for sending data to the bot or triggering specific actions.

<ParamField path="msgType" type="string" required>
  A string identifying the message.
</ParamField>

<ParamField path="data" type="unknown">
  Optional data to send with the message. This can be any JSON-serializable object.
</ParamField>

### sendClientRequest()

`async sendClientRequest(msgType: string, data: unknown, timeout?: number): Promise<unknown>`

Sends a custom request to the bot and expects a response. This is useful for querying the server or triggering specific actions that require a response. The method returns a Promise that resolves with the data from response.

<ParamField path="msgType" type="string" required>
  A string identifying the message.
</ParamField>

<ParamField path="data" type="unknown">
  Optional data to send with the message. This can be any JSON-serializable object.
</ParamField>

<ParamField path="timeout" type="number" default="10000">
  Optional timeout in milliseconds for the request. If the request does not receive a response within this time, it will reject with an RTVIMessage of type `'error-response'`.
</ParamField>

## Devices

### initDevices()

`async initDevices(): Promise<void>`

Initializes the media device selection machinery, based on `enableCam`/`enableMic` selections and defaults (i.e. turns on the local cam/mic). This method can be called before `connect()` to test and switch between camera and microphone sources.

```typescript  theme={null}
await pcClient.initDevices();
```

### getAllMics()

`async getAllMics(): Promise<MediaDeviceInfo[]>`

Returns a list of available microphones in the form of [`MediaDeviceInfo[]`](https://developer.mozilla.org/en-US/docs/Web/API/MediaDeviceInfo).

```typescript  theme={null}
mic_device_list = pcClient.getAllMics();
```

### getAllCams()

`async getAllCams(): Promise<MediaDeviceInfo[]>`

Returns a list of available cameras in the form of [`MediaDeviceInfo[]`](https://developer.mozilla.org/en-US/docs/Web/API/MediaDeviceInfo).

```typescript  theme={null}
cam_device_list = pcClient.getAllCams();
```

### getAllSpeakers()

`async getAllSpeakers(): Promise<MediaDeviceInfo[]>`

Returns a list of available speakers in the form of [`MediaDeviceInfo[]`](https://developer.mozilla.org/en-US/docs/Web/API/MediaDeviceInfo).

```typescript  theme={null}
speaker_device_list = pcClient.getAllSpeakers();
```

### selectedMic

`selectedMic: MediaDeviceInfo | {}`

The currently selected microphone, represented as a `MediaDeviceInfo` object. If no microphone is selected, it returns an empty object.

```typescript  theme={null}
current_mic = pcClient.selectedMic;
```

### selectedCam

`selectedCam: MediaDeviceInfo | {}`

The currently selected camera, represented as a `MediaDeviceInfo` object. If no camera is selected, it returns an empty object.

```typescript  theme={null}
current_cam = pcClient.selectedCam;
```

### selectedSpeaker

`selectedSpeaker: MediaDeviceInfo | {}`

The currently selected speaker, represented as a `MediaDeviceInfo` object. If no speaker is selected, it returns an empty object.

```typescript  theme={null}
current_speaker = pcClient.selectedSpeaker;
```

### updateMic()

`updateMic(micId: string): void`

Switches to the microphone identified by the provided `micId`, which should match a `deviceId` in the list returned from [`getAllMics()`](#getAllMics).

<ParamField path="micId" type="string">
  deviceId
</ParamField>

```typescript  theme={null}
pcClient.updateMic(deviceId);
```

### updateCam()

`updateCam(camId: string): void`

Switches to the camera identified by the provided `camId`, which should match a `deviceId` in the list returned from [`getAllCams()`](#getAllCams).

<ParamField path="camId" type="string">
  deviceId
</ParamField>

```typescript  theme={null}
pcClient.updateCam(deviceId);
```

### updateSpeaker()

`updateSpeaker(speakerId: string): void`

Switches to the speaker identified by the provided `speakerId`, which should match a `deviceId` in the list returned from [`getAllSpeakers()`](#getAllSpeakers).

<ParamField path="speakerId" type="string">
  deviceId
</ParamField>

```typescript  theme={null}
pcClient.updateSpeaker(deviceId);
```

### enableMic(enable: boolean)

`enableMic(enable: boolean): void`

Turn on or off (unmute or mute) the client mic input.

<ParamField path="enable" type="boolean" required>
  A boolean indicating whether to enable (`true`) or disable (`false`) the microphone.
</ParamField>

```typescript  theme={null}
pcClient.enableMic(true);
```

### enableCam(enable: boolean)

`enableCam(enable: boolean): void`

Turn on or off the client cam input.

<ParamField path="enable" type="boolean" required>
  A boolean indicating whether to enable (`true`) or disable (`false`) the camera.
</ParamField>

```typescript  theme={null}
pcClient.enableCam(true);
```

### enableScreenShare(enable: boolean)

`enableScreenShare(enable: boolean): void`

Start a screen share from the client's device.

<ParamField path="enable" type="boolean" required>
  A boolean indicating whether to enable (`true`) or disable (`false`) screen sharing.
</ParamField>

```typescript  theme={null}
pcClient.enableScreenShare(true);
```

### isMicEnabled

`isMicEnabled: boolean`

An accessor to determine if the client's microphone is enabled.

```typescript  theme={null}
mic_enabled = pcClient.isMicEnabled;
```

### isCamEnabled

`isCamEnabled: boolean`

An accessor to determine if the client's camera is enabled.

```typescript  theme={null}
cam_enabled = pcClient.isCamEnabled;
```

### isSharingScreen

An accessor to determine if the client is sharing their screen.

```typescript  theme={null}
screen_sharing = pcClient.isSharingScreen;
```

## Tracks (audio and video)

### tracks()

`tracks(): Tracks`

Returns a `Tracks` object with available `MediaStreamTrack` objects for both the client and the bot.

```typescript  theme={null}
live_tracks_list = pcClient.tracks()
```

**Tracks Type**

```typescript  theme={null}
{
  local: {
    audio?: MediaStreamTrack;
    video?: MediaStreamTrack;
  },
  bot?: {
    audio?: MediaStreamTrack;
    video?: MediaStreamTrack;
  }
}
```

## Advanced LLM Interactions

### sendText()

`async sendText(content: string, options?: SendTextOptions = {}): void`

A method to append text to the user's context. This is useful for providing text input as an alternative to audio input for the user.

<ParamField path="content" type="string" required>
  The text content to send to the bot.
</ParamField>

<ParamField path="options" type="SendTextOptions">
  An optional set of options for how the bot should handle the text input.

  <Expandable title="SendTextOptions" defaultOpen="true">
    <ParamField path="run_immediately" type="boolean" default="true">
      Whether to immediately run the bot with the updated context. If `false`, the context will be updated but the bot will not be run until the next message or action that triggers the bot to run (like the user speaking).
    </ParamField>

    <ParamField path="audio_response" type="boolean" default="true">
      Whether the bot should respond with audio. If `true`, the bot's response will be processed by TTS and be spoken. If `false`, the bot will bypass the TTS and respond with text only.
    </ParamField>
  </Expandable>
</ParamField>

### appendToContext()

`async appendToContext(context: LLMContextMessage): boolean`

<Warning>
  In 0.0.4, we deprecated `appendToContext` in favor of `send-text` to close a security vulnerability as well as lay the groundwork for proper support of passing other types of context, like images and files.
</Warning>

A method to append data to the bot's context. This is useful for providing additional information or context to the bot during the conversation.

<ParamField path="context" type="LLMContextMessage" required>
  The context to append. This should be an object with the following shape:

  <Expandable title="LLMContextMessage" defaultOpen="true">
    <ParamField path="role" type="string" required>
      The role to append the context to. Currently only "user" or "assistant" are supported.
    </ParamField>

    <ParamField path="content" type="unknown" required>
      The content to append to the context. This can be any JSON-serializable object.
    </ParamField>

    <ParamField path="run_immediately" type="boolean" default="false">
      Whether to immediately run the bot with the updated context. If `false`, the context will be updated but the bot will not be run until the next message or action that triggers the bot to run (like the user speaking).
    </ParamField>
  </Expandable>
</ParamField>

### registerFunctionCallHandler()

`registerFunctionCallHandler(functionName: string, callback: FunctionCallCallback): void`

Registers a function call handler that will be called when the bot requests a function call. This is useful for when the server-side function handler needs information from the client to execute the function call or when the client needs to perform some action based on the running of function call.

<ParamField path="functionName" type="string" required>
  The name of the function to handle. This should match the function name in the bot's context.
</ParamField>

<ParamField path="callback" type="FunctionCallCallback" required>
  `type FunctionCallCallback = (fn: FunctionCallParams) => Promise<LLMFunctionCallResult | void>`

  The callback function to call when the bot sends a function call request. This function should accept the following parameters:

  <Expandable title="FunctionCallParams" defaultOpen="true">
    <ParamField path="functionName" type="string" required>
      The name of the function being called. It should always match the name you registered the handler under.
    </ParamField>

    <ParamField path="arguments" type="Record<string, unknown>" required>
      The arguments passed to the function call. This is a key-value object where the keys are the argument names and the values are the argument values.
    </ParamField>
  </Expandable>

  The callback should return a Promise that resolves with the result of the function call or void if no result is needed. If returning a result, it should be a `string` or `Record<string, unknown>`.
</ParamField>

## Other

### transport

`transport: Transport`

A safe accessor for the transport instance used by the client. This is useful for accessing transport-specific methods or properties that are not exposed directly on the client.

```typescript  theme={null}
const transport = pcClient.transport as DailyTransport;
transport.getSessionInfo();
```

### setLogLevel()

`setLogLevel(level: LogLevel): void`

Sets the log level for the client. This is useful for debugging and controlling the verbosity of logs. The log levels are defined in the `LogLevel` enum:

```typescript  theme={null}
export enum LogLevel {
  NONE = 0,
  ERROR = 1,
  WARN = 2,
  INFO = 3,
  DEBUG = 4,
}
```

By default, the log level is set to `LogLevel.DEBUG`.

```typescript  theme={null}
pcClient.setLogLevel(LogLevel.INFO);
```

## Transport Compatibility

<TransportTable rows={methods()} />

---

# Source: https://docs.pipecat.ai/server/services/community-integrations.md

> ## Documentation Index
> Fetch the complete documentation index at: https://docs.pipecat.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Community Integrations

> Community-maintained service integrations for Pipecat

Community Integrations are service integrations built and maintained by developers in the Pipecat community. These are not officially supported by the Pipecat team but are listed here to help you discover what's available.

Want to add your integration? See our [Community Integrations Guide](https://github.com/pipecat-ai/pipecat/blob/main/COMMUNITY_INTEGRATIONS.md).

***

## Knowledge Retrieval

Semantic retrieval services enable context-aware search and retrieval of relevant information.

| Service                          | Repository                                                                         | Maintainer(s)                      |
| -------------------------------- | ---------------------------------------------------------------------------------- | ---------------------------------- |
| [Moss](https://www.usemoss.dev/) | [https://github.com/usemoss/pipecat-moss](https://github.com/usemoss/pipecat-moss) | [Moss](https://github.com/usemoss) |

## Large Language Models

LLMs receive text or audio based input and output a streaming text response.

| Service                          | Repository                                                                                                             | Maintainer(s)                           |
| -------------------------------- | ---------------------------------------------------------------------------------------------------------------------- | --------------------------------------- |
| [Anannas AI](https://anannas.ai) | [https://github.com/Anannas-AI/anannas-pipecat-integration](https://github.com/Anannas-AI/anannas-pipecat-integration) | [Haleshot](https://github.com/Haleshot) |

## Observability

Observability services enable telemetry and metrics data to be passed to a Open Telemetry backend or another service.

| Service                                                    | Repository                                                                                                                                                | Maintainer(s)                                                             |
| ---------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------- |
| [OpenInference](https://arize-ai.github.io/openinference/) | [openinference-instrumentation-pipecat](https://github.com/Arize-ai/openinference/tree/main/python/instrumentation/openinference-instrumentation-pipecat) | [Arize-ai](https://github.com/Arize-ai/openinference/graphs/contributors) |
| [Finchvox](https://finchvox.dev/)                          | [https://github.com/finchvox/finchvox](https://github.com/finchvox/finchvox)                                                                              | [Finchvox](https://github.com/finchvox)                                   |

## Telephony Serializers

Serializers convert between frames and media streams, enabling real-time communication over a websocket.

| Service                          | Repository                                                                               | Maintainer(s)                         |
| -------------------------------- | ---------------------------------------------------------------------------------------- | ------------------------------------- |
| [AwaazAI](https://www.awaaz.ai/) | [https://github.com/awaazde/pipecat-awaazai](https://github.com/awaazde/pipecat-awaazai) | [AwaazAI](https://github.com/awaazde) |

## Text-to-Speech

Text-to-Speech services receive text input and output audio streams or chunks.

| Service                                                          | Repository                                                                                             | Maintainer(s)                                   |
| ---------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------ | ----------------------------------------------- |
| [Murf AI](https://murf.ai/api)                                   | [https://github.com/murf-ai/pipecat-murf-tts](https://github.com/murf-ai/pipecat-murf-tts)             | [murf-ai](https://github.com/murf-ai)           |
| [Pipecat TTS Cache](https://pypi.org/project/pipecat-tts-cache/) | [https://github.com/omChauhanDev/pipecat-tts-cache](https://github.com/omChauhanDev/pipecat-tts-cache) | [omChauhanDev](https://github.com/omChauhanDev) |
| [Respeecher](https://www.respeecher.com/real-time-tts-api)       | [https://github.com/respeecher/pipecat-respeecher](https://github.com/respeecher/pipecat-respeecher)   | [respeecher](https://github.com/respeecher)     |
| [Typecast](https://typecast.ai/)                                 | [https://github.com/neosapience/pipecat-typecast](https://github.com/neosapience/pipecat-typecast)     | [neosapience](https://github.com/neosapience)   |
| [Voice.ai](https://voice.ai/)                                    | [https://github.com/voice-ai/voice-ai-pipecat-tts](https://github.com/voice-ai/voice-ai-pipecat-tts)   | [voice-ai](https://github.com/voice-ai)         |

## Video

Video services enable you to build an avatar where audio and video are synchronized.

| Service                                           | Repository                                                                       | Maintainer(s)                         |
| ------------------------------------------------- | -------------------------------------------------------------------------------- | ------------------------------------- |
| [Beyond Presence](https://www.beyondpresence.ai/) | [https://github.com/bey-dev/pipecat-bey](https://github.com/bey-dev/pipecat-bey) | [bey-dev](https://github.com/bey-dev) |

## Image Generation

Image generation services receive text inputs and output images.

| Service                         | Repository | Maintainer(s) |
| ------------------------------- | ---------- | ------------- |
| *No community integrations yet* |            |               |

## Speech-to-Text

Speech-to-Text services receive and audio input and output transcriptions.

| Service                         | Repository | Maintainer(s) |
| ------------------------------- | ---------- | ------------- |
| *No community integrations yet* |            |               |

## Vision

Vision services receive a streaming video input and output text describing the video input.

| Service                         | Repository | Maintainer(s) |
| ------------------------------- | ---------- | ------------- |
| *No community integrations yet* |            |               |

---

# Source: https://docs.pipecat.ai/client/react/components.md

> ## Documentation Index
> Fetch the complete documentation index at: https://docs.pipecat.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Components

> Ready-to-use React components for Pipecat applications

The Pipecat React SDK provides several components for handling audio, video, and visualization in your application.

## PipecatClientProvider

The root component for providing Pipecat client context to your application.

```jsx  theme={null}
<PipecatClientProvider client={pcClient}>
  {/* Child components */}
</PipecatClientProvider>
```

**Props**

<ParamField path="client" type="PipecatClient" required>
  A singleton instance of `PipecatClient`
</ParamField>

## PipecatClientAudio

Creates a new `<audio>` element that mounts the bot's audio track.

```jsx  theme={null}
<PipecatClientAudio />
```

**Props**

No props required

## PipecatClientVideo

Creates a new `<video>` element that renders either the bot or local participant's video track.

```jsx  theme={null}
<PipecatClientVideo
  participant="local"
  fit="cover"
  mirror
  onResize={({ aspectRatio, height, width }) => {
    console.log("Video dimensions changed:", { aspectRatio, height, width });
  }}
/>
```

**Props**

<ParamField path="participant" type="('local' | 'bot')" required>
  Defines which participant's video track is rendered
</ParamField>

<ParamField path="fit" type="('contain' | 'cover')">
  Defines whether the video should be fully contained or cover the box. Default: 'contain'
</ParamField>

<ParamField path="mirror" type="boolean">
  Forces the video to be mirrored, if set
</ParamField>

<ParamField path="onResize(dimensions: object)" type="function">
  Triggered whenever the video's rendered width or height changes
</ParamField>

## PipecatClientCamToggle

A headless component to read and set the local participant's camera state.

```jsx  theme={null}
<PipecatClientCamToggle
  onCamEnabledChanged={(enabled) => console.log("Camera enabled:", enabled)}
  disabled={false}
>
  {({ disabled, isCamEnabled, onClick }) => (
    <button disabled={disabled} onClick={onClick}>
      {isCamEnabled ? "Disable Camera" : "Enable Camera"}
    </button>
  )}
</PipecatClientCamToggle>
```

**Props**

<ParamField path="onCamEnabledChanged(enabled: boolean)" type="function">
  Triggered whenever the local participant's camera state changes
</ParamField>

<ParamField path="disabled" type="boolean">
  If true, the component will not allow toggling the camera state. Default: false
</ParamField>

<ParamField path="children" type="function">
  A render prop that provides state and handlers to the children
</ParamField>

## PipecatClientMicToggle

A headless component to read and set the local participant's microphone state.

```jsx  theme={null}
<PipecatClientMicToggle
  onMicEnabledChanged={(enabled) => console.log("Microphone enabled:", enabled)}
  disabled={false}
>
  {({ disabled, isMicEnabled, onClick }) => (
    <button disabled={disabled} onClick={onClick}>
      {isMicEnabled ? "Disable Microphone" : "Enable Microphone"}
    </button>
  )}
</PipecatClientMicToggle>
```

**Props**

<ParamField path="onMicEnabledChanged(enabled: boolean)" type="function">
  Triggered whenever the local participant's microphone state changes
</ParamField>

<ParamField path="disabled" type="boolean">
  If true, the component will not allow toggling the microphone state. Default: false
</ParamField>

<ParamField path="children" type="function">
  A render prop that provides state and handlers to the children
</ParamField>

## VoiceVisualizer

Renders a visual representation of audio input levels on a `<canvas>` element.

```jsx  theme={null}
<VoiceVisualizer
  participantType="local"
  backgroundColor="white"
  barColor="black"
  barGap={1}
  barWidth={4}
  barMaxHeight={24}
/>
```

**Props**

<ParamField path="participantType" type="string" required>
  The participant type to visualize audio for
</ParamField>

<ParamField path="backgroundColor" type="string">
  The background color of the canvas. Default: 'transparent'
</ParamField>

<ParamField path="barColor" type="string">
  The color of the audio level bars. Default: 'black'
</ParamField>

<ParamField path="barCount" type="number">
  The number of bars to display. Default: 5
</ParamField>

<ParamField path="barGap" type="number">
  The gap between bars in pixels. Default: 12
</ParamField>

<ParamField path="barWidth" type="number">
  The width of each bar in pixels. Default: 30
</ParamField>

<ParamField path="barMaxHeight" type="number">
  The maximum height at full volume of each bar in pixels. Default: 120
</ParamField>

---

# Source: https://docs.pipecat.ai/guides/learn/context-management.md

> ## Documentation Index
> Fetch the complete documentation index at: https://docs.pipecat.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Context Management

> A guide to working with Pipecat's Context and Context Aggregators

## What is Context in Pipecat?

In Pipecat, **context** refers to the conversation history that the LLM uses to generate responses. The context consists of a list of alternating user/assistant messages that represents the collective history of the entire conversation.

```python  theme={null}
# Example context structure
messages = [
    {"role": "system", "content": "You are a helpful assistant."},
    {"role": "user", "content": "Hello!"},
    {"role": "assistant", "content": "Hi there! How can I help you?"},
    # Context aggregators automatically add new messages here
]
```

Since Pipecat is a real-time voice AI framework, context management happens automatically as the conversation flows, but you can also control it manually when needed.

## How Context Updates During Conversations

Context updates happen automatically as frames flow through your pipeline:

**User Messages:**

1. User speaks → `InputAudioRawFrame` → STT Service → `TranscriptionFrame`
2. `context_aggregator.user()` receives `TranscriptionFrame` and adds user message to context

**Assistant Messages:**

1. LLM generates response → `LLMTextFrame` → TTS Service → `TTSTextFrame`
2. `context_aggregator.assistant()` receives `TTSTextFrame` and adds assistant message to context

**Frame types that update context:**

* **`TranscriptionFrame`**: Contains user speech converted to text by STT service
* **`LLMTextFrame`**: Contains LLM-generated responses
* **`TTSTextFrame`**: Contains bot responses converted to text by TTS service (represents what was actually spoken)

<Note>
  The TTS service processes `LLMTextFrame`s but outputs `TTSTextFrame`s, which
  represent the actual spoken text returned by the TTS provider. This ensures
  context matches what users actually hear.
</Note>

## Setting Up Context Management

Pipecat includes a context aggregator that creates and manages context for both user and assistant messages:

### 1. Create the Context and Context Aggregator

```python  theme={null}
# Create LLM service
llm = OpenAILLMService(api_key=os.getenv("OPENAI_API_KEY"))

# Create context with initial messages
messages = [
    {"role": "system", "content": "You are a helpful voice assistant."}
]

# Create context (messages only)
context = LLMContext(messages)

# Create context aggregator instance
user_aggregator, assistant_aggregator = LLMContextAggregatorPair(context)
```

<Note>
  The context aggregator also supports configuring [user turn
  strategies](/server/utilities/turn-management/user-turn-strategies) and [user
  mute strategies](/server/utilities/turn-management/user-mute-strategies) via
  `LLMUserAggregatorParams`.
</Note>

**About LLMContext:**

`LLMContext` is Pipecat's universal context container that stores conversation messages, tool definitions, and tool choice settings. It uses an OpenAI-compatible format that works across all LLM services through automatic adapter translation.

Key properties:

* **`messages`**: List of conversation messages (system, user, assistant, tool)
* **`tools`**: Optional `ToolsSchema` defining available functions
* **`tool_choice`**: Optional strategy for tool selection

### 2. Context with Function Calling

Context can also include tools (function definitions) that the LLM can call during conversations:

```python  theme={null}
from pipecat.adapters.schemas.function_schema import FunctionSchema
from pipecat.adapters.schemas.tools_schema import ToolsSchema

# Define available functions
weather_function = FunctionSchema(
    name="get_current_weather",
    description="Get the current weather",
    properties={
        "location": {
            "type": "string",
            "description": "The city and state, e.g. San Francisco, CA",
        },
        "format": {
            "type": "string",
            "enum": ["celsius", "fahrenheit"],
            "description": "The temperature unit to use.",
        },
    },
    required=["location", "format"],
)

# Create tools schema
tools = ToolsSchema(standard_tools=[weather_function])

# Create context with both messages and tools
context = LLMContext(messages, tools)
user_aggregator, assistant_aggregator = LLMContextAggregatorPair(context)
```

Function call results are also automatically stored in the context, maintaining a complete conversation history including tool interactions.

<Note>
  We'll cover function calling in detail in an upcoming section. The context
  aggregator handles function call storage automatically.
</Note>

### 3. Add Context Aggregators to Your Pipeline

```python  theme={null}
pipeline = Pipeline([
    transport.input(),
    stt,
    context_aggregator.user(),      # User context aggregator
    llm,
    tts,
    transport.output(),
    context_aggregator.assistant(), # Assistant context aggregator
])
```

## Context Aggregator Placement

The placement of context aggregator instances in your pipeline is **crucial** for proper operation:

### User Context Aggregator

Place the user context aggregator **downstream from the STT service**. Since the user's speech results in `TranscriptionFrame` objects pushed by the STT service, the user aggregator needs to be positioned to collect these frames.

### Assistant Context Aggregator

Place the assistant context aggregator **after `transport.output()`**. This positioning is important because:

* The TTS service outputs `TTSTextFrame`s in addition to audio
* The assistant aggregator must be downstream to collect those frames
* It ensures context updates happen word-by-word for specific services (e.g. Cartesia, ElevenLabs, and Rime)
* Your context stays updated at the word level in case an interruption occurs

<Tip>
  Always place the assistant context aggregator **after** `transport.output()`
  to ensure proper word-level context updates during interruptions.
</Tip>

## Manual Context Control

You can programmatically add new messages to the context by pushing or queueing specific frames:

### Adding Messages

* **`LLMMessagesAppendFrame`**: Appends a new message to the existing context
* **`LLMMessagesUpdateFrame`**: Completely replaces the existing context with new messages

```python  theme={null}
# Add a new system message to context
new_message = {"role": "system", "content": "You are now in expert mode."}
await task.queue_frames([
    LLMMessagesAppendFrame([new_message], run_llm=True), # Optionally trigger bot response, too
])
```

### Retrieving Current Context

The context aggregator provides a `context` property for getting the current context:

```python  theme={null}
context = context_aggregator.user().context
```

## Triggering Bot Responses

You may want to manually trigger the bot to speak in two scenarios:

1. **Starting a pipeline** where the bot should speak first
2. **After editing the context** using `LLMMessagesAppendFrame` or `LLMMessagesUpdateFrame`

```python  theme={null}
# Example: Bot speaks first when pipeline starts
@transport.event_handler("on_client_connected")
async def on_client_connected(transport, client):
    # Trigger a response
    await task.queue_frames([LLMRunFrame()])
```

```python  theme={null}
# Example: Bot speaks after context is edited
new_message = {"role": "user", "content": "Tell me a fun fact."}
await task.queue_frames([
    LLMMessagesAppendFrame([new_message], run_llm=True), # Trigger bot response
])
```

This gives you fine-grained control over when and how the bot responds during the conversation flow.

## Key Takeaways

* **Context is conversation history** - automatically maintained as users and bots exchange messages
* **Frame types matter** - `TranscriptionFrame` for users, `TTSTextFrame` for assistants
* **Placement matters** - user aggregator after STT, assistant aggregator after transport output
* **Tools are included** - function definitions and results are stored in context
* **Manual control available** - use frames to append messages or trigger responses when needed
* **Word-level precision** - proper placement ensures context accuracy during interruptions

## What's Next

Now that you understand context management, let's explore how to configure the LLM services that process this context to generate intelligent responses.

<Card title="LLM Inference" icon="arrow-right" href="/guides/learn/llm">
  Learn how to configure language models in your voice AI pipeline
</Card>

---

# Source: https://docs.pipecat.ai/guides/fundamentals/custom-frame-processor.md

> ## Documentation Index
> Fetch the complete documentation index at: https://docs.pipecat.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Custom FrameProcessor

> Learn how to write your own custom FrameProcessor

Pipecat's architecture is made up of a Pipeline, FrameProcessors, and Frames. See the [Core Concepts](/guides/learn/pipeline) for a full review. From that architecture, recall that FrameProcessors are the workers in the pipeline that receive frames and complete actions based on the frames received.

Pipecat comes with many FrameProcessors built in. These consist of services, like `OpenAILLMService` or `CartesiaTTSService`, utilities, like `LLMTextProcessor`, and other things. Largely, you can build most of your application with these built-in FrameProcessors, but commonly, your application code may require custom frame processing logic. For example, you may want to perform an action as a result of a frame that's pushed in the pipeline.

## Example: MetricsFrame logger

This custom FrameProcessor format and logs MetricsFrames:

```python  theme={null}
class MetricsFrameLogger(FrameProcessor):
    """MetricsFrameLogger formats and logs all MetericsFrames"""

    def __init__(self):
        super().__init__()

    async def process_frame(self, frame: Frame, direction: FrameDirection):
        await super().process_frame(frame, direction)

        if isinstance(frame, MetricsFrame):
            logger.info(f"{frame.name}\n    {format_metrics(frame.data)}")
            await self.push_frame(frame, direction)

        # ALWAYS push all frames
        else:
            # SUPER IMPORTANT: always push every frame!
            await self.push_frame(frame, direction)
```

This frame processor looks for `MetricsFrames`. When it sees one, it formats the data and logs it.

It uses this `format_metrics` function:

```python  theme={null}
def format_metrics(metrics, indent=0):
    lines = []
    tab = "\t" * indent

    for metric in metrics:
        lines.append(tab + type(metric).__name__)
        for field, value in vars(metric).items():
            if hasattr(value, "__dict__") and not isinstance(
                value, (str, int, float, bool, type(None))
            ):
                lines.append(f"{tab}\t{field}={type(value).__name__}")
                for k, v in vars(value).items():
                    lines.append(f"{tab}\t\t{k}={repr(v)}")
            else:
                lines.append(f"{tab}\t{field}={repr(value)}")

    return "\n".join(lines)
```

<Note>
  See this [working
  example](https://github.com/pipecat-ai/pipecat/blob/main/examples/foundational/08-custom-frame-processor.py)
  using the `MetricsFrameLogger` FrameProcessor
</Note>

## Add to a Pipeline

```python  theme={null}
# Create and initialize the custom FrameProcessor
metrics_frame_processor = MetricsFrameLogger()

pipeline = Pipeline(
    [
        transport.input(),
        stt,
        context_aggregator.user(),
        llm,
        tts,
        transport.output(),
        context_aggregator.assistant(),
        metrics_frame_processor,  # Our custom FrameProcessor that pretty prints metrics frames
    ]
)
```

With this positioning, the `MetricsFrameLogger` FrameProcessor will receive every MetericsFrame in the pipeline.

## Key Requirements

FrameProcessors must inherit from the base `FrameProcessor` class. This ensures that your custom FrameProcessor will correctly handle frames like `StartFrame`, `EndFrame`, `StartInterruptionFrame` without having to write custom logic for those frames. This inheritance also provides it with the ability to `process_frame()` and `push_frame()`:

* **`process_frame()`** is what allows the FrameProcessor to receive frames and add custom conditional logic based on the frames that are received.
* **`push_frame()`** allows the FrameProcessor to push frames to the pipeline. Normally, frames are pushed DOWNSTREAM, but based on which processors need the output, you can also push UPSTREAM or in both directions.

### Essential Implementation Details

To ensure proper base class inheritance, it's critical to include:

1. **`super().__init__()`** in your `__init__` method
2. **`await super().process_frame(frame, direction)`** in your `process_frame()` method

```python  theme={null}
class MyCustomProcessor(FrameProcessor):
    def __init__(self, **kwargs):
        super().__init__(**kwargs)  # ✅ Required
        # Your initialization code here

    async def process_frame(self, frame: Frame, direction: FrameDirection):
        await super().process_frame(frame, direction)  # ✅ Required

        # Your custom frame processing logic here
        if isinstance(frame, SomeSpecificFrame):
            # Handle the frame
            pass

        await self.push_frame(frame, direction)  # ✅ Required - pass frame through
```

## Critical Responsibility: Frame Forwarding

FrameProcessors receive **all** frames that are pushed through the pipeline. This gives them a lot of power, but also a great responsibility. Critically, they must push all frames through the pipeline; if they don't, they block frames from moving through the Pipeline, which will cause issues in how your application functions.

As well as formatting and logging MetricsFrames, `MetricsFrameLogger` also has an `await self.push_frame(frame, direction)` which pushes the frame through to the next processor in the pipeline.

## Frame Direction

When pushing frames, you can specify the direction:

```python  theme={null}
# Push downstream (default)
await self.push_frame(frame, FrameDirection.DOWNSTREAM)

# Push upstream
await self.push_frame(frame, FrameDirection.UPSTREAM)
```

Most custom FrameProcessors will push frames downstream, but upstream can be useful for sending control frames or error notifications back up the pipeline.

## Best Practices

1. **Always call the parent methods**: Use `super().__init__()` and `await super().process_frame()`
2. **Forward all frames**: Make sure every frame is pushed through with `await self.push_frame(frame, direction)`
3. **Handle frames conditionally**: Use `isinstance()` checks to handle specific frame types
4. **Use proper error handling**: Wrap risky operations in try/catch blocks
5. **Position carefully in pipeline**: Consider where in the pipeline your processor needs to be to receive the right frames

With these patterns, you can create powerful custom FrameProcessors that extend Pipecat's capabilities for your specific use case.

---

# Source: https://docs.pipecat.ai/deployment/pipecat-cloud/guides/telephony/daily-dial-in.md

> ## Documentation Index
> Fetch the complete documentation index at: https://docs.pipecat.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Daily PSTN Dial-in

> Using Daily's Transport to receive calls from external SIP Addresses and PSTN numbers

When someone calls your Daily phone number, the call is placed on hold and triggers a webhook to initialize your Pipecat bot.

## Setup Options

### Option 1: Automatic Webhook (Recommended)

Let Pipecat Cloud handle the webhook automatically.

**Using the Dashboard:**

1. Go to **Settings > Telephony** in Pipecat Cloud
2. Select your phone number
3. Choose your deployed agent
4. Save

**Using the REST API:**

```bash  theme={null}
curl --location 'https://api.daily.co/v1' \
--header 'Authorization: Bearer YOUR_DAILY_API_KEY' \
--header 'Content-Type: application/json' \
--data '{
    "properties": {
        "pinless_dialin": [{
            "phone_number": "YOUR_DAILY_NUMBER",
            "room_creation_api": "https://api.pipecat.daily.co/v1/public/webhooks/{org_id}/{agent_name}/dialin"
        }]
    }
}'
```

### Option 2: Custom Webhook Server

Host your own webhook server for custom logic (e.g., CRM integration, advanced routing, caller-based customization).

<CardGroup cols={2}>
  <Card title="FastAPI Webhook Server" icon="server" href="https://github.com/pipecat-ai/pipecat-examples/tree/main/deployment/pipecat-cloud-daily-pstn-server/fastapi-webhook-server">
    Python webhook server example
  </Card>

  <Card title="Next.js Webhook Server" icon="js" href="https://github.com/pipecat-ai/pipecat-examples/tree/main/deployment/pipecat-cloud-daily-pstn-server/nextjs-webhook-server">
    JavaScript webhook server example
  </Card>
</CardGroup>

**Webhook payload from Daily:**

```json  theme={null}
{
  "To": "+15559876543",
  "From": "+15551234567",
  "callId": "uuid",
  "callDomain": "uuid"
}
```

**Forward to Pipecat Cloud:**

```bash  theme={null}
curl --request POST \
--url https://api.pipecat.daily.co/v1/public/{service}/start \
--header 'Authorization: Bearer $API_KEY' \
--header 'Content-Type: application/json' \
--data '{
    "createDailyRoom": true,
    "dailyRoomProperties": {
        "sip": {
            "display_name": "+15551234567",
            "sip_mode": "dial-in",
            "num_endpoints": 1
        }
    },
    "body": {
        "dialin_settings": {
            "from": "+15551234567",
            "to": "+15559876543",
            "call_id": "uuid",
            "call_domain": "uuid"
        }
    }
}'
```

## Bot Configuration

Configure your bot to handle dial-in calls:

```python  theme={null}
from pipecat.runner.types import DailyDialinRequest, RunnerArguments
from pipecat.transports.daily import DailyTransport, DailyParams, DailyDialinSettings

async def bot(runner_args: RunnerArguments):
    # Parse dial-in request
    request = DailyDialinRequest.model_validate(runner_args.body)

    # Configure dial-in settings
    dialin_settings = DailyDialinSettings(
        call_id=request.dialin_settings.call_id,
        call_domain=request.dialin_settings.call_domain
    )

    # Create transport
    transport = DailyTransport(
        runner_args.room_url,
        runner_args.token,
        "Voice Bot",
        DailyParams(
            api_key=request.daily_api_key,
            api_url=request.daily_api_url,
            dialin_settings=dialin_settings,
            audio_in_enabled=True,
            audio_out_enabled=True,
        )
    )

    # Start speaking when caller joins
    @transport.event_handler("on_first_participant_joined")
    async def on_first_participant_joined(transport, participant):
        await task.queue_frames([LLMRunFrame()])

    # Your bot pipeline setup...
```

**Personalize with caller info:**

```python  theme={null}
caller_phone = request.dialin_settings.From
customer = await get_customer_by_phone(caller_phone)
# Use customer data in your system prompt
```

## Complete Example

<Card title="Daily PSTN Dial-in Bot" icon="phone" href="https://github.com/pipecat-ai/pipecat-examples/tree/main/phone-chatbot/daily-pstn-dial-in">
  Full implementation with STT, LLM, TTS, and caller personalization
</Card>

---

# Source: https://docs.pipecat.ai/deployment/pipecat-cloud/guides/telephony/daily-dial-out.md

> ## Documentation Index
> Fetch the complete documentation index at: https://docs.pipecat.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Daily PSTN Dial-out

> Using Daily’s Transport to make calls to external SIP Addresses and PSTN numbers

To initiate outbound calls, you first need to configure your Daily rooms to allow dial-out by setting `enable_dialout` in `dailyRoomProperties`. Then, include the target phone number(s) in the request `body` of `{service}/start`.

In this example, we use the `dialout_settings` array to pass one or more phone numbers for the bot to dial. Each object in the array must include a `phoneNumber` and can optionally specify a `callerId` to control the outbound caller ID.

<Info>
  If no `callerId` is provided, a random phone number will be used to make the
  call. To specify the caller ID, use the [phone number's
  `id`](/guides/telephony/daily-phone-numbers#list-all-purchased-phone-numbers)
  as the `callerId`.
</Info>

```bash  theme={null}
curl --request POST \
--url https://api.pipecat.daily.co/v1/public/{service}/start \
--header 'Authorization: Bearer $API_KEY' \
--header 'Content-Type: application/json' \
--data-raw '{
    'createDailyRoom': true,
    'dailyRoomProperties': {
        'enable_dialout': true,
        'exp': 1742353929
    },
    'body': {
        'dialout_settings': \
        [{'phoneNumber': '+1TARGET', 'callerId': 'UUID_OF_PURCHASED_NUM'}]
    }
}
```

<Info>
  The API\_KEY here is Pipcat Cloud's [public API
  key](../../fundamentals/accounts-and-organizations#api-keys), typically begins
  with "pk\_..."
</Info>

The bot should begin dialing out as soon as the call state transitions to the `joined` state.

Unlike dial-in scenarios where the bot typically speaks first, dial-out calls may require the bot to wait for the remote user to speak. In these cases, you can start listening for the remote user when the `on_dialout_answered` event fires.

This is also a good point to initiate voicemail detection—for example, by analyzing the remote user's audio to determine if the call has reached a voicemail.

```python  theme={null}
@transport.event_handler("on_call_state_updated")
async def on_call_state_updated(transport, state):
    if state == "joined" and dialout_settings:
        await start_dialout(transport, dialout_settings)

@transport.event_handler("on_dialout_answered")
    async def on_dialout_answered(transport, data):
        # the bot will wait for the user to speak before responding
        await rtvi.set_bot_ready()
```

### Examples of `dialout_settings`

Here are various configurations you can use in the `dialout_settings` array:

<CodeGroup>
  ```json Twilio SIP Address theme={null}
  "dialout_settings": [
    { 
      "sipUri": "sip:+1xxxx@domain.sip.twilio.com" 
    }
  ]
  ```

  ```json Phone Number with CallerID theme={null}

  "dialout_settings": [
    {
      "phoneNumber": "+1xxx",
      "callerId": "uuid"
    }
  ]

  ```

  ```json Multiple Phone Numbers theme={null}
  "dailout_settings": [
    {
      "phoneNumber": "+1xxx",
      "callerId": "uuid-1"
    },
    {
      "sipUri": "sip:+1xxxx@domain.sip.twilio.com"
    },
    {
      "phoneNumber": "+1xxx",
      "callerId": "uuid-1"
    }
  ]
  ```
</CodeGroup>

## Next Steps

After configuring your dial-out functionality, you can implement a Pipecat bot to handle the call interactions:

<Card title="Dial-in/Dial-out Bot Example" icon="phone" href="https://github.com/pipecat-ai/pipecat-examples/tree/main/phone-chatbot/daily-pstn-dial-out">
  Complete example of a Pipecat bot implementation that handles both incoming
  (dial-in) and outgoing (dial-out) calls with voicemail detection
</Card>

---

# Source: https://docs.pipecat.ai/guides/telephony/daily-phone-numbers.md

> ## Documentation Index
> Fetch the complete documentation index at: https://docs.pipecat.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Daily Phone Numbers

> Purchase and manage phone numbers for Daily's PSTN services

Use Daily's [REST API](https://docs.daily.co/reference/rest-api/phone-numbers) to find, buy, list, and release phone numbers programmatically.

<Tip>
  You can also purchase phone numbers through the Pipecat Cloud Dashboard by
  going to `Settings` > `Telephony` and following the UI.
</Tip>

### Search for available phone numbers

We offer several [search filters](https://docs.daily.co/reference/rest-api/phone-numbers/list-available-numbers), such as, `region`, `city`, `areacode`, etc that can help narrow down the search space for a phone number. The response contains phone numbers that meet the chosen criteria. For simplicty, in the below examples, we will use `+19499870006` as an example `phone_number`.

<CodeGroup>
  ```bash Request theme={null}
  curl --request GET \
    --url 'https://api.daily.co/v1/list-available-numbers?region=CA' \
    --header 'Authorization: Bearer $DAILY_API_KEY'
  ```

  ```js Success theme={null}
  {
      "total_count": 1,
      "data": [{
          "number": "+19499800001",
          "region": "CA"
      }]
  }
  ```
</CodeGroup>

### Buy a phone number

To purchase the chosen number, make a POST request to the [buy-phone-number](https://docs.daily.co/reference/rest-api/phone-numbers/buy-phone-number) and pass the chosen phone number in the data field. The response returns the phone number and a unique id. The unique id is needed when releasing the chosen phone number, for example.

<CodeGroup>
  ```bash Request theme={null}
  curl --request POST \
    --url 'https://api.daily.co/v1/buy-phone-number' \
    --header 'Authorization: Bearer $DAILY_API_KEY' \
    --header 'Content-Type: application/json' \
    --data '{
          "number": "+19499800001"
  }'
  ```

  ```js Success theme={null}
  {
      "id": "5ccc01ca-f448-4f17-b7b4-cf354b7b8256",
      "number": "+19499800001"
  }
  ```
</CodeGroup>

<Info>
  Use the id to select the desired phone number for dial-out. Pass it as
  `callerId` in the `start_dialout` API or include it in the `dialout_settings`
  JSON.
</Info>

### Release a phone number

To release a phone number associated with your account. Successfully deleting the number takes the phone number out of operation.

<Warning>
  For compliance reasons, a phone number cannot be released until 14 days after
  purchase. A successful release operation cannot be undone, and the phone
  number will be taken out of operation.
</Warning>

<CodeGroup>
  ```bash Request theme={null}
  curl -H "Content-Type: application/json" \
    -H "Authorization: Bearer $DAILY_API_KEY" \
    -XDELETE \
    https://api.daily.co/v1/release-phone-number/5ccc01ca-f448-4f17-b7b4-cf354b7b8256
  ```

  ```js Success theme={null}
  {
    "deleted": true,
    "id": "0cb313e1-211f-4be0-833d-8c7305b19902",
  }
  ```

  ```js Failure theme={null}
  {
    ("Failed to release new number, try again in 14 days");
  }
  ```
</CodeGroup>

### List all purchased phone numbers

You can also list all your purchased phone numbers using the following [API](https://docs.daily.co/reference/rest-api/phone-numbers/purchased-phone-numbers)

<CodeGroup>
  ```bash Request theme={null}
  curl --request GET \
    --url 'https://api.daily.co/v1/purchased-phone-numbers' \
    --header 'Authorization: Bearer $DAILY_API_KEY'
  ```

  ```js Success theme={null}
  {
      "total_count": 1,
      "data": [{
          "name": null,
          "id": "5ccc01ca-f448-4f17-b7b4-cf354b7b8256",
          "created_date": "Mon, 19 Aug 2024 18:50:34 +0000",
          "type": "purchased_phone_number",
          "number": "+19499800001",
          "status": "verified",
          "verified": true
      }]
  }
  ```
</CodeGroup>

### CNAM Registration

Please contact `help@daily.co` for adding Caller Name (CNAM) to the purchased phone numbers. Daily will need your Name or Company Name to register with the United States CNAM Registry. Once registered, the Caller ID will display the appropriate name on the calling party's device.

We need the following information:

* All phone numbers and the corresponding CNAM entry. A CNAM entry can only be 14 letter long, including spaces.
* Company documents
* An employee's ID/driver's license

<Info>
  It can take up to 48 hours for the CNAM to reflect on the national caller ID
  name database and can take longer to update for major carriers, who check the
  national database periodically.
</Info>

---

# Source: https://docs.pipecat.ai/guides/telephony/daily-pstn.md

> ## Documentation Index
> Fetch the complete documentation index at: https://docs.pipecat.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Daily PSTN

> Complete guide to Daily's PSTN capabilities including dial-in, dial-out, and call transfers

## Things you'll need

* An active [Daily](https://www.daily.co) developer account with API key
* At least one phone number purchased through Daily (covered below)
* For local development: ngrok to expose your bot to the Internet

<CardGroup cols={3}>
  <Card title="Dial-in Example" icon="phone-arrow-down-left" href="https://github.com/pipecat-ai/pipecat-examples/tree/main/phone-chatbot/daily-pstn-dial-in">
    Complete dial-in implementation with the Pipecat development runner
  </Card>

  <Card title="Dial-out Example" icon="phone-arrow-up-right" href="https://github.com/pipecat-ai/pipecat-examples/tree/main/phone-chatbot/daily-pstn-dial-out">
    Outbound calling with Daily PSTN integration
  </Card>

  <Card title="Cold Transfer Example" icon="arrow-right-arrow-left" href="https://github.com/pipecat-ai/pipecat-examples/tree/main/phone-chatbot/daily-pstn-cold-transfer">
    Cold transfer implementation for advanced call routing
  </Card>
</CardGroup>

## Phone Number Management

Before setting up dial-in or dial-out, you'll need to purchase phone numbers through Daily.

<Card title="Daily Phone Numbers Guide" icon="phone" href="./daily-phone-numbers">
  Complete guide to searching, purchasing, and managing phone numbers with
  Daily's REST API
</Card>

## Dial-in

Dial-in allows users to call your phone number and connect directly to your Pipecat bot.

### How It Works

Here's the sequence of events when someone calls your Daily phone number:

1. **Daily receives an incoming call** to your phone number
2. **Daily calls your webhook endpoint**
3. **The webhook creates a Daily room** with SIP configuration
4. **The webhook starts your bot** with the room details and caller information
5. **The caller is put on hold** with music
6. **The bot joins the Daily room** and signals readiness
7. **Daily forwards the call** to the Daily room
8. **The caller and bot are connected** for the conversation

### Local Development

The Pipecat development runner provides built-in webhook handling for Daily PSTN dial-in, eliminating the need for a separate webhook server.

#### 1. Run your bot with dial-in support

```bash  theme={null}
uv run bot.py -t daily --dialin
```

This starts a FastAPI server on port 7860 with a `/daily-dialin-webhook` endpoint.

#### 2. Expose your bot to the internet

```bash  theme={null}
ngrok http 7860
```

Copy the ngrok URL (e.g., `https://abc123.ngrok.io`).

<Tip>Use `ngrok http 7860 --subdomain your-subdomain` for a reusable URL.</Tip>

#### 3. Configure your Daily phone number

Set your phone number's `room_creation_api` webhook to:

```
https://your-ngrok-url.ngrok.io/daily-dialin-webhook
```

<Card title="Daily Dial-in Config API" icon="link" href="https://docs.daily.co/reference/rest-api/domainDialinConfig">
  Instructions for configuring the webhook URL for your phone number
</Card>

```shell Create pinless dial-in config theme={null}
curl --location 'https://api.daily.co/v1' \
--header 'Content-Type: application/json' \
--header 'Authorization: Bearer YOUR_DAILY_API_KEY' \
--data '{
    "properties": {
        "pinless_dialin": [
            {
                "phone_number": "DAILY_PROVISIONED_NUMBER_HERE",
                "room_creation_api": "https://your-ngrok-url.ngrok.io/daily-dialin-webhook"
            }
        ]
    }
}'
```

<Warning>
  Ensure your `DAILY_API_KEY` has the phone number associated with it. This is
  required for the `pinlessCallUpdate` API call that connects the caller to the
  room.
</Warning>

### Configure your Pipecat bot for dial-in

The bot receives dial-in information through `RunnerArguments` containing:

* `room_url`: Daily room URL for the call
* `token`: Daily room token for authentication
* `body`: Contains `DailyDialinRequest` with call details

**Bot Entry Point**:

```python bot.py theme={null}
from pipecat.runner.types import DailyDialinRequest, RunnerArguments
from pipecat.transports.daily import DailyTransport, DailyParams, DailyDialinSettings

async def bot(runner_args: RunnerArguments):
    # Parse dial-in request from runner
    request = DailyDialinRequest.model_validate(runner_args.body)

    # Configure dial-in settings with webhook data
    daily_dialin_settings = DailyDialinSettings(
        call_id=request.dialin_settings.call_id,
        call_domain=request.dialin_settings.call_domain
    )

    # Create transport with dial-in configuration
    transport = DailyTransport(
        runner_args.room_url,
        runner_args.token,
        "Voice Bot",
        DailyParams(
            api_key=request.daily_api_key,
            api_url=request.daily_api_url,
            dialin_settings=daily_dialin_settings,
            audio_in_enabled=True,
            audio_out_enabled=True,
        )
    )

    # Your bot setup and pipeline creation here...

if __name__ == "__main__":
    from pipecat.runner.run import main
    main()
```

<Card title="Complete Bot Implementation" icon="code" href="https://github.com/pipecat-ai/pipecat-examples/blob/main/phone-chatbot/daily-pstn-dial-in/bot.py">
  See the full bot.py with argument handling, transport setup, and pipeline
  configuration
</Card>

### Customize Your Bot with Caller Information

Use the caller's phone number to personalize the conversation:

```python  theme={null}
async def bot(runner_args: RunnerArguments):
    # Parse dial-in request
    request = DailyDialinRequest.model_validate(runner_args.body)

    # Get caller's phone number
    caller_phone = request.dialin_settings.From

    # Look up customer information from your database
    customer = await get_customer_by_phone(caller_phone)

    # Customize the system prompt
    messages = [
        {
            "role": "system",
            "content": f"You are a helpful assistant for {customer.name}. "
                      f"Their account status is {customer.status}. "
                      "Keep responses concise and conversational."
        }
    ]

    # Use the customized context in your bot...
```

### Run the Example

<Card title="Complete Setup Instructions" icon="book-open" href="https://github.com/pipecat-ai/pipecat-examples/tree/main/phone-chatbot/daily-pstn-dial-in">
  See the full README with step-by-step setup, environment variables, and
  troubleshooting tips
</Card>

## Dial-out

Dial-out allows your bot to initiate calls to phone numbers. Unlike dial-in, your bot starts the call rather than waiting for incoming calls.

<Warning>
  You must contact Daily to enable dial-out for your account. Submit a support
  request
  [here](https://docs.google.com/forms/d/1EHy2wsX20a2HjljAsJkyPqvATIM4WCvm7JzYKypGYis/edit).
</Warning>

### How It Works

Here's the sequence of events for dial-out calls:

1. **Your application triggers a dial-out** (via API call or user action)
2. **Server creates a Daily room** with dial-out capabilities enabled
3. **Bot joins the Daily room** and sets up the pipeline
4. **Bot initiates the dial-out call** to the target phone number
5. **Daily connects the call** to the phone number
6. **The recipient answers** and is connected to your bot
7. **The bot handles the conversation** with the called party

### Set up your server for dial-out

The dial-out server is simpler than dial-in since you're initiating calls rather than receiving webhooks. Your server needs to:

1. **Create Daily rooms** with dial-out enabled
2. **Start bot processes** with the target phone number
3. **Handle API requests** to trigger outbound calls

<Card title="Complete Server Implementation" icon="code" href="https://github.com/pipecat-ai/pipecat-examples/blob/main/phone-chatbot/daily-pstn-dial-out/server.py">
  See the full FastAPI server code with room creation and dial-out triggering
</Card>

### Configure your Pipecat bot for dial-out

The dial-out bot receives the target phone number and creates a transport without dial-in settings:

**Bot Entry Point**: The `bot()` method receives `RunnerArguments` containing:

* `room_url`: Daily room URL for the call
* `token`: Daily room token for authentication
* `phone_number`: Target phone number to call

**Transport Creation**: These arguments are used to configure the `DailyTransport`:

```python bot.py theme={null}
from pipecat.transports.daily import DailyTransport, DailyParams

async def bot(args: RunnerArguments):
    # Create transport for dial-out (no dial-in settings needed)
    transport = DailyTransport(
        args.room_url,
        args.token,
        "Voice Bot",
        DailyParams(
            api_url=os.getenv("DAILY_API_URL", "https://api.daily.co/v1"),
            api_key=os.getenv("DAILY_API_KEY"),
            audio_in_enabled=True,
            audio_out_enabled=True,
            video_out_enabled=False,
            transcription_enabled=True,
        )
    )

    # Start the dial-out call
    await transport.start_dialout(args.phone_number)

    # Pass transport to your bot pipeline
    # Your bot setup and pipeline creation here...
```

<Card title="Complete Bot Implementation" icon="code" href="https://github.com/pipecat-ai/pipecat-examples/blob/main/phone-chatbot/daily-pstn-dial-out/bot.py">
  See the full bot.py with dial-out configuration and pipeline setup
</Card>

### Run the Example

To test dial-out functionality:

1. **Start your server**: Run your FastAPI server
2. **Trigger a call**: Make an API request to start a dial-out call
3. **Answer your phone**: The bot will call the specified number
4. **Talk to your bot**: Have a conversation with your AI agent

<Card title="Complete Setup Instructions" icon="book-open" href="https://github.com/pipecat-ai/pipecat-examples/tree/main/phone-chatbot/daily-pstn-dial-out">
  See the full README with step-by-step setup, API usage, and configuration
  details
</Card>

## Call Transfers

Daily supports cold transfers, allowing you to transfer an active call to another phone number. The bot can initiate a transfer and then leave the call, connecting the caller directly to the transfer destination.

### How Call Transfers Work

1. **Bot receives transfer request** (via function call or user input)
2. **Bot informs the caller** about the transfer
3. **Bot initiates SIP call transfer** to the destination number
4. **Daily connects the transfer** to the destination
5. **Bot leaves the call** (cold transfer)
6. **Caller and destination** continue the conversation

### Implementing Call Transfers

Call transfers are typically implemented as LLM function calls that your bot can invoke:

```python bot.py theme={null}
async def dial_operator(transport: BaseTransport, params: FunctionCallParams):
    """Function the bot can call to transfer to an operator."""
    operator_number = os.getenv("OPERATOR_NUMBER", "+1234567890")

    # Inform the user about the transfer
    content = "I'm transferring you to a supervisor now. Please hold while I connect you."
    message = {"role": "system", "content": content}
    await params.llm.push_frame(LLMMessagesAppendFrame([message], run_llm=True))

    # Execute the SIP call transfer
    transfer_params = {"toEndPoint": operator_number}
    await transport.sip_call_transfer(transfer_params)
```

### Handling Transfer Events

Your bot should handle transfer-related events to manage the call flow:

```python bot.py theme={null}
@transport.event_handler("on_dialout_answered")
async def on_dialout_answered(transport, data):
    logger.info(f"Transfer successful: {data}")
    # Cold transfer: bot leaves, caller and operator continue
    await task.queue_frames([EndFrame()])
```

<Card title="Complete Transfer Implementation" icon="code" href="https://github.com/pipecat-ai/pipecat-examples/tree/main/phone-chatbot/daily-pstn-cold-transfer">
  See the full call transfer example with LLM function calls, event handling,
  and error management
</Card>

## Deployment

### Pipecat Cloud

For production deployment without managing your own infrastructure, use Pipecat Cloud. Pipecat Cloud handles all webhook infrastructure, room creation, and bot scaling automatically.

<CardGroup cols={2}>
  <Card title="Daily Dial-in on Pipecat Cloud" icon="phone-arrow-down-left" href="/deployment/pipecat-cloud/guides/telephony/daily-dial-in">
    Complete guide for deploying dial-in bots with automatic webhook handling
  </Card>

  <Card title="Daily Dial-out on Pipecat Cloud" icon="phone-arrow-up-right" href="/deployment/pipecat-cloud/guides/telephony/daily-dial-out">
    Complete guide for deploying dial-out bots with caller ID management
  </Card>
</CardGroup>

### Self-Hosted Deployment

For self-hosted production deployment, ensure your servers are:

* Publicly accessible with HTTPS
* Able to handle concurrent requests
* Properly configured with your Daily API credentials

## Next Steps

* Explore the [complete examples](https://github.com/pipecat-ai/pipecat/tree/main/examples) for full implementations
* Learn about [Daily's SIP integration](./twilio-daily-sip) for more advanced telephony scenarios
* Check out [Daily's REST API documentation](https://docs.daily.co/reference) for additional configuration options

---

# Source: https://docs.pipecat.ai/deployment/pipecat-cloud/guides/daily-webrtc.md

> ## Documentation Index
> Fetch the complete documentation index at: https://docs.pipecat.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Daily WebRTC

> Using Daily as a WebRTC transport for your Pipecat Cloud agents

[Daily](https://www.daily.co) is a WebRTC platform that provides real-time voice and video capabilities to connect users with voice agents.

Pipecat Cloud offers a first-class integration with Daily, making it straightforward to deploy WebRTC-enabled agents without managing complex infrastructure yourself.

## Using an Integrated Daily API Key

When you create a Pipecat Cloud account, you're automatically provisioned with a Daily API key that's fully integrated with the platform.

The integrated Daily API key provides:

* **Zero configuration**: No need to separately sign up for Daily or manage API keys
* **Free voice minutes**: Voice minutes for one human participant and one agent are free when using your Pipecat Cloud-provisioned Daily key
* **Simplified deployment**: Create Daily rooms and launch agents with a single command
* **Built-in compatibility**: All Pipecat base images work with Daily out of the box

<Info>
  Free voice minutes apply only to 1:1 voice sessions running on Pipecat Cloud.
  Video sessions, WebRTC sessions outside of Pipecat Cloud, and additional Daily
  features (recording, transcription, PSTN/SIP) are billed according to [Daily's
  standard pricing](https://www.daily.co/pricing/webrtc-infrastructure/).
</Info>

## Using Daily with Pipecat Cloud Agents

### Starting an Agent with Daily Transport

When starting a Pipecat Cloud agent, you can specify Daily as the transport using the `--use-daily` flag with the CLI or setting the appropriate parameters in the SDK or REST API calls.

<Tabs>
  <Tab title="CLI">
    ```bash  theme={null}
    # Start an agent instance with a Daily room
    pipecat cloud agent start my-agent-name --use-daily

    # Start with custom Daily room properties
    pipecat cloud agent start my-agent-name --use-daily --daily-properties '{"enable_recording": "cloud"}'
    ```

    This command creates a Daily room and provides a URL you can open in your browser to interact with your agent using voice.
  </Tab>

  <Tab title="Python SDK">
    ```python  theme={null}
    from pipecatcloud import Session, SessionParams

    # Create and start a session
    session = Session(
        agent_name="my-agent",
        api_key="pk_...",
        params=SessionParams(
            use_daily=True,
            # Optionally customize the Daily room
            daily_room_properties={
                "enable_recording": "cloud"
            }
        )
    )
    response = await session.start()

    # Access the Daily room URL and token
    daily_url = f"{response['dailyRoom']}?t={response['dailyToken']}"
    print(f"Join the call at: {daily_url}")
    ```
  </Tab>

  <Tab title="REST API">
    ```bash  theme={null}
    curl --location --request POST 'https://api.pipecat.daily.co/v1/public/my-agent-name/start' \
      --header 'Authorization: Bearer YOUR_API_KEY' \
      --header 'Content-Type: application/json' \
      --data-raw '{
        "createDailyRoom": true,
        "dailyRoomProperties": {
          "enable_recording": "cloud"
        }
      }'
    ```

    The response includes `dailyRoom` (URL) and `dailyToken` fields that can be used to join the call.
  </Tab>
</Tabs>

### Using Your Own Daily API Key

While Pipecat Cloud provides a Daily API key with included 1:1 voice minutes, you can optionally use your own Daily API key if you have specific requirements. When using your own key, all usage will be billed according to your Daily account's pricing plan rather than being included with your Pipecat Cloud subscription.

## WebRTC vs. WebSockets for Voice AI

When building voice AI applications, choosing the right transport technology is crucial:

* **WebRTC** is purpose-built for real-time audio and video communication between browsers and devices:

  * Optimized for real-time media streaming over unpredictable networks
  * Uses UDP protocol, prioritizing speed over guaranteed packet delivery
  * Provides built-in echo cancellation and noise suppression
  * Intelligently adapts bitrate based on changing network conditions
  * Handles NAT traversal for connections across different networks
  * Includes device management for cameras and microphones

* **WebSockets** work well for server-to-server communication because:
  * They operate in controlled network environments with stable connections
  * When server-to-server latency is low, packet retransmission doesn't add substantial delay
  * Server environments don't need device access or media quality enhancements
  * They're simpler to implement for pure data transmission
  * Many server platforms have built-in WebSocket support

<Info>
  For browser or mobile app voice applications, WebRTC delivers superior
  performance across varying network conditions. For more details, see [How to
  Talk to an LLM with Your
  Voice](https://www.daily.co/blog/how-to-talk-to-an-llm-with-your-voice/#webrtc).
</Info>

When connecting to telephony systems like Twilio or implementing server-to-server communication where network conditions are controlled and reliable, WebSockets remain an appropriate choice. However, for any user-facing voice or video application, WebRTC offers significant advantages in quality and reliability.

## How it Works

When you start an agent with Daily integration:

1. Pipecat Cloud creates a Daily room using your API key (either the provisioned one or your custom key)
2. A URL and access token is generated, creating a means to access the Daily room
3. The agent is started and configured to join the Daily room
4. Your code (in `bot.py`) receives the room URL and token as parameters
5. The Pipecat framework automatically connects to the Daily room using these parameters

The basic flow is handled for you, allowing you to focus on building your agent's conversation logic rather than WebRTC infrastructure.

## Next Steps

<CardGroup cols={3}>
  <Card title="Daily WebRTC docs" icon="globe" href="https://docs.daily.co">
    Explore Daily's complete documentation for additional features
  </Card>

  <Card title="DailyTransport Pipecat Docs" icon="puzzle-piece" href="/server/services/transport/daily">
    Explore Pipecat's DailyTransport documentation
  </Card>

  <Card title="Pipecat Examples" icon="code" href="https://github.com/pipecat-ai/pipecat-examples">
    View example Pipecat projects using Daily WebRTC
  </Card>
</CardGroup>

---

# Source: https://docs.pipecat.ai/server/services/transport/daily.md

# Source: https://docs.pipecat.ai/client/react-native/transports/daily.md

# Source: https://docs.pipecat.ai/client/js/transports/daily.md

# Source: https://docs.pipecat.ai/client/ios/transports/daily.md

# Source: https://docs.pipecat.ai/client/android/transports/daily.md

> ## Documentation Index
> Fetch the complete documentation index at: https://docs.pipecat.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Daily WebRTC Transport

> WebRTC implementation for Android using Daily

The Daily transport implementation enables real-time audio and video communication in your Pipecat Android applications using [Daily's](https://www.daily.co/) WebRTC infrastructure.

## Installation

Add the Daily transport dependency to your `build.gradle`:

```gradle  theme={null}
implementation "ai.pipecat:daily-transport:1.0.3"
```

## Usage

Create a client using the Daily transport:

```kotlin  theme={null}
val callbacks = object : PipecatEventCallbacks() {

    override fun onBackendError(message: String) {
        Log.e(TAG, "Error from backend: $message")
    }

    // ...
}

val options = PipecatClientOptions(callbacks = callbacks)

val client: PipecatClientDaily = PipecatClient(DailyTransport(context), options)

// Kotlin coroutines
client.startBotAndConnect(startBotParams).await()

// Callbacks
client.startBotAndConnect(startBotParams).withCallback {
    // ...
}
```

## Configuration

Your server endpoint should return Daily-specific configuration:

```json  theme={null}
{
  "dailyRoom": "https://your-domain.daily.co/room-name",
  "dailyToken": "your-daily-token"
}
```

## Resources

<CardGroup cols={2}>
  <Card horizontal title="Demo" icon="play" href="https://github.com/pipecat-ai/pipecat-examples/tree/main/simple-chatbot/client/android">
    Simple Chatbot Demo
  </Card>

  <Card horizontal title="Source" icon="github" href="https://github.com/pipecat-ai/pipecat-client-android-transports/">
    Client Transports
  </Card>
</CardGroup>

<Card title="Daily Transport Reference" icon="book-open" href="https://docs-android.rtvi.ai/">
  Complete API documentation for the Daily transport implementation
</Card>

---

# Source: https://docs.pipecat.ai/server/utilities/observers/debug-observer.md

> ## Documentation Index
> Fetch the complete documentation index at: https://docs.pipecat.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Debug Log Observer

> Comprehensive frame logging with configurable filtering in Pipecat

The `DebugLogObserver` provides detailed logging of frame activity in your Pipecat pipeline, with full visibility into frame content and flexible filtering options.

## Features

* Log all frame types and their content
* Filter by specific frame types
* Filter by source or destination components
* Automatic formatting of frame fields
* Special handling for complex data structures

## Usage

### Log All Frames

Log all frames passing through the pipeline:

```python  theme={null}
from pipecat.observers.loggers.debug_log_observer import DebugLogObserver

task = PipelineTask(
    pipeline,
    params=PipelineParams(
        observers=[DebugLogObserver()],
    ),
)
```

### Filter by Frame Types

Log only specific frame types:

```python  theme={null}
from pipecat.frames.frames import TranscriptionFrame, InterimTranscriptionFrame
from pipecat.observers.loggers.debug_log_observer import DebugLogObserver

task = PipelineTask(
    pipeline,
    params=PipelineParams(
        observers=[
            DebugLogObserver(frame_types=(
                TranscriptionFrame,
                InterimTranscriptionFrame
            ))
        ],
    ),
)
```

### Advanced Source/Destination Filtering

Filter frames based on their type and source/destination:

```python  theme={null}
from pipecat.frames.frames import StartInterruptionFrame, UserStartedSpeakingFrame, LLMTextFrame
from pipecat.observers.loggers.debug_log_observer import DebugLogObserver, FrameEndpoint
from pipecat.transports.base_output_transport import BaseOutputTransport
from pipecat.services.stt_service import STTService

task = PipelineTask(
    pipeline,
    params=PipelineParams(
        observers=[
            DebugLogObserver(frame_types={
                # Only log StartInterruptionFrame when source is BaseOutputTransport
                StartInterruptionFrame: (BaseOutputTransport, FrameEndpoint.SOURCE),

                # Only log UserStartedSpeakingFrame when destination is STTService
                UserStartedSpeakingFrame: (STTService, FrameEndpoint.DESTINATION),

                # Log LLMTextFrame regardless of source or destination
                LLMTextFrame: None
            })
        ],
    ),
)
```

## Log Output Format

The observer logs each frame with its complete details:

```
[Source] → [Destination]: [FrameType] [field1: value1, field2: value2, ...] at [timestamp]s
```

For example:

```
OpenAILLMService#0 → DailyTransport#0: LLMTextFrame text: 'Hello, how can I help you today?' at 1.24s
```

## Configuration Options

| Parameter        | Type                                                                                   | Description                                                     |
| ---------------- | -------------------------------------------------------------------------------------- | --------------------------------------------------------------- |
| `frame_types`    | `Tuple[Type[Frame], ...]` or `Dict[Type[Frame], Optional[Tuple[Type, FrameEndpoint]]]` | Frame types to log, with optional source/destination filtering  |
| `exclude_fields` | `Set[str]`                                                                             | Field names to exclude from logging (defaults to binary fields) |

## FrameEndpoint Enum

The `FrameEndpoint` enum is used for source/destination filtering:

* `FrameEndpoint.SOURCE`: Filter by source component
* `FrameEndpoint.DESTINATION`: Filter by destination component

---

# Source: https://docs.pipecat.ai/server/services/tts/deepgram.md

# Source: https://docs.pipecat.ai/server/services/stt/deepgram.md

> ## Documentation Index
> Fetch the complete documentation index at: https://docs.pipecat.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Deepgram

> Speech-to-text service implementations using Deepgram's real-time transcription and Flux APIs

## Overview

Deepgram provides two STT service implementations:

* `DeepgramSTTService` for real-time speech recognition using Deepgram's standard WebSocket API with support for interim results, language detection, and voice activity detection (VAD)
* `DeepgramFluxSTTService` for advanced conversational AI with Flux capabilities including intelligent turn detection, eager end-of-turn events, and enhanced speech processing for improved response timing.

<Note>
  Since Deepgram Flux provides its own user turn start and end detection, you
  should use `ExternalUserTurnStrategies` to let Flux handle turn management.
  See [User Turn
  Strategies](/server/utilities/turn-management/user-turn-strategies) for
  configuration details.
</Note>

<CardGroup cols={2}>
  <Card title="Deepgram STT API Reference" icon="code" href="https://reference-server.pipecat.ai/en/latest/api/pipecat.services.deepgram.stt.html">
    Pipecat's API methods for standard Deepgram STT
  </Card>

  <Card title="Deepgram Flux API Reference" icon="code" href="https://reference-server.pipecat.ai/en/latest/api/pipecat.services.deepgram.flux.stt.html">
    Pipecat's API methods for Deepgram Flux STT
  </Card>

  <Card title="Standard STT Example" icon="play" href="https://github.com/pipecat-ai/pipecat/blob/main/examples/foundational/07c-interruptible-deepgram.py">
    Complete example with standard Deepgram STT
  </Card>

  <Card title="Flux STT Example" icon="play" href="https://github.com/pipecat-ai/pipecat/blob/main/examples/foundational/07c-interruptible-deepgram-flux.py">
    Complete example with Deepgram Flux STT
  </Card>

  <Card title="Deepgram Documentation" icon="book" href="https://developers.deepgram.com/docs/pre-recorded-audio">
    Official Deepgram documentation and features
  </Card>

  <Card title="Deepgram Console" icon="microphone" href="https://console.deepgram.com/signup">
    Access API keys and transcription models
  </Card>
</CardGroup>

## Installation

To use Deepgram services, install the required dependencies:

```bash  theme={null}
pip install "pipecat-ai[deepgram]"
```

## Prerequisites

### Deepgram Account Setup

Before using Deepgram STT services, you need:

1. **Deepgram Account**: Sign up at [Deepgram Console](https://console.deepgram.com/signup)
2. **API Key**: Generate an API key from your console dashboard
3. **Model Selection**: Choose from available transcription models and features

### Required Environment Variables

* `DEEPGRAM_API_KEY`: Your Deepgram API key for authentication

---

# Source: https://docs.pipecat.ai/server/services/llm/deepseek.md

> ## Documentation Index
> Fetch the complete documentation index at: https://docs.pipecat.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# DeepSeek

> LLM service implementation using DeepSeek's API with OpenAI-compatible interface

## Overview

`DeepSeekLLMService` provides access to DeepSeek's language models through an OpenAI-compatible interface. It inherits from `OpenAILLMService` and supports streaming responses, function calling, and context management with advanced reasoning capabilities.

<CardGroup cols={2}>
  <Card title="DeepSeek LLM API Reference" icon="code" href="https://reference-server.pipecat.ai/en/latest/api/pipecat.services.deepseek.llm.html">
    Pipecat's API methods for DeepSeek integration
  </Card>

  <Card title="Example Implementation" icon="play" href="https://github.com/pipecat-ai/pipecat/blob/main/examples/foundational/14l-function-calling-deepseek.py">
    Complete example with function calling
  </Card>

  <Card title="DeepSeek Documentation" icon="book" href="https://api-docs.deepseek.com/api/create-chat-completion">
    Official DeepSeek API documentation and features
  </Card>

  <Card title="DeepSeek Platform" icon="microphone" href="https://platform.deepseek.com/">
    Access models and manage API keys
  </Card>
</CardGroup>

## Installation

To use DeepSeek services, install the required dependency:

```bash  theme={null}
pip install "pipecat-ai[deepseek]"
```

## Prerequisites

### DeepSeek Account Setup

Before using DeepSeek LLM services, you need:

1. **DeepSeek Account**: Sign up at [DeepSeek Platform](https://platform.deepseek.com/)
2. **API Key**: Generate an API key from your account dashboard
3. **Model Selection**: Choose from available DeepSeek models with reasoning capabilities

### Required Environment Variables

* `DEEPSEEK_API_KEY`: Your DeepSeek API key for authentication

---

# Source: https://docs.pipecat.ai/deployment/pipecat-cloud/fundamentals/deploy.md

# Source: https://docs.pipecat.ai/cli/cloud/deploy.md

> ## Documentation Index
> Fetch the complete documentation index at: https://docs.pipecat.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# deploy

> Create or modify an agent deployment

The `deploy` command creates a new agent deployment or updates an existing one. It builds a deployment manifest with the provided parameters and monitors the deployment status until the agent is ready.

If the agent name already exists, you'll be prompted to confirm the update unless the `--force` flag is used.

This command will wait for the active deployment / revision to enter a ready state before returning. If the deployment fails, the command will exit with an [error](/deployment/pipecat-cloud/fundamentals/error-codes) with more information.

## Usage

```shell  theme={null}
pipecat cloud deploy [ARGS] [OPTIONS]
```

**Arguments:**

<ParamField path="agent-name" type="string" required>
  Unique string identifier for the agent deployment. Must not contain spaces.
</ParamField>

<ParamField path="image" type="string" required>
  URL of the Docker image to deploy. Must be a valid Docker image URL. For
  example: `docker.io/my-repo/my-image:latest`.
</ParamField>

**Options:**

<ParamField path="--credentials / -c" type="string">
  Name of the image pull secret to use for accessing private repositories. The
  secret must be previously created using the `pipecat cloud secrets
    image-pull-secret` command.
</ParamField>

<ParamField path="--organization / -o" type="string">
  Organization to deploy the agent to. If not provided, uses the current
  organization from your configuration.
</ParamField>

<ParamField path="--secrets / -s" type="string">
  Name of the secret set to use for the deployment. The secret set must exist in
  the specified organization.
</ParamField>

<ParamField path="--min-agents / -min" type="number" default="0">
  Minimum number of agent instances to keep warm at all times. Default is 0,
  which means the agent will scale down to zero when not in use. Setting this to
  1 or higher avoids cold starts.
</ParamField>

<ParamField path="--max-agents / -max" type="number" default="50">
  Maximum number of allowed agent instances. Must be between 1 and 50. If you
  need more agents, please contact us at [help@daily.co](mailto:help@daily.co) or via
  [Discord](https://discord.gg/dailyco).
</ParamField>

<ParamField path="--krisp-viva-audio-filter" type="string">
  Enable Krisp VIVA noise cancellation with the specified audio filter model.
  Valid values are:

  * `tel`: Telephony model (up to 16kHz)
  * `pro`: WebRTC model (up to 32kHz)

  In addition to this flag, you also need to enable the `KrispVivaFilter()` for your transport. See the [Krisp VIVA](/server/utilities/audio/krisp-viva-filter) documentation for more information.
</ParamField>

<ParamField path="--enable-managed-keys" type="boolean" default="false">
  Enable [managed API keys](/deployment/pipecat-cloud/guides/managed-api-keys)
  for this deployed agent.
</ParamField>

<ParamField path="--profile / -p" type="string" default="agent-1x">
  The agent profile to use for resource allocation. Valid values are:
  `agent-1x`, `agent-2x`, `agent-3x`.

  See [Agent Profiles](/deployment/pipecat-cloud/fundamentals/deploy#agent-profiles) for more information.
</ParamField>

<ParamField path="--region / -r" type="string">
  Region where the agent will be deployed. If not specified, uses your
  organization's default region (typically `us-west`). Choose a region close to
  your users for optimal latency.
</ParamField>

<ParamField path="--force / -f" type="boolean" default="false">
  Force deployment and skip confirmation prompts. Use with caution.
</ParamField>

## Examples

**Deploy a new agent:**

```shell  theme={null}
pipecat cloud deploy my-first-agent your-docker-repository/my-first-agent:0.1
```

**Update an existing agent with a new image:**

```shell  theme={null}
pipecat cloud deploy my-first-agent your-docker-repository/my-first-agent:0.2
```

**Deploy with a specific secret set:**

```shell  theme={null}
pipecat cloud deploy my-first-agent your-docker-repository/my-first-agent:0.1 --secrets my-secret-set
```

**Deploy a private image using image pull credentials:**

```shell  theme={null}
pipecat cloud deploy my-first-agent your-docker-repository/my-first-agent:0.1 --credentials dockerhub-creds
```

**Keep one instance always warm to avoid cold starts:**

```shell  theme={null}
pipecat cloud deploy my-first-agent your-docker-repository/my-first-agent:0.1 --min-agents 1
```

**Limit the maximum number of agent instances:**

```shell  theme={null}
pipecat cloud deploy my-first-agent your-docker-repository/my-first-agent:0.1 --max-agents 5
```

**Deploy to a specific region:**

```shell  theme={null}
pipecat cloud deploy my-first-agent your-docker-repository/my-first-agent:0.1 --region eu-central
```

**Deploy with Krisp VIVA noise cancellation:**

```shell  theme={null}
pipecat cloud deploy my-first-agent your-docker-repository/my-first-agent:0.1 --krisp-viva-audio-filter tel
```

## Configuration File (pcc-deploy.toml)

The `deploy` command supports a configuration file for repeatable deployments. Create a `pcc-deploy.toml` file in your project root to define deployment settings that can be shared across your team and version controlled.

### File Location

Place `pcc-deploy.toml` in the same directory where you run the `pipecat cloud deploy` command. The CLI will automatically detect and use this file.

### Precedence

Values are applied with the following order of precedence:

1. CLI arguments (highest priority)
2. `pcc-deploy.toml` values
3. Default values (lowest priority)

This allows you to define defaults in the config file while still overriding specific values via CLI flags when needed.

### Configuration Options

#### Required Fields

<ParamField path="agent_name" type="string" required>
  Name of the agent to deploy. Must start with a lowercase letter or number, can
  include hyphens, and must end with a lowercase letter or number.

  ```toml  theme={null}
  agent_name = "my-voice-agent"
  ```
</ParamField>

<ParamField path="image" type="string" required>
  Docker image URL with tag.

  ```toml  theme={null}
  image = "your-dockername/my-agent:0.1"
  ```
</ParamField>

#### Optional Fields

<ParamField path="region" type="string">
  Region where the agent will be deployed. If not specified, uses your
  organization's default region (typically `us-west`).

  ```toml  theme={null}
  region = "us-east"
  ```
</ParamField>

<ParamField path="secret_set" type="string">
  Name of the secret set to use for environment variables. The secret set must
  exist in the same region as the agent.

  ```toml  theme={null}
  secret_set = "my-agent-secrets"
  ```
</ParamField>

<ParamField path="image_credentials" type="string">
  Name of the image pull secret for private registries. The image pull secret must exist in the same region as the agent.

  ```toml  theme={null}
  image_credentials = "dockerhub-credentials"
  ```
</ParamField>

<ParamField path="agent_profile" type="string" default="agent-1x">
  Agent profile for resource allocation. Valid values: `agent-1x`, `agent-2x`, `agent-3x`.

  ```toml  theme={null}
  agent_profile = "agent-2x"
  ```
</ParamField>

<ParamField path="enable_krisp" type="boolean" default="false">
  **Deprecated:** Enable legacy Krisp noise cancellation. Use `krisp_viva` instead.

  ```toml  theme={null}
  enable_krisp = false
  ```
</ParamField>

<ParamField path="enable_managed_keys" type="boolean" default="false">
  Enable managed API keys for this deployment.

  ```toml  theme={null}
  enable_managed_keys = false
  ```
</ParamField>

#### Scaling Configuration

Define auto-scaling behavior in a `[scaling]` section:

<ParamField path="scaling.min_agents" type="number" default="0">
  Minimum number of agent instances to keep warm. Setting to 0 allows scaling to zero but may result in cold starts.

  ```toml  theme={null}
  [scaling]
  min_agents = 1
  ```
</ParamField>

<ParamField path="scaling.max_agents" type="number" default="50">
  Maximum number of agent instances allowed.

  ```toml  theme={null}
  [scaling]
  max_agents = 20
  ```
</ParamField>

#### Krisp VIVA Configuration

Configure Krisp VIVA noise cancellation in a `[krisp_viva]` section:

<ParamField path="krisp_viva.audio_filter" type="string">
  Krisp VIVA audio filter model. Valid values: `tel` (telephony, up to 16kHz) or `pro` (WebRTC, up to 32kHz). Omit or set to `null` to disable.

  ```toml  theme={null}
  [krisp_viva]
  audio_filter = "tel"
  ```
</ParamField>

### Complete Example

```toml  theme={null}
# Basic agent configuration
agent_name = "my-voice-agent"
image = "your-dockername/my-voice-agent:0.1"
region = "us-west"

# Secrets and credentials
secret_set = "my-agent-secrets"
image_credentials = "dockerhub-credentials"

# Resource allocation
agent_profile = "agent-1x"

# Features
enable_managed_keys = false

# Auto-scaling configuration
[scaling]
min_agents = 1
max_agents = 20

# Krisp VIVA noise cancellation
[krisp_viva]
audio_filter = "tel"
```

### Using the Configuration File

Once you have a `pcc-deploy.toml` file, simply run:

```shell  theme={null}
pipecat cloud deploy
```

The CLI will automatically load your configuration. You can still override any value using CLI flags:

```shell  theme={null}
# Use config file but override the region
pipecat cloud deploy --region eu-central

# Use config file but force update without confirmation
pipecat cloud deploy --force
```

---

# Source: https://docs.pipecat.ai/guides/fundamentals/detecting-user-idle.md

> ## Documentation Index
> Fetch the complete documentation index at: https://docs.pipecat.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Detecting Idle Users

> Learn how to detect and respond when users are inactive in conversations

## Overview

In conversational applications, it's important to handle situations where users go silent or inactive. Pipecat provides built-in idle detection through `LLMUserAggregator` and `UserTurnProcessor`, allowing your bot to respond appropriately when users haven't spoken for a defined period.

## How It Works

Idle detection monitors user activity and:

1. Starts tracking after the first interaction (user or bot speaking)
2. Resets a timer whenever the user or bot speaks
3. Pauses during function calls (which may take longer than the timeout)
4. Emits an `on_user_turn_idle` event when the user is idle for longer than the timeout period
5. Allows you to implement escalating responses or gracefully end the conversation in your application code

<Note>
  Idle detection uses continuous activity frames (`UserSpeakingFrame`,
  `BotSpeakingFrame`) to track real-time conversation activity. It requires an
  active speech-to-text service or a transport with built-in speech detection.
</Note>

## Basic Implementation

### Step 1: Enable Idle Detection

Enable idle detection by setting the `user_idle_timeout` parameter when creating your aggregator:

```python  theme={null}
from pipecat.processors.aggregators.llm_response_universal import (
    LLMContextAggregatorPair,
    LLMUserAggregatorParams,
)

user_aggregator, assistant_aggregator = LLMContextAggregatorPair(
    context,
    user_params=LLMUserAggregatorParams(
        user_idle_timeout=5.0,  # Detect idle after 5 seconds
    ),
)
```

### Step 2: Handle Idle Events

Create an event handler to respond when the user becomes idle:

```python  theme={null}
@user_aggregator.event_handler("on_user_turn_idle")
async def on_user_turn_idle(aggregator):
    # Send a reminder to the user
    message = {
        "role": "system",
        "content": "The user has been quiet. Politely ask if they're still there.",
    }
    await aggregator.push_frame(LLMMessagesAppendFrame([message], run_llm=True))
```

### Step 3: Implement Retry Logic (Optional)

For escalating responses, track retry count in your application:

```python  theme={null}
class IdleHandler:
    def __init__(self):
        self._retry_count = 0

    def reset(self):
        self._retry_count = 0

    async def handle_idle(self, aggregator):
        self._retry_count += 1

        if self._retry_count == 1:
            # First attempt - gentle reminder
            message = {
                "role": "system",
                "content": "The user has been quiet. Politely ask if they're still there.",
            }
            await aggregator.push_frame(LLMMessagesAppendFrame([message], run_llm=True))
        elif self._retry_count == 2:
            # Second attempt - more direct
            message = {
                "role": "system",
                "content": "The user is still inactive. Ask if they'd like to continue.",
            }
            await aggregator.push_frame(LLMMessagesAppendFrame([message], run_llm=True))
        else:
            # Third attempt - end conversation
            await aggregator.push_frame(
                TTSSpeakFrame("It seems like you're busy. Have a nice day!")
            )
            await aggregator.push_frame(EndTaskFrame(), FrameDirection.UPSTREAM)

# Use the handler
idle_handler = IdleHandler()

@user_aggregator.event_handler("on_user_turn_idle")
async def on_user_turn_idle(aggregator):
    await idle_handler.handle_idle(aggregator)

@user_aggregator.event_handler("on_user_turn_started")
async def on_user_turn_started(aggregator, strategy):
    idle_handler.reset()  # Reset retry count when user speaks
```

## Best Practices

* **Set appropriate timeouts**: Shorter timeouts (5-10 seconds) work well for voice conversations
* **Use escalating responses**: Start with gentle reminders and gradually become more direct
* **Limit retry attempts**: After 2-3 unsuccessful attempts, consider [ending the conversation](/guides/learn/pipeline-termination) gracefully by pushing an `EndTaskFrame`
* **Reset on user activity**: Use the `on_user_turn_started` event to reset your retry counter when the user speaks
* **Let the LLM respond naturally**: Use system messages to prompt the LLM rather than hardcoded TTS responses for more natural interactions

## Next Steps

<CardGroup cols={2}>
  <Card title="Try the User Idle Example" icon="code" iconType="duotone" href="https://github.com/pipecat-ai/pipecat/blob/main/examples/foundational/17-detect-user-idle.py">
    Explore a complete working example that demonstrates how to detect and
    respond to user inactivity in Pipecat.
  </Card>

  <Card title="Turn Events" icon="book" iconType="duotone" href="/server/utilities/turn-management/turn-events">
    Learn about all available turn events and their parameters.
  </Card>
</CardGroup>

Implementing idle user detection improves the conversational experience by ensuring your bot can handle periods of user inactivity gracefully, either by prompting for re-engagement or politely ending the conversation when appropriate.

---

# Source: https://docs.pipecat.ai/cli/cloud/docker.md

> ## Documentation Index
> Fetch the complete documentation index at: https://docs.pipecat.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# docker

> Build and push Docker images for agent deployments

The `docker` command provides utilities for building, tagging, and pushing Docker images to container registries. This command automatically parses registry information from your deployment configuration and supports both Docker Hub and custom registries.

## build-push

Build, tag, and push a Docker image for your agent deployment. This command reads configuration from your `pcc-deploy.toml` file to automatically determine registry settings, image names, and versions.

**Usage:**

```shell  theme={null}
pipecat cloud docker build-push [ARGS] [OPTIONS]
```

**Arguments:**

<ParamField path="agent-name" type="string">
  Name of the agent to build image for. If not provided, uses the `agent_name`
  from your `pcc-deploy.toml` file.
</ParamField>

**Registry Configuration:**

<ParamField path="--registry / -r" type="string">
  Registry type to push to. Supported values: `dockerhub`, `custom`. When not
  specified, automatically detected from the `image` field in your
  `pcc-deploy.toml` file.
</ParamField>

<ParamField path="--username / -u" type="string">
  Registry username for authentication. When not specified, automatically parsed
  from the `image` field in your `pcc-deploy.toml` file (e.g., `myusername` from
  `myusername/app:1.0`).
</ParamField>

<ParamField path="--registry-url" type="string">
  Custom registry URL (required for custom registries). When not specified,
  automatically parsed from the `image` field for custom registries (e.g.,
  `gcr.io` from `gcr.io/project/app:1.0`).
</ParamField>

**Build Configuration:**

<ParamField path="--version / -v" type="string">
  Version tag for the image. When not specified, automatically extracted from
  the `image` field in your `pcc-deploy.toml` file (e.g., `1.0` from
  `myusername/app:1.0`).
</ParamField>

<ParamField path="--no-push" type="boolean" default="false">
  Build and tag only, do not push to registry. Useful for local testing or when
  you want to push manually later.
</ParamField>

<ParamField path="--no-latest" type="boolean" default="false">
  Do not tag the image as `latest`. By default, images are tagged with both the
  specified version and `latest`.
</ParamField>

## Configuration

The `docker build-push` command reads configuration from your `pcc-deploy.toml` file to minimize required command-line arguments. Here's how different registry setups work:

### Docker Hub (Default)

For Docker Hub repositories, the minimal configuration is:

```toml  theme={null}
agent_name = "my-agent"
image = "myusername/my-agent:1.0"
```

This automatically configures:

* Registry: `dockerhub`
* Username: `myusername`
* Agent name: `my-agent`
* Version: `1.0`

### Custom Registry

For custom registries like Google Container Registry, AWS ECR, or private registries:

```toml  theme={null}
agent_name = "my-agent"
image = "gcr.io/my-project/my-agent:1.0"
```

This automatically configures:

* Registry: `custom`
* Registry URL: `gcr.io`
* Username/Project: `my-project`
* Agent name: `my-agent`
* Version: `1.0`

### Docker Configuration Section

For advanced configuration, add a `[docker]` section:

```toml  theme={null}
agent_name = "my-agent"
image = "myusername/my-agent:1.0"

[docker]
auto_latest = false  # Don't tag as 'latest'
```

Available `[docker]` options:

<ParamField path="auto_latest" type="boolean" default="true">
  Whether to automatically tag the image as `latest` in addition to the
  specified version.
</ParamField>

## Examples

### Basic Usage (Recommended)

With a properly configured `pcc-deploy.toml`:

```shell  theme={null}
# Build and push using all configuration from pcc-deploy.toml
pipecat cloud docker build-push
```

### Override Version

```shell  theme={null}
# Use a different version than what's in pcc-deploy.toml
pipecat cloud docker build-push --version 2.0
```

### Build Only

```shell  theme={null}
# Build and tag locally without pushing
pipecat cloud docker build-push --no-push
```

### Different Registry

```shell  theme={null}
# Override registry settings for one-time builds
pipecat cloud docker build-push --registry custom --registry-url my-registry.com --username myuser
```

### Skip Latest Tag

```shell  theme={null}
# Only tag with the specific version, not 'latest'
pipecat cloud docker build-push --no-latest
```

## Platform Support

All images are built for the `linux/arm64` platform, which is required for Pipecat Cloud deployments. This is automatically configured and cannot be changed.

## Error Handling

The command provides helpful error messages for common issues:

* **Authentication errors**: Suggests the appropriate `docker login` command
* **Missing Dockerfile**: Indicates that a Dockerfile must be present in the current directory
* **Registry access issues**: Provides guidance on checking permissions and authentication

---

# Source: https://docs.pipecat.ai/server/utilities/dtmf-aggregator.md

> ## Documentation Index
> Fetch the complete documentation index at: https://docs.pipecat.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# DTMFAggregator

> Aggregates DTMF (phone keypad) input into meaningful sequences for LLM processing

## Overview

`DTMFAggregator` processes incoming DTMF (Dual-Tone Multi-Frequency) frames from phone keypad input and aggregates them into complete sequences that can be understood by LLM services. It buffers individual digit presses and flushes them as transcription frames when a termination digit is pressed, a timeout occurs, or an interruption happens.

This aggregator is essential for telephony applications where users interact via phone keypad buttons, converting raw DTMF input into structured text that LLMs can process alongside voice transcriptions.

## Constructor

```python  theme={null}
aggregator = DTMFAggregator(
    timeout=2.0,
    termination_digit=KeypadEntry.POUND,
    prefix="DTMF: "
)
```

<ParamField path="timeout" type="float" default="2.0">
  Idle timeout in seconds before flushing the aggregated digits
</ParamField>

<ParamField path="termination_digit" type="KeypadEntry" default="KeypadEntry.POUND">
  Digit that triggers immediate flush of the aggregation
</ParamField>

<ParamField path="prefix" type="str" default="DTMF: ">
  Prefix added to DTMF sequence in the output transcription
</ParamField>

## Input Frames

<ParamField path="InputDTMFFrame" type="Frame">
  Contains a single keypad button press with a KeypadEntry value
</ParamField>

<ParamField path="StartInterruptionFrame" type="Frame">
  Flushes any pending aggregation when user interruption begins
</ParamField>

<ParamField path="EndFrame" type="Frame">
  Flushes pending aggregation and stops the aggregation task
</ParamField>

## Output Frames

<ParamField path="TranscriptionFrame" type="Frame">
  Contains the aggregated DTMF sequence as text with the configured prefix
</ParamField>

All input frames are passed through downstream, including the original `InputDTMFFrame` instances.

## Keypad Entries

The aggregator processes these standard phone keypad entries:

| KeypadEntry           | Value         | Description       |
| --------------------- | ------------- | ----------------- |
| `ZERO` through `NINE` | `"0"` - `"9"` | Numeric digits    |
| `STAR`                | `"*"`         | Star/asterisk key |
| `POUND`               | `"#"`         | Pound/hash key    |

## Aggregation Behavior

The aggregator flushes (emits a TranscriptionFrame) when:

1. **Termination digit**: The configured termination digit is pressed (default: `#`)
2. **Timeout**: No new digits received within the timeout period (default: 2 seconds)
3. **Interruption**: A `StartInterruptionFrame` is received
4. **Pipeline end**: An `EndFrame` is received

## Usage Examples

### Basic Telephony Integration

```python  theme={null}
from pipecat.processors.aggregators.dtmf_aggregator import DTMFAggregator
from pipecat.serializers.twilio import TwilioFrameSerializer

# Create DTMF aggregator with default settings
dtmf_aggregator = DTMFAggregator()

# Set up Twilio serializer for phone integration
serializer = TwilioFrameSerializer(
    stream_sid=stream_sid,
    call_sid=call_sid,
    account_sid=os.getenv("TWILIO_ACCOUNT_SID"),
    auth_token=os.getenv("TWILIO_AUTH_TOKEN")
)

# Create pipeline with DTMF processing
pipeline = Pipeline([
    transport.input(),      # Websocket input from Twilio
    dtmf_aggregator,       # Process DTMF before STT
    stt,                   # Speech-to-text service
    context_aggregator.user(),
    llm,                   # LLM processes both voice and DTMF
    tts,                   # Text-to-speech
    transport.output(),
    context_aggregator.assistant(),
])
```

### Custom Configuration for Menu Systems

```python  theme={null}
# Configure for menu system with star termination
menu_dtmf = DTMFAggregator(
    timeout=5.0,                    # Longer timeout for menu selection
    termination_digit=KeypadEntry.STAR,  # Use * to confirm selection
    prefix="Menu selection: "       # Clear prefix for LLM
)

# Update system prompt to handle DTMF input
messages = [
    {
        "role": "system",
        "content": """You are a phone menu assistant.

When you receive input starting with "Menu selection:", this represents
button presses on the phone keypad:
- Single digits (1-9): Menu options
- 0: Often "speak to operator"
- *: Confirmation or "go back"
- #: Usually "repeat menu"

Respond appropriately to both voice and keypad input."""
    }
]
```

## Sequence Examples

| User Input         | Aggregation Trigger    | Output TranscriptionFrame |
| ------------------ | ---------------------- | ------------------------- |
| `1`, `2`, `3`, `#` | Termination digit      | `"DTMF: 123#"`            |
| `*`, `0`           | 2-second timeout       | `"DTMF: *0"`              |
| `5`, interruption  | StartInterruptionFrame | `"DTMF: 5"`               |
| `9`, `9`, EndFrame | Pipeline shutdown      | `"DTMF: 99"`              |

## Frame Flow

```mermaid  theme={null}
graph TD
    A[Twilio WebSocket] --> B[TwilioFrameSerializer]
    B --> C[DTMFAggregator]
    C --> D[STT Service]
    C --> E[TranscriptionFrame]
    E --> F[LLM Context Aggregator]
    D --> F
    F --> G[LLM Service]
```

## Error Handling

The aggregator gracefully handles:

* Invalid DTMF digits (logged and ignored)
* Pipeline interruptions (flushes pending sequences)
* Rapid key presses (buffers efficiently)
* Mixed voice and DTMF input (processes independently)

## Best Practices

1. **System Prompt Design**: Train your LLM to recognize and respond to DTMF prefixed input
2. **Timeout Configuration**: Use shorter timeouts (1-2s) for rapid entry, longer (3-5s) for menu selection
3. **Termination Strategy**: Use `#` for confirmation, `*` for cancel/back operations
4. **Pipeline Placement**: Always place before the user context aggregator to ensure proper frame ordering

---

# Source: https://docs.pipecat.ai/getting-started/ecosystem.md

> ## Documentation Index
> Fetch the complete documentation index at: https://docs.pipecat.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Ecosystem

> Client SDKs, tools, and AI services in the Pipecat ecosystem

Pipecat provides a complete ecosystem of client SDKs, developer tools, and AI service integrations to build voice and multimodal applications for any platform.

## Client SDKs

The client SDKs connect users to your bot via mobile or web applications. They handle real-time audio transport (using Daily WebRTC, SmallWebRTC, etc.) and provide built-in messaging and events to build responsive voice AI interfaces—tracking when your bot is speaking, what it's saying, and more.

Build client applications for any platform with our official SDKs:

<CardGroup cols={3}>
  <Card title="JavaScript" icon="JS" color="#f7e014" href="/client/js/introduction">
    Build web applications with vanilla JavaScript
  </Card>

  <Card title="React" icon="react" color="#56c4db" href="/client/react/introduction">
    Create React applications with hooks and components
  </Card>

  <Card title="React Native" icon="react" color="#56c4db" href="/client/react-native/introduction">
    Build mobile apps with React Native
  </Card>

  <Card title="Swift" icon="swift" color="#F05138" href="/client/ios/introduction">
    Native iOS applications
  </Card>

  <Card title="Kotlin" icon="android" color="#78C257" href="/client/android/introduction">
    Native Android applications
  </Card>

  <Card title="C++" icon="C" color="#679cd3" href="/client/c++/introduction">
    High-performance native applications
  </Card>
</CardGroup>

## Tools & Frameworks

### Pipecat CLI

Command-line tool for scaffolding new projects, deploying to Pipecat Cloud, and monitoring live sessions.

<Card title="CLI Overview" icon="terminal" href="/cli/overview">
  Learn about scaffolding, deployment, and monitoring commands
</Card>

### Pipecat Flows

A conversation and context management library designed to help you handle complex, structured conversations with state management.

<Card title="Pipecat Flows" icon="diagram-project" href="https://github.com/pipecat-ai/pipecat-flows">
  Build structured conversation flows
</Card>

### Voice UI Kit

Pre-built React components and custom hooks that accelerate building voice AI applications on top of the JavaScript and React SDKs.

<Card title="Voice UI Kit" icon="palette" href="https://github.com/pipecat-ai/voice-ui-kit">
  Ready-to-use components for voice interfaces
</Card>

### Tail

A terminal dashboard for monitoring your Pipecat sessions in real-time with logs, conversations, metrics, and audio levels.

<Card title="Tail" icon="chart-line" href="https://github.com/pipecat-ai/tail">
  Real-time monitoring dashboard
</Card>

## AI Services

Pipecat is a modular system where you can plug in different AI services without changing your code. This flexibility is one of Pipecat's core strengths—swap Deepgram for Azure Speech, OpenAI for Anthropic, or Cartesia for ElevenLabs with minimal changes.

The framework integrates with 70+ services across:

* **Speech-to-Text (STT)** - Deepgram, Azure, Google, Whisper, and more
* **Language Models (LLM)** - OpenAI, Anthropic, Gemini, Groq, and more
* **Text-to-Speech (TTS)** - Cartesia, ElevenLabs, Azure, Google, and more
* **Vision** - Image and video analysis capabilities
* **Transports** - WebRTC, WebSockets, telephony providers
* **And more** - Image generation, memory systems, analytics

<Card title="View All Services" icon="grid" href="/server/services/supported-services">
  Browse the complete list of 70+ supported AI services
</Card>

---

# Source: https://docs.pipecat.ai/server/services/tts/elevenlabs.md

# Source: https://docs.pipecat.ai/server/services/stt/elevenlabs.md

> ## Documentation Index
> Fetch the complete documentation index at: https://docs.pipecat.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# ElevenLabs

> Speech-to-text service implementation using ElevenLabs' file-based transcription API

## Overview

`ElevenLabsSTTService` provides speech recognition using ElevenLabs' Speech-to-Text API with segmented audio processing. The service uploads audio files to ElevenLabs and receives transcription results directly, supporting multiple languages and audio event tagging for comprehensive transcription.

<CardGroup cols={2}>
  <Card title="ElevenLabs STT API Reference" icon="code" href="https://reference-server.pipecat.ai/en/latest/api/pipecat.services.elevenlabs.stt.html">
    Pipecat's API methods for ElevenLabs STT integration
  </Card>

  <Card title="Example Implementation" icon="play" href="https://github.com/pipecat-ai/pipecat/blob/main/examples/foundational/07d-interruptible-elevenlabs-http.py">
    Complete example with ElevenLabs STT and TTS
  </Card>

  <Card title="ElevenLabs Documentation" icon="book" href="https://elevenlabs.io/docs/api-reference/speech-to-text/get">
    Official ElevenLabs STT API documentation
  </Card>

  <Card title="ElevenLabs Platform" icon="microphone" href="https://elevenlabs.io/">
    Access API keys and speech-to-text models
  </Card>
</CardGroup>

## Installation

To use ElevenLabs STT services, install the required dependencies:

```bash  theme={null}
pip install "pipecat-ai[elevenlabs]"
```

## Prerequisites

### ElevenLabs Account Setup

Before using ElevenLabs STT services, you need:

1. **ElevenLabs Account**: Sign up at [ElevenLabs Platform](https://elevenlabs.io/)
2. **API Key**: Generate an API key from your account dashboard
3. **Model Access**: Ensure access to the Scribe v1 transcription model
4. **HTTP Session**: Configure aiohttp session for file uploads

### Required Environment Variables

* `ELEVENLABS_API_KEY`: Your ElevenLabs API key for authentication

---

# Source: https://docs.pipecat.ai/deployment/pipecat-cloud/fundamentals/error-codes.md

> ## Documentation Index
> Fetch the complete documentation index at: https://docs.pipecat.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Troubleshooting

> Common issues and solutions when deploying agents to Pipecat Cloud

## Error codes

### PCC-1002

> Attempt to start agent without public api key.

Ensure you have provided a valid public API key when starting an agent. [Learn more](./accounts-and-organizations#api-keys) about creating and managing API keys.

### PCC-1004

> Billing credentials not set. Please set billing credentials via the Pipecat Cloud dashboard.

In order to deploy or start an agent on Pipecat Cloud, you must have valid billing credentials set up on the namespace or organization you have specified. You can set up these up via the [Pipecat Cloud dashboard](https://pipecat.daily.co/).

### PCC\_IMAGE\_NOT\_FOUND

> The provided image address or tag could not be found.

Check your provided image, tag and host address to ensure they are correct.

### PCC\_IMAGE\_PULL\_UNAUTHORIZED

> Unauthorized to pull image at given address.

If you are using a private repository, ensure you have provided valid credentials to access the repository via [image pull secrets](./secrets#image-pull-secrets).

### PCC\_IMAGE\_TOO\_LARGE

> Image is too large.

Ensure the image you are attempting to pull is not too large. Currently, Pipecat Cloud has no maximum image size limit, but excessively large images may cause issues when deploying agents.

### PCC\_IMAGE\_PULL\_RATE\_LIMIT

> Remote repository rate limited the pull request.

Check that your image repository is not rate limiting your pull requests. This can be caused by excessive requests to the repository, or by the repository owner setting rate limits.

### PCC\_INVALID\_IMAGE\_PLATFORM

> Image platform is not supported.

Ensure the image you are attempting to pull is supported by Pipecat Cloud. Currently, Pipecat Cloud only supports images for the `linux/arm64` platform.

### PCC\_IMAGE\_PULL\_UNKNOWN\_ERROR

> Unknown error occurred when attempting to pull the image.

If you receive this error, please contact support for further assistance.

### PCC\_UNKNOWN\_ERROR

> Unknown error occurred when attempting to start provided image.

Indicates an error in your image that prevent your agent from starting. Typically `pipecat cloud agent logs [AGENT_NAME]` will provide more information on the error.

### PCC-AGENT-AT-CAPACITY

> Service maximum agent instances reached

This error indicates that the maximum active sessions for your deployment have been reached. You can either increase the maximum active sessions for your deployment (`max-agents`), or place the end-user in a queue until a session becomes available.

## Enabling extra logs

If you wish to print a summary of the features available in the base image, set the environment variable `PCC_LOG_FEATURES_SUMMARY` to `true`.
This will print all the features enabled in the base image that you can use. For example:

```
2025-10-22 22:04:41.411 | INFO     | feature_manager:log_features_summary:287 - ============================================================
2025-10-22 22:04:41.411 | INFO     | feature_manager:log_features_summary:288 - Features available in this base image:
2025-10-22 22:04:41.412 | INFO     | feature_manager:log_features_summary:289 - ============================================================
2025-10-22 22:04:41.412 | INFO     | feature_manager:log_features_summary:299 - ✅ Daily Transport: ENABLED
2025-10-22 22:04:41.412 | INFO     | feature_manager:log_features_summary:299 - ✅ Websocket Transport: ENABLED
2025-10-22 22:04:41.412 | INFO     | feature_manager:log_features_summary:299 - ✅ SmallWebRTC Session Arguments: ENABLED
2025-10-22 22:04:41.412 | INFO     | feature_manager:log_features_summary:299 - ✅ SmallWebRTC Transport: ENABLED
2025-10-22 22:04:41.412 | INFO     | feature_manager:log_features_summary:299 - ✅ SmallWebRTC ICE Candidates: ENABLED
2025-10-22 22:04:41.412 | INFO     | feature_manager:log_features_summary:299 - ✅ WhatsApp Integration: ENABLED
2025-10-22 22:04:41.412 | INFO     | feature_manager:log_features_summary:314 - ============================================================
```

---

# Source: https://docs.pipecat.ai/client/js/api-reference/errors.md

> ## Documentation Index
> Fetch the complete documentation index at: https://docs.pipecat.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Errors

## RTVIError Type

Base `PipecatClient` error type, extends [`Error`](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Error) and primarily introduces the `status` field. Most methods will try to throw an error of this type when something goes wrong. This is different from the RTVI error event and its corresponding [`onError` callback](./callbacks#param-on-error), which are used for communicating errors that are sent by the bot.

<ParamField path="status" type="number">
  A unique identifier (or HTTP code if applicable) for the error.
</ParamField>

<ParamField path="message" type="string">
  A human-readable message describing the error.
</ParamField>

## Pre-defined RTVIErrors

### ConnectionTimeoutError

Emitted when the bot does not enter a ready state within the specified timeout period during the `connect()` method call.

### StartBotError

Emitted from `startBot()` or `startBotAndConnect()` when the endpoint responds with an error or the `fetch` itself fails. This may be due to the endpoint being unavailable, or the server failing to parse the provided data.

<ParamField path="error" type="string" required>
  All `StartBotError` instances will have an `error` field set to `invalid-request-error`.
</ParamField>

<ParamField path="status" type="number">
  HTTP status code returned by the endpoint, if applicable.
</ParamField>

<ParamField path="message" type="string">
  Verbose error message returned by the endpoint, if provided. To take advantage of this, the endpoint should return an error response with a JSON object with an `info` field containing the error message.
</ParamField>

### TransportStartError

Emitted when the Transport is not able to connect. You may need to check the connection parameters provided or returned from you endpoint.

### BotNotReadyError

Emitted when the client attempts to perform an action or method that requires the bot to be in a ready state, but the bot is not ready. You must call `connect()` first and wait for the bot to be ready before performing such actions.

### DeviceError

Emitted when there is an issue with acquiring or using a media device, such as a camera or microphone. This could be due to permissions issues, device unavailability, or other related problems.

<ParamField path="devices" type="array<'cam' | 'mic' | 'speaker'>" required>
  List of media devices, `'cam'`, `'mic'`, and/or `'speaker'`, that are unavailable or could not be accessed.
</ParamField>

<ParamField path="type" type="string" required>
  The `type` field will indicate what type of device error occurred. Options include:

  * `"in-use"`: A device is currently in use by another application and cannot be accessed. *windows only*
  * `"permissions"`: The user has not granted permission to access the media device.
  * `"undefined-mediadevices"`: `getUserMedia()` is not an available API on the current platform or browser.
  * `"not-found"`: The specified media device could not be found.
  * `"constraints"`: The media device could not be configured with the specified constraints.
  * `"unknown"`: An unknown error occurred while accessing the media device.
</ParamField>

<ParamField path="details" type="object">
  Additional details about the device error, if available.
</ParamField>

### UnsupportedFeatureError

Not all Transports are created equal, and some may not support certain features. This error is thrown when a feature is requested that the current Transport does not support.

<ParamField path="feature" type="string" required>
  This custom field will contain the name of the unsupported feature.
</ParamField>

---

# Source: https://docs.pipecat.ai/examples.md

# Source: https://docs.pipecat.ai/deployment/pipecat-cloud/sdk-reference/examples.md

> ## Documentation Index
> Fetch the complete documentation index at: https://docs.pipecat.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Examples

> Common usage patterns for the Pipecat Cloud Python SDK

This page provides examples of common tasks with the Pipecat Cloud Python SDK.

## Starting an Agent Session

This example shows how to start a session with various configurations:

```python  theme={null}
import asyncio

from pipecatcloud.exception import AgentStartError
from pipecatcloud.session import Session, SessionParams


async def main():
    try:
        # Create session object
        session = Session(
            agent_name="my-first-agent",
            api_key=API_KEY,  # Replace with your actual API key
            params=SessionParams(
                use_daily=True,  # Optional: Creates a Daily room
                daily_room_properties={"start_video_off": False},
                data={"key": "value"},
            ),
        )

        # Start the session
        response = await session.start()

        # Get Daily room URL
        daily_url = f"{response['dailyRoom']}?t={response['dailyToken']}"
        print(f"Join Daily room: {daily_url}")

    except AgentStartError as e:
        print(f"Error starting agent: {e}")
    except Exception as e:
        print(f"Unexpected error: {e}")


# Run the async function
if __name__ == "__main__":
    asyncio.run(main())
```

## Building a Bot Entry Point with Daily Arguments

```python  theme={null}
from loguru import logger
from pipecat.runner.types import DailyRunnerArguments

async def bot(args: DailyRunnerArguments):
    """Main bot entry point compatible with the FastAPI route handler.

    Args:
        room_url: The Daily room URL
        token: The Daily room token
        body: The configuration object from the request body
    """
    logger.info(f"Bot process initialized {args.room_url} {args.token}")

    try:
        await main(args.room_url, args.token)
        logger.info("Bot process completed")
    except Exception as e:
        logger.exception(f"Error in bot process: {str(e)}")
        raise
```

## Building a Bot Entry Point with WebSocket Arguments

```python  theme={null}
from loguru import logger
from pipecat.runner.types import WebSocketRunnerArguments

async def bot(args: WebSocketRunnerArguments):
    """Main bot entry point for WebSocket connections.

    Args:
        ws: The WebSocket connection
    """
    logger.info("WebSocket bot process initialized")

    try:
        await main(args.websocket)
        logger.info("WebSocket bot process completed")
    except Exception as e:
        logger.exception(f"Error in WebSocket bot process: {str(e)}")
        raise
```

---

# Source: https://docs.pipecat.ai/deployment/pipecat-cloud/sdk-reference/exceptions.md

> ## Documentation Index
> Fetch the complete documentation index at: https://docs.pipecat.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Error Handling

> Managing errors with the Pipecat Cloud Python SDK

The Pipecat Cloud SDK defines several exception classes to help you handle different error conditions.

## Base Exception Class

### Error

Base class for all Pipecat Cloud exceptions.

```python  theme={null}
from pipecatcloud.exception import Error

try:
    # SDK operation
except Error as e:
    print(f"A Pipecat Cloud error occurred: {e}")
```

## Session Errors

### AgentStartError

Raised when an agent fails to start.

```python  theme={null}
from pipecatcloud.exception import AgentStartError

try:
    response = await session.start()
except AgentStartError as e:
    print(f"Failed to start agent: {e}")
    if e.error_code == "429":
        print("Agent pool at capacity. Try again later.")
    elif e.error_code == "404":
        print("Agent not found.")
```

#### Properties

<ParamField path="message" type="string">
  Error message with details about the failure.
</ParamField>

<ParamField path="error_code" type="string">
  Error code that can be used for conditional handling of different error types.
</ParamField>

### AgentNotHealthyError

Raised when attempting to interact with an agent that is not in a ready state.

```python  theme={null}
from pipecatcloud.exception import AgentNotHealthyError

try:
    response = await session.start()
except AgentNotHealthyError as e:
    print(f"Agent is not ready: {e}")
    print("Check agent status with: pipecat cloud agent status my-agent")
```

#### Properties

<ParamField path="message" type="string">
  Error message with details about the agent's status.
</ParamField>

<ParamField path="error_code" type="string">
  Error code identifying the specific issue with the agent.
</ParamField>

## Authentication Errors

### AuthError

Raised when authentication fails or token has expired.

```python  theme={null}
from pipecatcloud.exception import AuthError

try:
    # Operation requiring authentication
except AuthError:
    print("Your session has expired. Please log in again.")
    # Prompt user to reauthenticate
```

#### Properties

<ParamField path="message" type="string" default="'Unauthorized / token expired. Please run `pipecat cloud auth login` to login again.'">
  Message explaining the authentication failure.
</ParamField>

## Configuration Errors

### ConfigError

Raised when there are issues with configuration storage or retrieval.

```python  theme={null}
from pipecatcloud.exception import ConfigError

try:
    # Operation requiring config
except ConfigError as e:
    print(f"Configuration error: {e.message}")
    # Guide user to fix configuration
```

#### Properties

<ParamField path="message" type="string" default="'Failed to update configuration'">
  Message explaining the configuration issue.
</ParamField>

### ConfigFileError

Raised when the configuration file is malformed.

```python  theme={null}
from pipecatcloud.exception import ConfigFileError

try:
    # Operation requiring config file
except ConfigFileError:
    print("Your configuration file is invalid or corrupted.")
    print("Try recreating it with: pipecat cloud auth login")
```

### InvalidError

Raised when an invalid operation is attempted.

```python  theme={null}
from pipecatcloud.exception import InvalidError

try:
    # Potentially invalid operation
except InvalidError as e:
    print(f"Invalid operation: {e}")
```

---

# Source: https://docs.pipecat.ai/deployment/pipecat-cloud/guides/telephony/exotel-websocket.md

> ## Documentation Index
> Fetch the complete documentation index at: https://docs.pipecat.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Exotel Websocket Transport

> Using Exotel's WebSocket Transport for your Pipecat Cloud agents

<Note>
  This guide covers Pipecat Cloud-specific configuration for Exotel WebSocket
  integration. For a complete guide including dial-in, dial-out, and advanced
  features, see the [Exotel WebSocket Integration
  guide](/guides/telephony/exotel-websockets).
</Note>

Native support for Exotel's WebSocket Transport with Pipecat Cloud allows you to connect your AI agents with Exotel's voice infrastructure. This integration enables your Pipecat bots to handle real phone calls using Exotel's Voice Streaming.

## How It Works

Pipecat Cloud implements Exotel's bidirectional voice streaming protocol. While audio streams flow through WebSockets, the call session is controlled through Exotel's App Bazaar configuration and the Voicebot Applet.

When Pipecat Cloud receives an incoming WebSocket connection from Exotel, it processes the `Connected` and `Start` messages to initialize a new bot instance. All WebSocket messages are forwarded to your bot, including any custom parameters set in your App Bazaar configuration. This allows your bot to leverage Exotel's API for advanced call control - such as recording conversations, transferring to human agents, or implementing complex call flows.

## Prerequisites

Before setting up this integration, ensure you have:

* An Exotel account with voice streaming enabled
* A Pipecat Cloud account with an Exotel Websockets-compatible bot

A ready-to-build example of an Exotel websockets bot with complete source code is available in the [pipecat-examples repo](https://github.com/pipecat-ai/pipecat-examples/tree/main/exotel-chatbot).

## Pipecat Cloud Configuration

### 1. Purchase and Configure Phone Number

[Purchase a phone number from Exotel](https://my.exotel.com/) if you haven't already:

* Navigate to ExoPhones and purchase a number
* Complete KYC verification if required

### 2. Enable Voice Streaming

Voice streaming may not be enabled by default on all accounts:

* Contact Exotel support at `hello@exotel.com`
* Request: "Enable Voicebot Applet for voice streaming for account \[Your Account SID]"
* Include your use case: "AI voice bot integration"

### 3. Get Your Organization Name

Retrieve your Pipecat Cloud organization name using the Pipecat CLI:

```bash  theme={null}
$ pipecat cloud organizations list
```

This command will output a list of organizations associated with your account. For example:

```bash  theme={null}
Organization        Name
──────────────────────────────────────
Default Workspace   three-random-words-randomnumber (active)
```

### 4. Create App Bazaar Flow

1. Navigate to App Bazaar in your Exotel dashboard
2. Click "Create" to create a new app
3. Build your call flow:

**Add Voicebot Applet:**

* Drag the "Voicebot" applet to your call flow (not "Stream" or "Passthru")
* Configure the WebSocket URL:

```
wss://api.pipecat.daily.co/ws/exotel?serviceHost=AGENT_NAME.ORGANIZATION_NAME
```

Replace the placeholder values:

* `AGENT_NAME` with your deployed bot's name (e.g., `customer-support`)
* `ORGANIZATION_NAME` with your organization name from step 3 (e.g., `three-random-words-randomnumber`)

For example: `wss://api.pipecat.daily.co/ws/exotel?serviceHost=customer-support.industrious-purple-cat-12345`

<Note>
  **Using Regional Endpoints**

  If you deployed your agent to a specific region, use the regional WebSocket endpoint:

  `wss://{region}.api.pipecat.daily.co/ws/exotel?serviceHost=AGENT_NAME.ORGANIZATION_NAME`

  For example, for Europe: `wss://eu-central.api.pipecat.daily.co/ws/exotel?serviceHost=my-agent.my-org`

  Learn more about [regional endpoints](/deployment/pipecat-cloud/guides/regions#regional-websocket-endpoints).
</Note>

**Add Hangup Applet:**

* Drag a "Hangup" applet at the end to properly terminate calls

Your final flow should look like:

```
Call Start → [Voicebot Applet] → [Hangup Applet]
```

### 5. Link Phone Number to App

1. Navigate to "ExoPhones" in your dashboard
2. Find your purchased number and click the edit/pencil icon
3. Under "App", select the custom app you created
4. Save the configuration

## Testing Your Integration

To test your integration, simply dial your Exotel phone number from any phone. The call will connect to your Pipecat Cloud bot, which will respond according to your bot's configuration.

## Custom Data Limitations

<Warning>
  Currently, there are limitations with passing custom data through query
  parameters in Exotel's WebSocket URLs. While the platform theoretically
  supports query parameters, they may be stripped during call processing.
</Warning>

## Next Steps

For complete implementation details including dial-out, caller personalization, and advanced call control features, see the [Exotel WebSocket Integration guide](/guides/telephony/exotel-websockets).

---

# Source: https://docs.pipecat.ai/guides/telephony/exotel-websockets.md

> ## Documentation Index
> Fetch the complete documentation index at: https://docs.pipecat.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Exotel WebSocket Integration

> Complete guide to using Exotel Voice Streaming with Pipecat for dial-in and dial-out functionality

## Things you'll need

* An active [Exotel](https://exotel.com) account with voice streaming enabled
* One or more Exotel provisioned phone numbers
* A public-facing server or tunneling service like [ngrok](https://ngrok.com/) (for dial-out only)
* API keys for speech-to-text, text-to-speech, and LLM services

<CardGroup cols={2}>
  <Card title="Exotel Dial-in Example" icon="phone-arrow-down-left" href="https://github.com/pipecat-ai/pipecat-examples/tree/main/exotel-chatbot/inbound">
    Complete dial-in implementation using Exotel Voice Streaming over WebSocket
  </Card>

  <Card title="Exotel Dial-out Example" icon="phone-arrow-up-right" href="https://github.com/pipecat-ai/pipecat-examples/tree/main/exotel-chatbot/outbound">
    Outbound calling using Exotel Voice Streaming with WebSocket transport
  </Card>
</CardGroup>

## Phone Number Setup

You'll need Exotel phone numbers for both dial-in and dial-out functionality:

* Visit [exotel.com](https://exotel.com) and purchase phone numbers
* Complete KYC verification if required
* Ensure voice streaming is enabled on your account (contact Exotel support if needed)

## Environment Setup

Configure your environment variables for Exotel and AI services:

```shell .env theme={null}
EXOTEL_ACCOUNT_SID=...
EXOTEL_API_KEY=...
EXOTEL_API_TOKEN=...
OPENAI_API_KEY=...
DEEPGRAM_API_KEY=...
CARTESIA_API_KEY=...
```

## Dial-in

Dial-in allows users to call your Exotel number and connect to your Pipecat bot via WebSocket Voice Streaming. Unlike other providers, Exotel automatically provides caller information (to/from numbers) in the WebSocket messages, so no custom server is needed for basic dial-in functionality.

### How It Works

Here's the sequence of events when someone calls your Exotel number:

1. **Exotel receives an incoming call** to your phone number
2. **Exotel executes your App Bazaar flow** which includes a Voicebot applet
3. **Exotel opens a WebSocket** to your server with real-time audio and call metadata
4. **Your bot processes the audio** using the Pipecat pipeline
5. **The bot responds with audio** sent back to Exotel over WebSocket
6. **Exotel plays the audio** to the caller in real-time

### Set up your App Bazaar flow

Exotel uses App Bazaar to configure call flows. For dial-in, you create an app with a Voicebot applet that establishes a WebSocket connection directly to your bot:

1. **Navigate to App Bazaar** in your Exotel dashboard
2. **Create a new app** or edit an existing one
3. **Add a Voicebot applet** (not "Stream" or "Passthru")
4. **Configure the WebSocket URL**: `wss://your-server.com/ws`
5. **Add a Hangup applet** at the end to properly terminate calls

Your flow should look like: `Call Start → [Voicebot Applet] → [Hangup Applet]`

### Personalizing Your Bot with Caller Information

Exotel automatically includes caller information (to/from numbers) in the WebSocket messages. You can use the caller's phone number to:

* Look up customer information in your database
* Personalize greetings and responses based on caller identity
* Route calls to different bot behaviors (e.g., VIP handling, support vs sales)
* Implement custom business logic based on the caller

The example bot demonstrates how to extract caller information from Exotel's WebSocket messages and use it to personalize the conversation.

### Configure your Pipecat bot for dial-in

Your bot receives the WebSocket connection and automatically gets call information from Exotel's WebSocket messages. The key components are:

**WebSocket Parsing**: Pipecat's built-in parser extracts call data from Exotel's WebSocket messages:

```python bot.py theme={null}
from pipecat.runner.utils import parse_telephony_websocket

async def run_bot(websocket: WebSocket):
    # Parse Exotel WebSocket data
    transport_type, call_data = await parse_telephony_websocket(websocket)

    # Extract call information (automatically provided by Exotel)
    # Data format: {
    #     "stream_id": str,
    #     "call_id": str,
    #     "account_sid": str,
    #     "from": str,
    #     "to": str,
    # }
    stream_id = call_data["stream_id"]
    call_id = call_data["call_id"]  # Exotel's CallSid
    account_sid = call_data["account_sid"]
    from_number = call_data["from"]  # Caller's number
    to_number = call_data["to"]      # Your Exotel number
```

**Transport Creation**: Configure the WebSocket transport with Exotel serialization:

```python bot.py theme={null}
# Create Exotel serializer with call details
serializer = ExotelFrameSerializer(
    stream_id=stream_id,
    call_id=call_id,
    account_sid=account_sid,
    api_key=os.getenv("EXOTEL_API_KEY"),
    api_token=os.getenv("EXOTEL_API_TOKEN"),
)

# Configure WebSocket transport
transport = FastAPIWebsocketTransport(
    websocket=websocket,
    params=FastAPIWebsocketParams(
        audio_in_enabled=True,
        audio_out_enabled=True,
        add_wav_header=False,
        serializer=serializer,
    ),
)

# Your bot pipeline setup here...
```

<Card title="Complete Bot Implementation" icon="code" href="https://github.com/pipecat-ai/pipecat-examples/blob/main/exotel-chatbot/inbound/bot.py">
  See the full bot.py with WebSocket parsing, transport setup, and pipeline
  configuration
</Card>

### Set up Exotel phone number

Configure your Exotel phone number to use your App Bazaar flow:

1. Go to the [Exotel Dashboard](https://my.exotel.com)
2. Navigate to ExoPhones
3. Find your phone number and click the edit icon
4. Under "App", select your App Bazaar flow
5. Save the configuration

### Run the Example

For dial-in, you can run your bot directly without a separate server:

```shell  theme={null}
# Start your bot directly
python bot.py

# For local development, use ngrok to expose your WebSocket
ngrok http 8000
# Use the ngrok WebSocket URL (wss://abc123.ngrok.io/ws) in your App Bazaar flow
```

<Card title="Complete Setup Instructions" icon="book-open" href="https://github.com/pipecat-ai/pipecat-examples/tree/main/exotel-chatbot/inbound">
  See the full README with step-by-step setup, App Bazaar configuration, and
  testing
</Card>

## Dial-out

Dial-out allows your bot to initiate calls to phone numbers using Exotel's outbound calling capabilities with WebSocket Voice Streaming.

### How It Works

Here's the sequence of events for dial-out calls:

1. **Your application triggers a dial-out** (via API call or user action)
2. **Server initiates an Exotel call** using the Voice API
3. **Exotel establishes the call** and opens a WebSocket connection
4. **Your bot joins the WebSocket** and sets up the pipeline
5. **The recipient answers** and is connected to your bot
6. **The bot handles the conversation** with real-time audio streaming

### Set up your server for dial-out

The dial-out server creates outbound calls and manages WebSocket connections. When you trigger a dial-out call, your server:

1. **Receives the dial-out request** with target phone number and parameters
2. **Creates an Exotel call** using the Voice API with a callback URL
3. **Accepts the WebSocket** connection from Exotel
4. **Parses call data** from the WebSocket messages
5. **Runs the Pipecat bot** with the call information

<Card title="Complete Server Implementation" icon="code" href="https://github.com/pipecat-ai/pipecat-examples/blob/main/exotel-chatbot/outbound/server.py">
  See the full FastAPI server code with outbound call creation and WebSocket
  handling
</Card>

### Configure your Pipecat bot for dial-out

The dial-out bot configuration is similar to dial-in, with Exotel automatically providing call information.

**Call Information**: Extract call details from Exotel's WebSocket messages:

```python bot.py theme={null}
# Parse WebSocket data (same as dial-in)
transport_type, call_data = await parse_telephony_websocket(websocket)

# Extract call information
stream_id = call_data["stream_id"]
call_id = call_data["call_id"]
account_sid = call_data["account_sid"]
from_number = call_data["from"]  # Your Exotel number
to_number = call_data["to"]      # Target number you're calling

# Customize bot behavior for outbound calls
greeting = f"Hi! This is an automated call from {from_number}. How are you today?"
```

**Custom Data Limitations for Dial-out**:

<Warning>
  Currently, there are limitations with passing custom data through query
  parameters in Exotel's WebSocket URLs. While the platform theoretically
  supports query parameters, they may be stripped during call processing. This
  is an ongoing limitation that may be addressed in future Exotel updates.
</Warning>

If you need to pass custom data for dial-out, consider these alternatives:

* Use different WebSocket endpoints for different call types
* Configure multiple App Bazaar flows for different scenarios
* Handle customization based on the called number (available in WebSocket messages)

<Card title="Complete Bot Implementation" icon="code" href="https://github.com/pipecat-ai/pipecat-examples/blob/main/exotel-chatbot/outbound/bot.py">
  See the full bot.py with outbound call handling and call information usage
</Card>

### Run the Example

To test dial-out functionality:

1. **Start your server**: Run your FastAPI server
2. **Trigger a call**: Make an API request to start an outbound call
3. **Answer your phone**: The bot will call the specified number
4. **Talk to your bot**: Have a conversation with your AI agent

<Card title="Complete Setup Instructions" icon="book-open" href="https://github.com/pipecat-ai/pipecat-examples/tree/main/exotel-chatbot/outbound">
  See the full README with step-by-step setup, API usage, and outbound call
  configuration
</Card>

## Key Features

### Audio Format and Sample Rate

Exotel Voice Streaming uses 8kHz mono audio with 16-bit PCM encoding. Configure your pipeline accordingly:

```python  theme={null}
task = PipelineTask(
    pipeline,
    params=PipelineParams(
        audio_in_sample_rate=8000,
        audio_out_sample_rate=8000,
    ),
)
```

### Built-in Call Information

Like Telnyx, Exotel automatically includes comprehensive call information (to/from numbers, account details) in the WebSocket messages, eliminating the need for custom webhook servers in basic dial-in scenarios.

## Deployment

### Local Development

Use [ngrok](https://ngrok.com/) to expose your local server for testing:

```shell  theme={null}
python server.py  # For dial-out
# OR
python bot.py     # For dial-in

ngrok http 8000
# Use the ngrok URL in your App Bazaar Voicebot applet
```

### Pipecat Cloud Deployment

For production deployment without managing your own infrastructure, use Pipecat Cloud:

<Card title="Exotel WebSocket on Pipecat Cloud" icon="cloud" href="/deployment/pipecat-cloud/guides/telephony/exotel-websocket">
  Deploy Exotel WebSocket bots with automatic scaling and managed infrastructure
</Card>

### Self-Hosted Production Deployment

For fully self-hosted production deployment, ensure your servers are:

* Publicly accessible with HTTPS
* Able to handle concurrent WebSocket connections
* Properly configured with Exotel API credentials
* Implementing proper error handling and logging

## Next Steps

* Explore the [complete examples](https://github.com/pipecat-ai/pipecat-examples/tree/main/exotel-chatbot) for full implementations
* Learn about [Daily + Twilio SIP integration](./twilio-daily-sip) for advanced telephony scenarios
* Check out [Daily PSTN integration](./daily-pstn) for direct phone number provisioning

---

# Source: https://docs.pipecat.ai/server/services/serializers/exotel.md

> ## Documentation Index
> Fetch the complete documentation index at: https://docs.pipecat.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# ExotelFrameSerializer

> Serializer for Exotel WebSocket media streaming protocol

## Overview

`ExotelFrameSerializer` enables integration with Exotel's WebSocket media streaming protocol, allowing your Pipecat application to handle phone calls via Exotel's voice services with bidirectional audio conversion and DTMF event handling for Indian telephony infrastructure.

<CardGroup cols={2}>
  <Card title="Exotel Serializer API Reference" icon="code" href="https://reference-server.pipecat.ai/en/latest/api/pipecat.serializers.exotel.html">
    Pipecat's API methods for Exotel WebSocket integration
  </Card>

  <Card title="Example Implementation" icon="play" href="https://github.com/pipecat-ai/pipecat-examples/tree/main/exotel-chatbot">
    Complete telephony examples with Exotel
  </Card>

  <Card title="Exotel Documentation" icon="book" href="https://developer.exotel.com/">
    Official Exotel developer documentation
  </Card>

  <Card title="Exotel Console" icon="microphone" href="https://my.exotel.com/">
    Manage phone numbers and streaming configuration
  </Card>
</CardGroup>

## Installation

The `ExotelFrameSerializer` does not require any additional dependencies beyond the core Pipecat library:

```bash  theme={null}
pip install "pipecat-ai"
```

## Prerequisites

### Exotel Account Setup

Before using ExotelFrameSerializer, you need:

1. **Exotel Account**: Sign up at [Exotel Console](https://my.exotel.com/)
2. **Phone Number**: Purchase an Exotel phone number with voice capabilities
3. **Media Streaming**: Configure your phone number for WebSocket streaming
4. **Webhook Configuration**: Set up webhook endpoints for call handling

### Required Configuration

* **Stream ID**: Provided by Exotel during WebSocket connection
* **Call SID**: Associated Exotel Call SID (optional)

### Key Features

* **Bidirectional Audio**: Convert between Pipecat and Exotel audio formats
* **DTMF Handling**: Process touch-tone events from callers
* **Indian Telephony**: Optimized for Indian voice infrastructure
* **WebSocket Streaming**: Real-time audio streaming via WebSocket protocol

---

# Source: https://docs.pipecat.ai/server/utilities/turn-management/external-turn-management.md

> ## Documentation Index
> Fetch the complete documentation index at: https://docs.pipecat.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# External Turn Management

> Handle turn detection externally using UserTurnProcessor or external services

## Overview

In some scenarios, turn detection happens externally, either through a dedicated processor or an external service. Pipecat provides `ExternalUserTurnStrategies`, a [user turn strategy](/server/utilities/turn-management/user-turn-strategies) that defers turn handling to these external sources.

External turn management might be needed when:

* **Multiple context aggregators**: Parallel pipelines with multiple LLMs need a single, shared source of turn events
* **External services with turn detection**: Services like [Deepgram Flux](/server/services/stt/deepgram) or [Speechmatics](/server/services/stt/speechmatics) provide their own turn detection

In both cases, you need to configure your context aggregators with `ExternalUserTurnStrategies` to defer turn handling to the external source.

## External Services

Some speech-to-text services provide built-in turn detection. When using these services, configure your context aggregator with `ExternalUserTurnStrategies` to let the service handle turn management:

```python  theme={null}
from pipecat.processors.aggregators.llm_response_universal import (
    LLMContextAggregatorPair,
    LLMUserAggregatorParams,
)
from pipecat.turns.user_turn_strategies import ExternalUserTurnStrategies

# Configure aggregator to use external turn strategies
user_aggregator, assistant_aggregator = LLMContextAggregatorPair(
    context,
    user_params=LLMUserAggregatorParams(
        user_turn_strategies=ExternalUserTurnStrategies()
    ),
)
```

## UserTurnProcessor

`UserTurnProcessor` is a frame processor for managing user turn lifecycle when you need a single source of turn events shared across multiple context aggregators. It emits `UserStartedSpeakingFrame` and `UserStoppedSpeakingFrame` frames and handles interruptions.

<Note>
  `UserTurnProcessor` only manages user turn start and end events. It does not
  handle transcription aggregation, that remains the responsibility of the
  context aggregators.
</Note>

### Constructor Parameters

<ParamField path="user_turn_strategies" type="UserTurnStrategies" default="UserTurnStrategies()">
  Configured strategies for starting and stopping user turns. See [User Turn
  Strategies](/server/utilities/turn-management/user-turn-strategies) for
  available options.
</ParamField>

<ParamField path="user_turn_stop_timeout" type="float" default="5.0">
  Timeout in seconds to automatically stop a user turn if no stop strategy
  triggers.
</ParamField>

### Event Handlers

`UserTurnProcessor` provides event handlers for turn lifecycle events:

```python  theme={null}
@user_turn_processor.event_handler("on_user_turn_started")
async def on_user_turn_started(processor, strategy):
    # Called when a user turn starts
    pass

@user_turn_processor.event_handler("on_user_turn_stopped")
async def on_user_turn_stopped(processor, strategy):
    # Called when a user turn stops
    pass

@user_turn_processor.event_handler("on_user_turn_stop_timeout")
async def on_user_turn_stop_timeout(processor):
    # Called if no stop strategy triggers before timeout
    pass
```

### Usage with Parallel Pipelines

When using parallel pipelines with multiple context aggregators, place `UserTurnProcessor` before the parallel pipeline and configure each context aggregator with `ExternalUserTurnStrategies`:

```python  theme={null}
from pipecat.audio.turn.smart_turn.local_smart_turn_v3 import LocalSmartTurnAnalyzerV3
from pipecat.pipeline.parallel_pipeline import ParallelPipeline
from pipecat.pipeline.pipeline import Pipeline
from pipecat.processors.aggregators.llm_context import LLMContext
from pipecat.processors.aggregators.llm_response_universal import (
    LLMContextAggregatorPair,
    LLMUserAggregatorParams,
)
from pipecat.turns.user_stop import TurnAnalyzerUserTurnStopStrategy
from pipecat.turns.user_turn_processor import UserTurnProcessor
from pipecat.turns.user_turn_strategies import ExternalUserTurnStrategies, UserTurnStrategies

# Create the external user turn processor with your preferred strategies
user_turn_processor = UserTurnProcessor(
    user_turn_strategies=UserTurnStrategies(
        stop=[
            TurnAnalyzerUserTurnStopStrategy(
                turn_analyzer=LocalSmartTurnAnalyzerV3()
            )
        ]
    ),
)

# Create contexts for each LLM
openai_context = LLMContext(openai_messages)
groq_context = LLMContext(groq_messages)

# Configure aggregators to use external turn strategies
openai_context_aggregator = LLMContextAggregatorPair(
    openai_context,
    user_params=LLMUserAggregatorParams(
        user_turn_strategies=ExternalUserTurnStrategies()
    ),
)
groq_context_aggregator = LLMContextAggregatorPair(
    groq_context,
    user_params=LLMUserAggregatorParams(
        user_turn_strategies=ExternalUserTurnStrategies()
    ),
)

# Build the pipeline with UserTurnProcessor before the parallel branches
pipeline = Pipeline(
    [
        transport.input(),
        stt,
        user_turn_processor,  # Handles turn management for all branches
        ParallelPipeline(
            [
                openai_context_aggregator.user(),
                openai_llm,
                transport.output(),
                openai_context_aggregator.assistant(),
            ],
            [
                groq_context_aggregator.user(),
                groq_llm,
                groq_context_aggregator.assistant(),
            ],
        ),
    ]
)
```

## Related

* [User Turn Strategies](/server/utilities/turn-management/user-turn-strategies) - Configure turn detection strategies
* [Parallel Pipeline](/server/pipeline/parallel-pipeline) - Run multiple pipeline branches concurrently
* [Turn Events](/server/utilities/turn-management/turn-events) - Handle turn lifecycle events

---

# Source: https://docs.pipecat.ai/server/utilities/smart-turn/fal-smart-turn.md

> ## Documentation Index
> Fetch the complete documentation index at: https://docs.pipecat.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Fal Smart Turn

> Cloud-hosted Smart Turn detection using Fal.ai

<Warning>
  DEPRECATED: `FalSmartTurnAnalyzer` is deprecated. Please use
  [LocalSmartTurnAnalyzerV3](/server/utilities/smart-turn/smart-turn-overview#local-smart-turn)
  instead, which provides fast CPU inference without requiring external API
  calls.
</Warning>

## Overview

`FalSmartTurnAnalyzer` provides an easy way to use Smart Turn detection via Fal.ai's cloud infrastructure. This implementation requires minimal setup - just an API key - and offers scalable inference without having to manage your own servers.

## Installation

```bash  theme={null}
pip install "pipecat-ai[remote-smart-turn]"
```

## Requirements

* A Fal.ai account and API key (get one at [Fal.ai](https://fal.ai))
* Internet connectivity for making API calls

## Configuration

### Constructor Parameters

<ParamField path="api_key" type="Optional[str]" default="None">
  Your Fal.ai API key for authentication (required unless using a custom
  deployment)
</ParamField>

<ParamField path="url" type="str" default="https://fal.run/fal-ai/smart-turn/raw">
  URL endpoint for the Smart Turn API (defaults to the official Fal deployment)
</ParamField>

<ParamField path="aiohttp_session" type="aiohttp.ClientSession" required>
  An aiohttp client session for making HTTP requests
</ParamField>

<ParamField path="sample_rate" type="Optional[int]" default="None">
  Audio sample rate (will be set by the transport if not provided)
</ParamField>

<ParamField path="params" type="SmartTurnParams" default="SmartTurnParams()">
  Configuration parameters for turn detection. See
  [SmartTurnParams](/server/utilities/smart-turn/smart-turn-overview#configuration)
  for details.
</ParamField>

## Example

```python  theme={null}
import os
import aiohttp
from pipecat.audio.turn.smart_turn.fal_smart_turn import FalSmartTurnAnalyzer
from pipecat.audio.vad.silero import SileroVADAnalyzer
from pipecat.audio.vad.vad_analyzer import VADParams
from pipecat.processors.aggregators.llm_response_universal import (
    LLMContextAggregatorPair,
    LLMUserAggregatorParams,
)
from pipecat.transports.base_transport import TransportParams
from pipecat.turns.user_stop import TurnAnalyzerUserTurnStopStrategy
from pipecat.turns.user_turn_strategies import UserTurnStrategies

async def setup_transport():
    async with aiohttp.ClientSession() as session:
        transport = SmallWebRTCTransport(
            webrtc_connection=webrtc_connection,
            params=TransportParams(
                audio_in_enabled=True,
                audio_out_enabled=True,
            ),
        )

        # Configure Smart Turn Detection via user turn strategies
        user_aggregator, assistant_aggregator = LLMContextAggregatorPair(
            context,
            user_params=LLMUserAggregatorParams(
                user_turn_strategies=UserTurnStrategies(
                    stop=[TurnAnalyzerUserTurnStopStrategy(
                        turn_analyzer=FalSmartTurnAnalyzer(
                            api_key=os.getenv("FAL_SMART_TURN_API_KEY"),
                            aiohttp_session=session
                        )
                    )]
                ),
                vad_analyzer=SileroVADAnalyzer(params=VADParams(stop_secs=0.2)),
            ),
        )

        # Continue with pipeline setup...
```

## Custom Deployment

You can also deploy the Smart Turn model yourself on Fal.ai and point to your custom deployment:

```python  theme={null}
TurnAnalyzerUserTurnStopStrategy(
    turn_analyzer=FalSmartTurnAnalyzer(
        url="https://fal.run/your-username/your-deployment/raw",
        api_key=os.getenv("FAL_API_KEY"),
        aiohttp_session=session
    )
)
```

## Performance Considerations

* **Latency**: While Fal provides global infrastructure, there will be network latency compared to local inference
* **Reliability**: Depends on network connectivity and Fal.ai service availability
* **Scalability**: Handles scaling automatically based on your usage

## Notes

* Fal handles the model hosting, scaling, and infrastructure management
* The session timeout is controlled by the `stop_secs` parameter
* For high-throughput applications, consider deploying your own inference service

---

# Source: https://docs.pipecat.ai/server/services/stt/fal.md

# Source: https://docs.pipecat.ai/server/services/image-generation/fal.md

> ## Documentation Index
> Fetch the complete documentation index at: https://docs.pipecat.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# fal

> Image generation service implementation using fal's fast SDXL models

## Overview

`FalImageGenService` provides high-speed image generation capabilities using fal's optimized Stable Diffusion XL models. It supports various image sizes, formats, and generation parameters with a focus on fast inference and low-latency image creation.

<CardGroup cols={2}>
  <Card title="Fal Image Gen API Reference" icon="code" href="https://reference-server.pipecat.ai/en/latest/api/pipecat.services.fal.image.html">
    Pipecat's API methods for fal image generation integration
  </Card>

  <Card title="Example Implementation" icon="play" href="https://github.com/pipecat-ai/pipecat/blob/main/examples/foundational/05-sync-speech-and-image.py">
    Browse examples using fal image generation
  </Card>

  <Card title="fal Documentation" icon="book" href="https://fal.ai/docs">
    Official fal API documentation and model guides
  </Card>

  <Card title="fal Platform" icon="image" href="https://fal.ai/">
    Access fast SDXL models and manage API keys
  </Card>
</CardGroup>

## Installation

To use fal image generation services, install the required dependencies:

```bash  theme={null}
pip install "pipecat-ai[fal]"
```

## Prerequisites

### fal Account Setup

Before using fal image generation services, you need:

1. **fal Account**: Sign up at [fal Platform](https://fal.ai/)
2. **API Key**: Generate an API key from your account dashboard
3. **Model Selection**: Choose from available fast SDXL models
4. **HTTP Session**: Configure aiohttp session for image downloading

### Required Environment Variables

* `FAL_KEY`: Your fal API key for authentication

---

# Source: https://docs.pipecat.ai/server/services/transport/fastapi-websocket.md

> ## Documentation Index
> Fetch the complete documentation index at: https://docs.pipecat.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# FastAPIWebsocketTransport

> WebSocket transport implementation for FastAPI web applications with telephony integration

## Overview

`FastAPIWebsocketTransport` provides WebSocket support for FastAPI web applications, enabling real-time audio communication over WebSocket connections. It's primarily designed for telephony integrations with providers like Twilio, Telnyx, and Plivo, supporting bidirectional audio streams with configurable serializers and voice activity detection.

<Warning>
  FastAPIWebsocketTransport is best suited for telephony applications and server-side WebSocket integrations.

  For general client/server applications, we recommend using WebRTC-based transports for more robust network and media handling.
</Warning>

<CardGroup cols={2}>
  <Card title="FastAPI WebSocket API Reference" icon="code" href="https://reference-server.pipecat.ai/en/latest/api/pipecat.transports.websocket.fastapi.html">
    Pipecat's API methods for FastAPI WebSocket integration
  </Card>

  <Card title="Example Implementation" icon="play" href="https://github.com/pipecat-ai/pipecat-examples/tree/main/twilio-chatbot">
    Complete Twilio telephony integration example
  </Card>

  <Card title="FastAPI Documentation" icon="book" href="https://fastapi.tiangolo.com/advanced/websockets/">
    Official FastAPI WebSocket documentation
  </Card>

  <Card title="Telephony Serializers" icon="settings" href="/server/services/serializers/introduction">
    Learn about supported FrameSerializers for telephony providers
  </Card>
</CardGroup>

## Installation

To use FastAPIWebsocketTransport, install the required dependencies:

```bash  theme={null}
pip install "pipecat-ai[websocket]"
```

## Prerequisites

### FastAPI Application Setup

Before using FastAPIWebsocketTransport, you need:

1. **FastAPI Application**: Set up a FastAPI web application
2. **WebSocket Endpoint**: Configure WebSocket routes for real-time communication
3. **Telephony Provider**: Set up integration with Twilio, Telnyx, or Plivo
4. **Frame Serializers**: Configure appropriate serializers for your telephony provider

### Configuration Options

* **Serializer Selection**: Choose frame serializer based on telephony provider
* **Audio Parameters**: Configure sample rates and audio formats
* **VAD Integration**: Set up voice activity detection for optimal performance
* **Connection Management**: Handle WebSocket lifecycle and reconnections

### Key Features

* **Telephony Integration**: Optimized for Twilio, Telnyx, and Plivo WebSocket streams
* **Frame Serialization**: Built-in support for telephony provider audio formats
* **FastAPI Integration**: Seamless WebSocket handling within FastAPI applications
* **Bidirectional Audio**: Real-time audio streaming in both directions

## Usage

FastAPIWebsocketTransport integrates with your FastAPI application to handle telephony WebSocket connections. It works with telephony frame serializers to process audio streams from phone calls.

See the [complete example](https://github.com/pipecat-ai/pipecat-examples/tree/main/twilio-chatbot) for a full implementation including:

* FastAPI WebSocket endpoint configuration
* Telephony provider integration setup
* Frame serializer configuration
* Audio processing pipeline integration

---

# Source: https://docs.pipecat.ai/server/utilities/turn-management/filter-incomplete-turns.md

> ## Documentation Index
> Fetch the complete documentation index at: https://docs.pipecat.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Filter Incomplete User Turns

> Use LLM-based detection to suppress responses when users are cut off mid-thought

## Overview

Filter Incomplete Turns is an LLM-powered feature that detects when a user's conversational turn was incomplete (they were cut off or need time to think) and suppresses the bot's response accordingly. Instead of responding to partial input, the bot waits for the user to continue, then automatically re-engages if they remain silent.

This creates more natural conversations by:

* Preventing the bot from responding to incomplete thoughts
* Giving users time to finish speaking without interruption
* Automatically prompting users to continue after pauses

## How It Works

When enabled, the LLM outputs a turn completion marker as the first character of every response:

| Marker | Meaning                                              | Bot Behavior                             |
| ------ | ---------------------------------------------------- | ---------------------------------------- |
| `✓`    | **Complete** - User finished their thought           | Respond normally                         |
| `○`    | **Incomplete Short** - User was cut off mid-sentence | Suppress response, wait 5s, then prompt  |
| `◐`    | **Incomplete Long** - User needs time to think       | Suppress response, wait 10s, then prompt |

The system automatically:

1. Injects turn completion instructions into the LLM's system prompt
2. Detects markers in the LLM's streaming response
3. Suppresses bot speech for incomplete turns
4. Starts a timeout based on the incomplete type
5. Re-prompts the LLM when the timeout expires

## Configuration

Enable the feature via `LLMUserAggregatorParams` when creating an `LLMContextAggregatorPair`:

```python  theme={null}
from pipecat.processors.aggregators.llm_context import LLMContext
from pipecat.processors.aggregators.llm_response_universal import (
    LLMContextAggregatorPair,
    LLMUserAggregatorParams,
)

context = LLMContext(messages)
user_aggregator, assistant_aggregator = LLMContextAggregatorPair(
    context,
    user_params=LLMUserAggregatorParams(
        filter_incomplete_user_turns=True,
    ),
)
```

### LLMUserAggregatorParams

<ParamField path="filter_incomplete_user_turns" type="bool" default="False">
  Enable LLM-based turn completion detection. When True, the system
  automatically injects turn completion instructions into the LLM context and
  configures the LLM service to process turn markers.
</ParamField>

<ParamField path="user_turn_completion_config" type="UserTurnCompletionConfig" default="None">
  Optional configuration object for customizing turn completion behavior. If not
  provided, default values are used.
</ParamField>

## UserTurnCompletionConfig

Use `UserTurnCompletionConfig` to customize timeouts, prompts, and instructions:

```python  theme={null}
from pipecat.turns.user_turn_completion_mixin import UserTurnCompletionConfig

user_aggregator, assistant_aggregator = LLMContextAggregatorPair(
    context,
    user_params=LLMUserAggregatorParams(
        filter_incomplete_user_turns=True,
        user_turn_completion_config=UserTurnCompletionConfig(
            incomplete_short_timeout=5.0,
            incomplete_long_timeout=10.0,
            incomplete_short_prompt="Custom prompt for short pauses...",
            incomplete_long_prompt="Custom prompt for long pauses...",
            instructions="Custom turn completion instructions...",
        ),
    ),
)
```

### Parameters

<ParamField path="incomplete_short_timeout" type="float" default="5.0">
  Seconds to wait after detecting `○` (incomplete short) before re-prompting the
  LLM. Use shorter values for more responsive re-engagement.
</ParamField>

<ParamField path="incomplete_long_timeout" type="float" default="10.0">
  Seconds to wait after detecting `◐` (incomplete long) before re-prompting the
  LLM. Use longer values to give users more time to think.
</ParamField>

<ParamField path="incomplete_short_prompt" type="str" default="...">
  System prompt sent to the LLM when the short timeout expires. Should instruct
  the LLM to generate a brief, natural prompt encouraging the user to continue.
</ParamField>

<ParamField path="incomplete_long_prompt" type="str" default="...">
  System prompt sent to the LLM when the long timeout expires. Should instruct
  the LLM to generate a friendly check-in message.
</ParamField>

<ParamField path="instructions" type="str" default="...">
  Complete turn completion instructions appended to the system prompt. Override
  this to customize how the LLM determines turn completeness.
</ParamField>

## Markers Explained

### Complete (✓)

The user has provided enough information for a meaningful response:

```
User: "I'd go to Japan because I love the culture and food."
LLM: "✓ Japan is a wonderful choice! The blend of ancient traditions..."
```

The `✓` marker tells the system to push the response normally. The marker itself is not spoken (marked with `skip_tts`).

### Incomplete Short (○)

The user was cut off mid-sentence and will likely continue soon:

```
User: "I'd go to Japan because I love"
LLM: "○"
```

The `○` marker suppresses the bot's response entirely. After 5 seconds (configurable), the LLM is prompted to re-engage with something like "Go ahead, I'm listening."

### Incomplete Long (◐)

The user needs more time to think or explicitly asked for time:

```
User: "That's a good question. Let me think..."
LLM: "◐"
```

The `◐` marker also suppresses the response, but waits 15 seconds (configurable) before prompting. This handles cases like:

* "Hold on a second"
* "Let me think about that"
* "Hmm, that's interesting..."

## Usage Examples

### Basic Usage

Enable turn completion with default settings:

```python  theme={null}
from pipecat.processors.aggregators.llm_context import LLMContext
from pipecat.processors.aggregators.llm_response_universal import (
    LLMContextAggregatorPair,
    LLMUserAggregatorParams,
)

messages = [
    {
        "role": "system",
        "content": "You are a helpful assistant...",
    }
]

context = LLMContext(messages)
user_aggregator, assistant_aggregator = LLMContextAggregatorPair(
    context,
    user_params=LLMUserAggregatorParams(
        filter_incomplete_user_turns=True,
    ),
)
```

<Note>
  You don't need to modify your system prompt. Turn completion instructions are
  automatically appended when `filter_incomplete_user_turns` is enabled.
</Note>

### Custom Timeouts

Adjust timeouts for your use case:

```python  theme={null}
from pipecat.turns.user_turn_completion_mixin import UserTurnCompletionConfig

user_aggregator, assistant_aggregator = LLMContextAggregatorPair(
    context,
    user_params=LLMUserAggregatorParams(
        filter_incomplete_user_turns=True,
        user_turn_completion_config=UserTurnCompletionConfig(
            incomplete_short_timeout=3.0,  # More responsive
            incomplete_long_timeout=20.0,  # More patient
        ),
    ),
)
```

### Custom Prompts

Customize what the LLM says when re-engaging:

```python  theme={null}
user_aggregator, assistant_aggregator = LLMContextAggregatorPair(
    context,
    user_params=LLMUserAggregatorParams(
        filter_incomplete_user_turns=True,
        user_turn_completion_config=UserTurnCompletionConfig(
            incomplete_short_prompt="""The user paused briefly.
Generate a contextual prompt to encourage them to continue.
Respond with ✓ followed by your message.""",
            incomplete_long_prompt="""The user has been quiet for a while.
Generate a friendly check-in.
Respond with ✓ followed by your message.""",
        ),
    ),
)
```

<Warning>
  Custom prompts must instruct the LLM to respond with `✓` followed by the
  message. This ensures the re-engagement message is spoken normally.
</Warning>

### With Smart Turn Detection

Combine with smart turn detection for better end-of-turn detection:

```python  theme={null}
from pipecat.audio.turn.smart_turn.local_smart_turn_v3 import LocalSmartTurnAnalyzerV3
from pipecat.turns.user_stop import TurnAnalyzerUserTurnStopStrategy
from pipecat.turns.user_turn_strategies import UserTurnStrategies

user_aggregator, assistant_aggregator = LLMContextAggregatorPair(
    context,
    user_params=LLMUserAggregatorParams(
        user_turn_strategies=UserTurnStrategies(
            stop=[
                TurnAnalyzerUserTurnStopStrategy(
                    turn_analyzer=LocalSmartTurnAnalyzerV3()
                )
            ]
        ),
        filter_incomplete_user_turns=True,
    ),
)
```

<Tip>
  Smart turn detection helps determine when the user stops speaking, while turn
  completion filtering determines whether to respond. They work well together
  for natural conversations.
</Tip>

## Transcripts

Turn completion markers are automatically stripped from assistant transcripts emitted via the `on_assistant_turn_stopped` event. Your transcript handlers will receive clean text without markers:

```python  theme={null}
@assistant_aggregator.event_handler("on_assistant_turn_stopped")
async def on_assistant_turn_stopped(aggregator, message: AssistantTurnStoppedMessage):
    # message.content will be "Japan is a wonderful choice!"
    # NOT "✓ Japan is a wonderful choice!"
    print(f"Assistant: {message.content}")
```

## Supported LLM Services

Turn completion detection works with any LLM service that inherits from `LLMService`:

* OpenAI (`OpenAILLMService`)
* Anthropic (`AnthropicLLMService`)
* Google Gemini (`GoogleLLMService`)
* AWS Bedrock (`AWSLLMService`)
* And other compatible services

## Graceful Degradation

If the LLM fails to output a turn marker:

1. The system logs a warning indicating markers were expected but not found
2. The buffered text is pushed normally to avoid losing the response
3. The conversation continues without interruption

This ensures the feature doesn't break conversations if the LLM occasionally disobeys instructions.

## Related

* [User Turn Strategies](/server/utilities/turn-management/user-turn-strategies) - Configure turn detection
* [Smart Turn Detection](/server/utilities/smart-turn/smart-turn-overview) - AI-powered end-of-turn detection
* [Transcriptions](/server/utilities/turn-management/transcriptions) - Working with conversation transcripts

---

# Source: https://docs.pipecat.ai/server/services/llm/fireworks.md

> ## Documentation Index
> Fetch the complete documentation index at: https://docs.pipecat.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Fireworks AI

> LLM service implementation using Fireworks AI's API with OpenAI-compatible interface

## Overview

`FireworksLLMService` provides access to Fireworks AI's language models through an OpenAI-compatible interface. It inherits from `OpenAILLMService` and supports streaming responses, function calling, and context management with optimized inference infrastructure.

<CardGroup cols={2}>
  <Card title="Fireworks LLM API Reference" icon="code" href="https://reference-server.pipecat.ai/en/latest/api/pipecat.services.fireworks.llm.html">
    Pipecat's API methods for Fireworks AI integration
  </Card>

  <Card title="Example Implementation" icon="play" href="https://github.com/pipecat-ai/pipecat/blob/main/examples/foundational/14i-function-calling-fireworks.py">
    Complete example with function calling
  </Card>

  <Card title="Fireworks Documentation" icon="book" href="https://docs.fireworks.ai/api-reference/post-chatcompletions">
    Official Fireworks AI API documentation and features
  </Card>

  <Card title="Fireworks Platform" icon="microphone" href="https://fireworks.ai/">
    Access models and manage API keys
  </Card>
</CardGroup>

## Installation

To use Fireworks AI services, install the required dependency:

```bash  theme={null}
pip install "pipecat-ai[fireworks]"
```

## Prerequisites

### Fireworks AI Account Setup

Before using Fireworks AI LLM services, you need:

1. **Fireworks Account**: Sign up at [Fireworks AI](https://fireworks.ai/)
2. **API Key**: Generate an API key from your account dashboard
3. **Model Selection**: Choose from available open-source and proprietary models

### Required Environment Variables

* `FIREWORKS_API_KEY`: Your Fireworks AI API key for authentication

---

# Source: https://docs.pipecat.ai/server/services/tts/fish.md

> ## Documentation Index
> Fetch the complete documentation index at: https://docs.pipecat.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Fish Audio

> Real-time text-to-speech service using Fish Audio's WebSocket API

## Overview

`FishTTSService` provides real-time text-to-speech synthesis through Fish Audio's WebSocket-based streaming API. The service offers custom voice models, prosody controls, and multiple audio formats optimized for conversational AI applications with low latency.

<CardGroup cols={2}>
  <Card title="Fish TTS API Reference" icon="code" href="https://reference-server.pipecat.ai/en/latest/api/pipecat.services.fish.tts.html">
    Pipecat's API methods for Fish Audio TTS integration
  </Card>

  <Card title="Example Implementation" icon="play" href="https://github.com/pipecat-ai/pipecat/blob/main/examples/foundational/07t-interruptible-fish.py">
    Complete example with custom voice model
  </Card>

  <Card title="Fish Audio Documentation" icon="book" href="https://docs.fish.audio/developer-platform/getting-started/introduction">
    Official Fish Audio documentation
  </Card>

  <Card title="Voice Models" icon="microphone" href="https://fish.audio/app/my-voices/">
    Create and manage custom voice models
  </Card>
</CardGroup>

## Installation

To use Fish Audio services, install the required dependencies:

```bash  theme={null}
pip install "pipecat-ai[fish]"
```

## Prerequisites

### Fish Audio Account Setup

Before using Fish Audio TTS services, you need:

1. **Fish Audio Account**: Sign up at [Fish Audio Console](https://console.fish.audio/)
2. **API Key**: Generate an API key from your account dashboard
3. **Voice Models**: Create or select custom voice models for synthesis

### Required Environment Variables

* `FISH_API_KEY`: Your Fish Audio API key for authentication

---

# Source: https://docs.pipecat.ai/deployment/platforms/fly.md

> ## Documentation Index
> Fetch the complete documentation index at: https://docs.pipecat.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Fly.io

> Deploy Pipecat bots to Fly.io machines

## Project setup

Let's explore how we can use [fly.io](https://fly.io/) to make our app scalable for production by spawning our Pipecat bots on virtual machines with their own resources.

We mentioned [before](../pattern) that you would ideally containerize the `bot_runner.py` web service and the `bot.py` separately. To keep this example simple, we'll use the same container image for both services.

<Frame><img src="https://mintcdn.com/daily/2bYrACcmgvvzC075/deployment/images/deployment-fly.png?fit=max&auto=format&n=2bYrACcmgvvzC075&q=85&s=e71a797a2ade273971d8b29cfa4755fb" alt="Fly.io Pipecat deployment" data-og-width="1148" width="1148" data-og-height="651" height="651" data-path="deployment/images/deployment-fly.png" data-optimize="true" data-opv="3" srcset="https://mintcdn.com/daily/2bYrACcmgvvzC075/deployment/images/deployment-fly.png?w=280&fit=max&auto=format&n=2bYrACcmgvvzC075&q=85&s=6319f66da875dd238cb45d639f7be73d 280w, https://mintcdn.com/daily/2bYrACcmgvvzC075/deployment/images/deployment-fly.png?w=560&fit=max&auto=format&n=2bYrACcmgvvzC075&q=85&s=1a335f6015e200374ab672abcd3f39a4 560w, https://mintcdn.com/daily/2bYrACcmgvvzC075/deployment/images/deployment-fly.png?w=840&fit=max&auto=format&n=2bYrACcmgvvzC075&q=85&s=8d233b6e9476ff30db03e1d774b3e6d0 840w, https://mintcdn.com/daily/2bYrACcmgvvzC075/deployment/images/deployment-fly.png?w=1100&fit=max&auto=format&n=2bYrACcmgvvzC075&q=85&s=fde906e0e1e18a1ffa8f441895efc940 1100w, https://mintcdn.com/daily/2bYrACcmgvvzC075/deployment/images/deployment-fly.png?w=1650&fit=max&auto=format&n=2bYrACcmgvvzC075&q=85&s=4fb4988ce085eeda857e0cfca8d23676 1650w, https://mintcdn.com/daily/2bYrACcmgvvzC075/deployment/images/deployment-fly.png?w=2500&fit=max&auto=format&n=2bYrACcmgvvzC075&q=85&s=570291b24c92483782df502c7f47cf54 2500w" /></Frame>

### Install the Fly CLI

You can find instructions for creating and setting up your fly account [here](https://fly.io/docs/getting-started/).

## Creating the Pipecat project

<Note>
  We have created a template project
  [here](https://github.com/pipecat-ai/pipecat-examples/tree/main/deployment/flyio-example)
  which you can clone. Since we're targeting production use-cases, this example
  uses Daily (WebRTC) as a transport, but you can configure your bot however you
  like.
</Note>

### Adding a fly.toml

Add a `fly.toml` to the root of your project directory. Here is a basic example:

```toml fly.toml theme={null}
app = 'some-unique-app-name'
primary_region = 'sjc'

[build]

[env]
FLY_APP_NAME = 'some-unique-app-name'

[http_service]
internal_port = 7860
force_https = true
auto_stop_machines = true
auto_start_machines = true
min_machines_running = 0
processes = ['app']

[[vm]]
memory = 512
cpu_kind = 'shared'
cpus = 1
```

For apps with lots of users, consider what resources your HTTP service will require to meet load. We'll define our `bot.py` resources later, so you can set and scale these as you like (`fly scale ...`)

### Environment setup

Our bot requires some API keys and configuration, so create a `.env` in your project root:

```shell .env theme={null}
DAILY_API_KEY=
OPENAI_API_KEY=
ELEVENLABS_API_KEY=
ELEVENLABS_VOICE_ID=
FLY_API_KEY=
FLY_APP_NAME=
```

Of course, the exact keys you need will depend on which services you are using within your `bot.py`.

<Warning>
  **Important:** your `FLY_APP_NAME` should match the name of your fly instance,
  such as that declared in your fly.toml.
</Warning>

The `.env` will allow us to test in local development, but is not included in the deployment. You'll need to set them as Fly app secrets, which you can do via the Fly dashboard or cli.

`fly secrets set ...`

## Containerize our app

Our Fly deployment will need a container image; let's create a simple `Dockerfile` in the root of the project:

<CodeGroup>
  ```shell Dockerfile theme={null}
  FROM python:3.11-slim-bookworm

  # Open port 7860 for http service

  ENV FAST_API_PORT=7860
  EXPOSE 7860

  # Install Python dependencies

  COPY \*.py .
  COPY ./requirements.txt requirements.txt
  RUN pip3 install --no-cache-dir --upgrade -r requirements.txt

  # Install models

  RUN python3 install_deps.py

  # Start the FastAPI server

  CMD python3 bot_runner.py --port ${FAST_API_PORT}

  ```

  ```shell .dockerignore theme={null}
  **/.DS_Store
  .env
  .env.*
  fly.toml
  ```
</CodeGroup>

<Note>You can use any base image as long as Python is available</Note>

Our container does the following:

* Opens port `7860` to serve our `bot_runner.py` FastAPI service.
* Downloads the necessary python dependencies.
* Download / cache the model dependencies the `bot.py` requires.
* Runs the `bot_runner.py` and listens for web requests.

### What models are we downloading?

To support voice activity detection, we're using Silero VAD. Whilst the filesize is not huge, having each new machine download the Silero model at runtime will impact bootup time. Instead, we include the model as part of the Docker image so it's cached and available.

You could, of course, also attach a network volume to each instance if you plan to include larger files as part of your deployment and don't want to bloat the size of your image.

## Launching new machines in `bot_runner.py`

When a user starts a session with our Pipecat bot, we want to launch a new machine on fly.io with it's own system resources.

Let's grab the bot\_runner.py from the example repo [here](https://github.com/pipecat-ai/pipecat/blob/jpt/deployment-examples/examples/deployment/flyio-example/bot_runner.py).

This runner differs from others in the Pipecat repo; we've added a new method that sends a REST request to Fly to provision a new machine for the session.

This method is invoked as part of the `/start_bot` endpoint:

```python bot_runner.py theme={null}
FLY_API_HOST = os.getenv("FLY_API_HOST", "https://api.machines.dev/v1")
FLY_APP_NAME = os.getenv("FLY_APP_NAME", "your-fly-app-name")
FLY_API_KEY = os.getenv("FLY_API_KEY", "")
FLY_HEADERS = {
    'Authorization': f"Bearer {FLY_API_KEY}",
    'Content-Type': 'application/json'
}

def spawn_fly_machine(room_url: str, token: str):
    # Use the same image as the bot runner
    res = requests.get(f"{FLY_API_HOST}/apps/{FLY_APP_NAME}/machines", headers=FLY_HEADERS)
    if res.status_code != 200:
        raise Exception(f"Unable to get machine info from Fly: {res.text}")
    image = res.json()[0]['config']['image']

    # Machine configuration
    cmd = f"python3 bot.py -u {room_url} -t {token}"
    cmd = cmd.split()
    worker_props = {
        "config": {
            "image": image,
            "auto_destroy": True,
            "init": {
                "cmd": cmd
            },
            "restart": {
                "policy": "no"
            },
            "guest": {
                "cpu_kind": "shared",
                "cpus": 1,
                "memory_mb": 1024 # Note: 512 is just enough to run VAD, but 1gb is better
            }
        },

    }

    # Spawn a new machine instance
    res = requests.post(
        f"{FLY_API_HOST}/apps/{FLY_APP_NAME}/machines",
        headers=FLY_HEADERS,
        json=worker_props)

    if res.status_code != 200:
        raise Exception(f"Problem starting a bot worker: {res.text}")

    # Wait for the machine to enter the started state
    vm_id = res.json()['id']

    res = requests.get(
        f"{FLY_API_HOST}/apps/{FLY_APP_NAME}/machines/{vm_id}/wait?state=started",
        headers=FLY_HEADERS)

    if res.status_code != 200:
        raise Exception(f"Bot was unable to enter started state: {res.text}")
```

We want to make sure the machine started ok before returning any data to the user. Fly launches machines pretty fast, but will timeout if things take longer than they should. Depending on your transport method, you may want to optimistically return a response to the user, so they can join the room and poll for the status of their bot.

## Launch the Fly project

Getting your bot on Fly is as simple as:

`fly launch` or `fly launch --org orgname` if you're part of a team.

This will step you through some configuration, and build and deploy your Docker image.

Be sure to configure your app secrets with the necessary environment variables once the deployment has complete.

Assuming all goes well, you can update with any changes with `fly deploy`.

### Test it out

Start a new bot instance by sending a `POST` request to `https://your-fly-url.fly.dev/start_bot`. All being well, this will return a room URL and token.

A nice feature of Fly is the ability to monitor your machines (with live logs) via their dashboard:

[https://fly.io/apps/YOUR-APP\_NAME/machines](https://fly.io/apps/YOUR-APP_NAME/machines)

This is really helpful for monitoring the status of your spawned machine, and debugging if things do not work as expected.

<Note>
  This example is configured to expire after 5 minutes. The bot process is also
  configured to exit after the user leaves the room. This is a good way to
  ensure we don't have any hanging VMs, although you'll likely need to configure
  this behaviour this to meet your own needs.

  You'll also notice that we set `restart policy` to `no`. This prevents the machine attempting to restart after the session has concluded and the process exits.
</Note>

***

## Important considerations

This example does little in the way of load balancing or app security. Indeed, a user can spawn a new machine on your account simply by sending a `POST` request to the `bot_runner.py`. Be sure to configure a maximum number of instances, or authenticate requests to avoid costs getting out of control.

We also deployed our `bot.py` on a machine with the same image as our `bot_runner.py`. To optimize container file sizes and increase security, consider individual images that only deploy resources they require.

---

# Source: https://docs.pipecat.ai/server/utilities/filters/frame-filter.md

> ## Documentation Index
> Fetch the complete documentation index at: https://docs.pipecat.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# FrameFilter

> Processor that selectively passes through only specified frame types

## Overview

`FrameFilter` is a processor that filters frames based on their types, only passing through frames that match specified types (plus some system frames like `EndFrame` and `SystemFrame`).

## Constructor Parameters

<ParamField path="types" type="Tuple[Type[Frame], ...]" required>
  Tuple of frame types that should be passed through the filter
</ParamField>

## Functionality

When a frame passes through the filter, it is checked against the provided types. Only frames that match one of the specified types (or are system frames) will be passed downstream. All other frames are dropped.

## Output Frames

The processor always passes through:

* Frames matching any of the specified types
* `EndFrame` and `SystemFrame` instances (always allowed, so as to not block the pipeline)

## Usage Example

```python  theme={null}
from pipecat.frames.frames import TextFrame, AudioRawFrame, Frame
from pipecat.processors.filters import FrameFilter
from typing import Tuple, Type

# Create a filter that only passes TextFrames and AudioRawFrames
text_and_audio_filter = FrameFilter(
    types=(TextFrame, AudioRawFrame)
)

# Add to pipeline
pipeline = Pipeline([
    source,
    text_and_audio_filter,  # Filters out all other frame types
    destination
])
```

## Frame Flow

```mermaid  theme={null}
graph TD
    A[Input Frames] --> B[FrameFilter]
    B --> C{Frame Type Check}
    C -->|Matches Allowed Types| D[Output Frame]
    C -->|System Frame| D
    C -->|Other Frame Types| E[Dropped]
```

## Notes

* Simple but powerful way to restrict which frame types flow through parts of your pipeline
* Always allows system frames to pass through for proper pipeline operation
* Can be used to isolate specific parts of your pipeline from certain frame types
* Efficient implementation with minimal overhead

---

# Source: https://docs.pipecat.ai/guides/learn/function-calling.md

> ## Documentation Index
> Fetch the complete documentation index at: https://docs.pipecat.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Function Calling

> Enable LLMs to interact with external services and APIs in your voice AI pipeline

**Function calling** (also known as tool calling) allows LLMs to request information from external services and APIs during conversations. This extends your voice AI bot's capabilities beyond its training data to access real-time information and perform actions.

## Pipeline Integration

Function calling works seamlessly within your existing pipeline structure. The LLM service handles function calls automatically when they're needed:

```python  theme={null}
pipeline = Pipeline([
    transport.input(),
    stt,
    context_aggregator.user(),     # Collects user transcriptions
    llm,                          # Processes context, calls functions when needed
    tts,
    transport.output(),
    context_aggregator.assistant(), # Collects function results and responses
])
```

**Function call flow:**

1. User asks a question requiring external data
2. LLM recognizes the need and calls appropriate function
3. Your function handler executes and returns results
4. LLM incorporates results into its response
5. Response flows to TTS and user as normal

**Context integration:** Function calls and their results are automatically stored in conversation context by the context aggregators, maintaining complete conversation history.

## Understanding Function Calling

Function calling allows your bot to access real-time data and perform actions that aren't part of its training data.

For example, you could give your bot the ability to:

* Check current weather conditions
* Look up stock prices
* Query a database
* Control smart home devices
* Schedule appointments

Here's how it works:

1. You define functions the LLM can use and register them to the LLM service used in your pipeline
2. When needed, the LLM requests a function call
3. Your application executes any corresponding functions
4. The result is sent back to the LLM
5. The LLM uses this information in its response

## Implementation

### 1. Define Functions

Pipecat provides a standardized `FunctionSchema` that works across all supported LLM providers. This makes it easy to define functions once and use them with any provider.

As a shorthand, you could also bypass specifying a function configuration at all and instead use "direct" functions. Under the hood, these are converted to `FunctionSchema`s.

#### Using the Standard Schema (Recommended)

```python  theme={null}
from pipecat.adapters.schemas.function_schema import FunctionSchema
from pipecat.adapters.schemas.tools_schema import ToolsSchema

# Define a function using the standard schema
weather_function = FunctionSchema(
    name="get_current_weather",
    description="Get the current weather in a location",
    properties={
        "location": {
            "type": "string",
            "description": "The city and state, e.g. San Francisco, CA",
        },
        "format": {
            "type": "string",
            "enum": ["celsius", "fahrenheit"],
            "description": "The temperature unit to use.",
        },
    },
    required=["location", "format"]
)

# Create a tools schema with your functions
tools = ToolsSchema(standard_tools=[weather_function])

# Pass this to your LLM context
context = LLMContext(
    messages=[{"role": "system", "content": "You are a helpful assistant."}],
    tools=tools
)
user_aggregator, assistant_aggregator = LLMContextAggregatorPair(context)
```

The `ToolsSchema` will be automatically converted to the correct format for your LLM provider through adapters.

#### Using Direct Functions (Shorthand)

You can bypass specifying a function configuration (as a `FunctionSchema` or in a provider-specific format) and instead pass the function directly to your `ToolsSchema`. Pipecat will auto-configure the function, gathering relevant metadata from its signature and docstring. Metadata includes:

* name
* description
* properties (including individual property descriptions)
* list of required properties

Note that the function signature is a bit different when using direct functions. The first parameter is `FunctionCallParams`, followed by any others necessary for the function.

```python  theme={null}
from pipecat.adapters.schemas.tools_schema import ToolsSchema
from pipecat.services.llm_service import FunctionCallParams

# Define a direct function
async def get_current_weather(params: FunctionCallParams, location: str, format: str):
    """Get the current weather.

    Args:
        location: The city and state, e.g. "San Francisco, CA".
        format: The temperature unit to use. Must be either "celsius" or "fahrenheit".
    """
    weather_data = {"conditions": "sunny", "temperature": "75"}
    await params.result_callback(weather_data)

# Create a tools schema, passing your function directly to it
tools = ToolsSchema(standard_tools=[get_current_weather])

# Pass this to your LLM context
context = LLMContext(
    messages=[{"role": "system", "content": "You are a helpful assistant."}],
    tools=tools
)
user_aggregator, assistant_aggregator = LLMContextAggregatorPair(context)
```

#### Using Provider-Specific Formats (Alternative)

You can also define functions in the provider-specific format if needed:

<CodeGroup>
  ```python OpenAI theme={null}
  from openai.types.chat import ChatCompletionToolParam

  # OpenAI native format
  tools = [
      ChatCompletionToolParam(
          type="function",
          function={
              "name": "get_current_weather",
              "description": "Get the current weather",
              "parameters": {
                  "type": "object",
                  "properties": {
                      "location": {
                          "type": "string",
                          "description": "The city and state, e.g. San Francisco, CA",
                      },
                      "format": {
                          "type": "string",
                          "enum": ["celsius", "fahrenheit"],
                          "description": "The temperature unit to use.",
                      },
                  },
                  "required": ["location", "format"],
              },
          },
      )
  ]
  ```

  ```python Anthropic theme={null}
  # Anthropic native format
  tools = [
      {
          "name": "get_weather",
          "description": "Get the weather in a given location",
          "input_schema": {
              "type": "object",
              "properties": {
                  "location": {
                      "type": "string",
                      "description": "The city, e.g. San Francisco",
                  },
                  "format": {
                      "type": "string",
                      "enum": ["celsius", "fahrenheit"],
                      "description": "The temperature unit to use.",
                  },
              },
              "required": ["location", "format"],
          },
      }
  ]
  ```

  ```python Gemini theme={null}
  # Gemini native format
  tools = [
      {
          "function_declarations": [
              {
                  "name": "get_weather_current",
                  "description": "Get the current weather",
                  "parameters": {
                      "type": "object",
                      "properties": {
                          "location": {
                              "type": "string",
                              "description": "Location as 'city,state,country'",
                          },
                          "format": {
                              "type": "string",
                              "enum": ["celsius", "fahrenheit"],
                              "description": "The temperature unit to use.",
                          },
                      },
                      "required": ["location", "format"],
                  },
              }
          ]
      }
  ]
  ```
</CodeGroup>

#### Provider-Specific Custom Tools

Some providers support unique tools that don't fit the standard function schema. For these cases, you can add custom tools:

```python  theme={null}
from pipecat.adapters.schemas.tools_schema import AdapterType, ToolsSchema

# Standard functions
weather_function = FunctionSchema(
    name="get_current_weather",
    description="Get the current weather",
    properties={"location": {"type": "string"}},
    required=["location"]
)

# Custom Gemini search tool
gemini_search_tool = {
    "web_search": {
        "description": "Search the web for information"
    }
}

# Create a tools schema with both standard and custom tools
tools = ToolsSchema(
    standard_tools=[weather_function],
    custom_tools={
        AdapterType.GEMINI: [gemini_search_tool]
    }
)
```

<Tip>
  See the provider-specific documentation for details on custom tools and their
  formats.
</Tip>

### 2. Register Function Handlers

Register handlers for your functions using one of these [LLM service methods](https://reference-server.pipecat.ai/en/latest/api/pipecat.services.llm_service.html#llm-service):

* `register_function`
* `register_direct_function`

Which one you use depends on whether your function is a ["direct" function](#using-direct-functions-shorthand).

<CodeGroup>
  ```python Non-Direct Function theme={null}
  from pipecat.services.llm_service import FunctionCallParams

  llm = OpenAILLMService(api_key="your-api-key")

  # Main function handler - called to execute the function
  async def fetch_weather_from_api(params: FunctionCallParams):
      # Fetch weather data from your API
      weather_data = {"conditions": "sunny", "temperature": "75"}
      await params.result_callback(weather_data)

  # Register the function
  llm.register_function(
      "get_current_weather",
      fetch_weather_from_api,
      cancel_on_interruption=True,  # Cancel if user interrupts (default: True)
  )
  ```

  ```python Direct Function theme={null}
  from pipecat.services.llm_service import FunctionCallParams

  llm = OpenAILLMService(api_key="your-api-key")

  # Direct function
  async def get_current_weather(params: FunctionCallParams, location: str, format: str):
      """Get the current weather.

      Args:
          location: The city and state, e.g. "San Francisco, CA".
          format: The temperature unit to use. Must be either "celsius" or "fahrenheit".
      """
      weather_data = {"conditions": "sunny", "temperature": "75"}
      await params.result_callback(weather_data)

  # Register direct function
  llm.register_direct_function(
      get_current_weather,
      cancel_on_interruption=False,  # Don't cancel on interruption
  )
  ```
</CodeGroup>

**Key registration options:**

* **`cancel_on_interruption=True`** (default): Function call is cancelled if user interrupts
* **`cancel_on_interruption=False`**: Function call continues even if user interrupts

Use `cancel_on_interruption=False` for critical operations that should complete even if the user starts speaking. Function calls are async, so you can continue the conversation while the function executes. Once the result returns, the LLM will automatically incorporate it into the conversation context. LLMs vary in terms of how well they incorporate changes to *previous* messages, so you may need to experiment with your LLM provider to see how it handles this.

### 3. Create the Pipeline

Include your LLM service in your pipeline with the registered functions:

```python  theme={null}
# Initialize the LLM context with your function schemas
context = LLMContext(
    messages=[{"role": "system", "content": "You are a helpful assistant."}],
    tools=tools
)

# Create the context aggregator to collect the user and assistant context
user_aggregator, assistant_aggregator = LLMContextAggregatorPair(context)

# Create the pipeline
pipeline = Pipeline([
    transport.input(),     # Input from the transport
    stt,                   # STT processing
    user_aggregator,       # User context aggregation
    llm,                   # LLM processing
    tts,                   # TTS processing
    transport.output(),    # Output to the transport
    assistant_aggregator,  # Assistant context aggregation
])
```

## Function Handler Details

### FunctionCallParams

Every function handler receives a `FunctionCallParams` object containing all the information needed for execution:

```python  theme={null}
@dataclass
class FunctionCallParams:
    function_name: str                          # Name of the called function
    tool_call_id: str                           # Unique identifier for this call
    arguments: Mapping[str, Any]                # Arguments from the LLM
    llm: LLMService                             # Reference to the LLM service
    context: LLMContext                         # Current conversation context
    result_callback: FunctionCallResultCallback # Return results here
```

**Using the parameters:**

```python  theme={null}
async def example_function_handler(params: FunctionCallParams):
    # Access function details
    print(f"Called function: {params.function_name}")
    print(f"Call ID: {params.tool_call_id}")

    # Extract arguments
    location = params.arguments["location"]

    # Access LLM context for conversation history
    messages = params.context.messages

    # Use LLM service for additional operations
    await params.llm.push_frame(TTSSpeakFrame("Looking up weather data..."))

    # Return results
    await params.result_callback({"conditions": "nice", "temperature": "75"})
```

See the [API reference](https://reference-server.pipecat.ai/en/latest/api/pipecat.services.llm_service.html#pipecat.services.llm_service.FunctionCallParams) for complete details.

### Handler Structure

Your function handler should:

1. Receive necessary arguments, either:
   * From `params.arguments`
   * Directly from function arguments, if using [direct functions](#using-direct-functions-shorthand)
2. Process data or call external services
3. Return results via `params.result_callback(result)`

<CodeGroup>
  ```python Non-Direct Function theme={null}
  async def fetch_weather_from_api(params: FunctionCallParams):
      try:
          # Extract arguments
          location = params.arguments.get("location")
          format_type = params.arguments.get("format", "celsius")

          # Call external API
          api_result = await weather_api.get_weather(location, format_type)

          # Return formatted result
          await params.result_callback({
              "location": location,
              "temperature": api_result["temp"],
              "conditions": api_result["conditions"],
              "unit": format_type
          })
      except Exception as e:
          # Handle errors
          await params.result_callback({
              "error": f"Failed to get weather: {str(e)}"
          })
  ```

  ```python Direct Function theme={null}
  async def get_current_weather(params: FunctionCallParams, location: str, format: str):
      """Get the current weather.

      Args:
          location: The city and state, e.g. "San Francisco, CA".
          format: The temperature unit to use. Must be either "celsius" or "fahrenheit".
      """
      try:
          # Call external API
          api_result = await weather_api.get_weather(location, format)

          # Return formatted result
          await params.result_callback({
              "location": location,
              "temperature": api_result["temp"],
              "conditions": api_result["conditions"],
              "unit": format
          })
      except Exception as e:
          # Handle errors
          await params.result_callback({
              "error": f"Failed to get weather: {str(e)}"
          })
  ```
</CodeGroup>

## Controlling Function Call Behavior (Advanced)

When returning results from a function handler, you can control how the LLM processes those results using a `FunctionCallResultProperties` object passed to the result callback.

### Properties

`FunctionCallResultProperties` provides fine-grained control over LLM execution:

```python  theme={null}
@dataclass
class FunctionCallResultProperties:
    run_llm: Optional[bool] = None                 # Whether to run LLM after this result
    on_context_updated: Optional[Callable] = None  # Callback when context is updated
```

**Property options:**

* **`run_llm=True`**: Run LLM after function call (default behavior)
* **`run_llm=False`**: Don't run LLM after function call (useful for chained calls)
* **`on_context_updated`**: Async callback executed after the function result is added to context

<Tip>
  Skip LLM execution (`run_llm=False`) when you have back-to-back function
  calls. If you skip a completion, you must manually trigger one from the
  context aggregator.
</Tip>

See the [API reference](https://reference-server.pipecat.ai/en/latest/api/pipecat.frames.frames.html#pipecat.frames.frames.FunctionCallResultProperties) for complete details.

### Example Usage

```python  theme={null}
from pipecat.frames.frames import FunctionCallResultProperties
from pipecat.services.llm_service import FunctionCallParams

async def fetch_weather_from_api(params: FunctionCallParams):
    # Fetch weather data
    weather_data = {"conditions": "sunny", "temperature": "75"}

    # Don't run LLM after this function call
    properties = FunctionCallResultProperties(run_llm=False)

    await params.result_callback(weather_data, properties=properties)

async def query_database(params: FunctionCallParams):
    # Query database
    results = await db.query(params.arguments["query"])

    async def on_update():
        await notify_system("Database query complete")

    # Run LLM after function call and notify when context is updated
    properties = FunctionCallResultProperties(
        run_llm=True,
        on_context_updated=on_update
    )

    await params.result_callback(results, properties=properties)
```

## Key Takeaways

* **Function calling extends LLM capabilities** beyond training data to real-time information
* **Context integration is automatic** - function calls and results are stored in conversation history
* **Multiple definition approaches** - use standard schema for portability, direct functions for simplicity
* **Pipeline integration is seamless** - functions work within your existing voice AI architecture
* **Advanced control available** - fine-tune LLM execution and monitor function call lifecycle

## What's Next

Now that you understand function calling, let's explore how to configure text-to-speech services to convert your LLM's responses (including function call results) into natural-sounding speech.

<Card title="Text to Speech" icon="arrow-right" href="/guides/learn/text-to-speech">
  Learn how to configure speech synthesis in your voice AI pipeline
</Card>

---

# Source: https://docs.pipecat.ai/server/utilities/filters/function-filter.md

> ## Documentation Index
> Fetch the complete documentation index at: https://docs.pipecat.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# FunctionFilter

> Processor that filters frames using a custom filter function

## Overview

`FunctionFilter` is a flexible processor that uses a custom async function to determine which frames to pass through. This allows for complex, dynamic filtering logic beyond simple type checking.

## Constructor Parameters

<ParamField path="filter" type="Callable[[Frame], Awaitable[bool]]" required>
  Async function that examines each frame and returns True to allow it or False
  to filter it out
</ParamField>

<ParamField path="direction" type="FrameDirection" default="FrameDirection.DOWNSTREAM">
  Which direction of frames to filter (DOWNSTREAM or UPSTREAM)
</ParamField>

## Functionality

When a frame passes through the processor:

1. System frames and end frames are always passed through
2. Frames moving in a different direction than specified are always passed through
3. Other frames are passed to the filter function
4. If the filter function returns True, the frame is passed through

## Output Frames

The processor conditionally passes through frames based on:

* Frame type (system frames and end frames always pass)
* Frame direction (only filters in the specified direction)
* Result of the custom filter function

## Usage Example

```python  theme={null}
from pipecat.frames.frames import TextFrame, Frame
from pipecat.processors.filters import FunctionFilter
from pipecat.processors.frame_processor import FrameDirection

# Create filter that only allows TextFrames with more than 10 characters
async def long_text_filter(frame: Frame) -> bool:
    if isinstance(frame, TextFrame):
        return len(frame.text) > 10
    return False

# Apply filter to downstream frames only
text_length_filter = FunctionFilter(
    filter=long_text_filter,
    direction=FrameDirection.DOWNSTREAM
)

# Add to pipeline
pipeline = Pipeline([
    source,
    text_length_filter,  # Filters out short text frames
    destination
])
```

## Frame Flow

```mermaid  theme={null}
graph TD
    A[Input Frames] --> B[FunctionFilter]
    B --> C{System/End Frame?}
    C -->|Yes| F[Output Frame]
    C -->|No| D{Correct Direction?}
    D -->|No| F
    D -->|Yes| E{Filter Function}
    E -->|Returns True| F
    E -->|Returns False| G[Dropped]
```

## Notes

* Provides maximum flexibility for complex filtering logic
* Can incorporate dynamic conditions that change at runtime
* Only filters frames moving in the specified direction
* Always passes through system frames for proper pipeline operation
* Can be used to create sophisticated content-based filters
* Supports async filter functions for complex processing

---

# Source: https://docs.pipecat.ai/deployment/pipecat-cloud/guides/container-registries/gcp-artifact-registry.md

> ## Documentation Index
> Fetch the complete documentation index at: https://docs.pipecat.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# GCP Artifact Registry

> Deploy Pipecat Cloud agents using Google Cloud Artifact Registry

For GCP Artifact Registry, you can use a service account JSON key to authenticate with your private registry.

## Authenticate Docker

First, authenticate Docker with your service account key:

```bash  theme={null}
# Authenticate Docker with your service account key
cat key-file.json | docker login -u _json_key --password-stdin https://<location>-docker.pkg.dev
```

## Configure Image Pull Secrets

Create image credentials in Pipecat Cloud using the same service account:

```bash  theme={null}
pipecat cloud secrets image-pull-secret my-gcp-credentials https://<location>-docker.pkg.dev
```

When prompted, enter:

* Username: `_json_key`
* Password: The contents of your service account JSON key file

## Configure Your Deployment

Create a `pcc-deploy.toml` file with your Artifact Registry image configuration:

```toml  theme={null}
agent_name = "my-gcp-agent"
image = "<location>-docker.pkg.dev/<project-id>/<repo-name>/my-agent:0.1"
secret_set = "my-agent-secrets"
image_credentials = "my-gcp-credentials"

[scaling]
    min_instances = 0
```

## Build and Push to Artifact Registry

Build and push your agent image using the Pipecat Cloud CLI:

```bash  theme={null}
# Build and push using your pcc-deploy.toml configuration
pipecat cloud docker build-push
```

This command automatically builds for the correct platform (`linux/arm64`) and pushes to your configured Artifact Registry repository.

## Deploy Your Agent

Deploy using your configured `pcc-deploy.toml`:

```bash  theme={null}
pipecat cloud deploy
```

## Configuration Reference

Replace the following placeholders with your actual values:

* `<location>` with your Artifact Registry location (e.g., `us-central1`, `asia-east1`)
* `<project-id>` with your Google Cloud project ID
* `<repo-name>` with your Artifact Registry repository name
* `key-file.json` with the path to your service account JSON key file

<Note>
  The service account used for image credentials should have the `Artifact
    Registry Reader` role or equivalent permissions to pull images from your
  private repository.
</Note>

---

# Source: https://docs.pipecat.ai/server/services/s2s/gemini-live-vertex.md

> ## Documentation Index
> Fetch the complete documentation index at: https://docs.pipecat.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Gemini Live Vertex AI

> A real-time, multimodal conversational AI service powered by Google's Gemini via Vertex AI

## Overview

`GeminiLiveVertexLLMService` enables natural, real-time conversations with Google's Gemini model through Vertex AI. It provides built-in audio transcription, voice activity detection, and context management for creating interactive AI experiences with multimodal capabilities including audio, video, and text processing.

<Tip>
  Want to start building? Check out our [Gemini Live
  Guide](/guides/features/gemini-live) for general concepts, then follow the
  Vertex AI-specific setup below.
</Tip>

<CardGroup cols={2}>
  <Card title="Gemini Live Vertex API Reference" icon="code" href="https://reference-server.pipecat.ai/en/latest/api/pipecat.services.google.gemini_live.llm_vertex.html">
    Pipecat's API methods for Gemini Live Vertex AI integration
  </Card>

  <Card title="Example Implementation" icon="play" href="https://github.com/pipecat-ai/pipecat/blob/main/examples/foundational/26h-gemini-live-vertex-function-calling.py">
    Complete Gemini Live Vertex AI function calling example
  </Card>

  <Card title="Vertex AI Gemini Documentation" icon="book" href="https://cloud.google.com/vertex-ai/generative-ai/docs/model-reference/multimodal-live">
    Official Vertex AI Gemini Live API documentation
  </Card>

  <Card title="Gemini Live Model Card" icon="book" href="https://docs.cloud.google.com/vertex-ai/generative-ai/docs/models/gemini/2-5-flash-live-api">
    Gemini Live available models
  </Card>
</CardGroup>

## Installation

To use Gemini Live Vertex AI services, install the required dependencies:

```bash  theme={null}
pip install "pipecat-ai[google]"
```

## Prerequisites

### Google Cloud Setup

Before using Gemini Live Vertex AI services, you need:

1. **Google Cloud Project**: Set up a project in the [Google Cloud Console](https://console.cloud.google.com/)
2. **Vertex AI API**: Enable the Vertex AI API in your project
3. **Service Account**: Create a service account with `roles/aiplatform.user` and `roles/ml.developer` permissions
4. **Authentication**: Set up service account credentials or Application Default Credentials

### Required Environment Variables

* `GOOGLE_VERTEX_TEST_CREDENTIALS`: JSON string of service account credentials (optional if using ADC)
* `GOOGLE_CLOUD_PROJECT_ID`: Your Google Cloud project ID
* `GOOGLE_CLOUD_LOCATION`: Vertex AI region (e.g., "us-east4")

### Key Features

* **Enterprise Authentication**: Secure service account-based authentication
* **Multimodal Processing**: Handle audio, video, and text inputs simultaneously
* **Real-time Streaming**: Low-latency audio and video processing
* **Voice Activity Detection**: Automatic speech detection and turn management
* **Function Calling**: Advanced tool integration and API calling capabilities
* **Context Management**: Intelligent conversation history and system instruction handling

---

# Source: https://docs.pipecat.ai/server/services/s2s/gemini-live.md

# Source: https://docs.pipecat.ai/guides/features/gemini-live.md

> ## Documentation Index
> Fetch the complete documentation index at: https://docs.pipecat.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Building with Gemini Live

> Create real-time voice AI agents using Google's Gemini Live API and Pipecat

Gemini Live is Google's speech-to-speech API that enables natural, real-time voice conversations with AI. With Pipecat, you can build production-ready voice agents that leverage Gemini Live for telephony, web, and mobile applications.

<CardGroup cols={2}>
  <Card title="API Reference" icon="book" href="/server/services/s2s/gemini-live">
    Gemini Live service documentation
  </Card>

  <Card title="Pipecat CLI" icon="terminal" href="/cli/overview">
    Scaffold and deploy projects
  </Card>
</CardGroup>

## Capabilities

Pipecat's Gemini Live integration supports multiple modalities and deployment targets:

<CardGroup cols={2}>
  <Card title="Voice" icon="microphone">
    Real-time speech-to-speech conversations with natural turn-taking and voice activity detection
  </Card>

  <Card title="Vision" icon="eye">
    Process video and screenshare alongside audio for multimodal interactions
  </Card>

  <Card title="Telephony" icon="phone">
    Build phone-based voice agents with Twilio WebSocket integration
  </Card>

  <Card title="Tool Use" icon="wrench">
    Function calling support for external integrations and dynamic responses
  </Card>
</CardGroup>

### Architecture

Pipecat manages connections between your client and Gemini Live:

<Frame><img src="https://mintcdn.com/daily/jHf4uyQMOanEqe_B/images/gemini-live-architecture.png?fit=max&auto=format&n=jHf4uyQMOanEqe_B&q=85&s=af85a896b9d5d5d74b507492035ce543" alt="Gemini Live Architecture" data-og-width="2110" width="2110" data-og-height="484" height="484" data-path="images/gemini-live-architecture.png" data-optimize="true" data-opv="3" srcset="https://mintcdn.com/daily/jHf4uyQMOanEqe_B/images/gemini-live-architecture.png?w=280&fit=max&auto=format&n=jHf4uyQMOanEqe_B&q=85&s=eefd4bbc48e1d83cf4b21d391db47fab 280w, https://mintcdn.com/daily/jHf4uyQMOanEqe_B/images/gemini-live-architecture.png?w=560&fit=max&auto=format&n=jHf4uyQMOanEqe_B&q=85&s=5d059414d6a89ff3abf37e8cb350de30 560w, https://mintcdn.com/daily/jHf4uyQMOanEqe_B/images/gemini-live-architecture.png?w=840&fit=max&auto=format&n=jHf4uyQMOanEqe_B&q=85&s=ebfcd44e5feefd35c8e39eb7ac2fb66d 840w, https://mintcdn.com/daily/jHf4uyQMOanEqe_B/images/gemini-live-architecture.png?w=1100&fit=max&auto=format&n=jHf4uyQMOanEqe_B&q=85&s=720298eb0705ce18ef895beb6e651f2d 1100w, https://mintcdn.com/daily/jHf4uyQMOanEqe_B/images/gemini-live-architecture.png?w=1650&fit=max&auto=format&n=jHf4uyQMOanEqe_B&q=85&s=535f1afd22b3e1ddcd290b8e9ce2d2bb 1650w, https://mintcdn.com/daily/jHf4uyQMOanEqe_B/images/gemini-live-architecture.png?w=2500&fit=max&auto=format&n=jHf4uyQMOanEqe_B&q=85&s=7ae2588872eee308dd9c9c14be4163fb 2500w" /></Frame>

The Pipecat server handles media streaming with clients via WebRTC (web/mobile) or WebSockets (telephony), while maintaining a persistent connection to Gemini Live for real-time AI processing.

## Quick Start

The fastest way to start building is with the Pipecat CLI:

```bash  theme={null}
# Install the CLI
uv tool install pipecat-ai-cli

# Create a new project
pipecat init
```

The CLI will guide you through selecting:

* **Bot type**: Gemini Live (speech-to-speech)
* **Transport**: Daily WebRTC, Twilio, or others
* **Deployment target**: Local development or Pipecat Cloud

<Tip>
  All CLI commands can use either `pipecat` or the shorter `pc` alias.
</Tip>

## Starter Projects

These complete examples demonstrate Gemini Live in production scenarios. Each includes local development setup and Pipecat Cloud deployment configuration.

### Phone Bot (Twilio)

A telephone-based voice agent using Gemini Live with Twilio WebSockets. The demo plays "Two Truths and a Lie" to showcase natural conversation flow.

<Card title="Phone Bot Starter" icon="phone" href="https://github.com/pipecat-ai/pipecat-examples/tree/main/gemini-live-starters/phone-bot">
  Build a production phone agent with Twilio integration
</Card>

**Try it now**: Call **1-970-LIVE-API** (1-970-548-3274) to talk to a live demo.

**What you'll learn**:

* Twilio WebSocket transport configuration
* Google STT/TTS integration alongside Gemini Live
* TwiML setup for incoming calls
* Pipecat Cloud deployment with telephony

### Web Bot (Vision)

A browser-based agent with screensharing and vision capabilities, built with the Pipecat Voice UI Kit and Daily WebRTC transport.

<Card title="Web Bot Starter" icon="browser" href="https://github.com/pipecat-ai/pipecat-examples/tree/main/gemini-live-starters/web-bot">
  Build a web agent with vision and screensharing
</Card>

**What you'll learn**:

* Daily WebRTC transport for web clients
* Vision/screenshare processing with Gemini Live
* Next.js client with Voice UI Kit components
* Resizable panels and event logging

## Deployment

Both starter projects include configuration for [Pipecat Cloud](/deployment/pipecat-cloud/introduction), which handles scaling, monitoring, and global deployment.

```bash  theme={null}
# Authenticate with Pipecat Cloud
pipecat cloud auth login

# Build and push your Docker image
pipecat cloud docker build-push

# Deploy your agent
pipecat cloud deploy
```

Each starter includes a `pcc-deploy.toml` file with sensible defaults for agent configuration and scaling.

<Card title="Pipecat Cloud Deployment" icon="cloud" href="/deployment/pipecat-cloud/fundamentals/deploy">
  Learn more about deploying to production
</Card>

## Next Steps

<CardGroup cols={2}>
  <Card title="Function Calling" icon="code" href="/guides/learn/function-calling">
    Add external integrations and dynamic responses
  </Card>

  <Card title="React Client SDK" icon="react" href="/client/react/introduction">
    Build custom web interfaces
  </Card>

  <Card title="Telephony Guide" icon="phone-volume" href="/deployment/pipecat-cloud/guides/telephony/twilio-websocket">
    Deep dive into phone integrations
  </Card>

  <Card title="Core Concepts" icon="graduation-cap" href="/guides/learn/overview">
    Understand Pipecat pipelines and processors
  </Card>
</CardGroup>

---

# Source: https://docs.pipecat.ai/client/ios/transports/gemini-websocket.md

# Source: https://docs.pipecat.ai/client/android/transports/gemini-websocket.md

> ## Documentation Index
> Fetch the complete documentation index at: https://docs.pipecat.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Gemini Live Websocket Transport

> Websocket implementation for Android using Gemini

The Gemini Live Websocket transport implementation enables real-time audio communication with the Gemini Multimodal Live service, using a direct websocket connection.

<Note>
  Transports of this type are designed primarily for development and testing
  purposes. For production applications, you will need to build a server
  component with a server-friendly transport, like the
  [DailyTransport](./daily), to securely handle API keys.
</Note>

## Installation

Add the transport dependency to your `build.gradle`:

```gradle  theme={null}
implementation "ai.pipecat:gemini-live-websocket-transport:0.3.7"
```

## Usage

Create a client:

```kotlin  theme={null}
val transport = GeminiLiveWebsocketTransport.Factory(context)

val options = RTVIClientOptions(
    params = RTVIClientParams(
        baseUrl = null,
        config = GeminiLiveWebsocketTransport.buildConfig(
            apiKey = "<your Gemini api key>",
            generationConfig = Value.Object(
                "speech_config" to Value.Object(
                    "voice_config" to Value.Object(
                        "prebuilt_voice_config" to Value.Object(
                            "voice_name" to Value.Str("Puck")
                        )
                    )
                )
            ),
            initialUserMessage = "How tall is the Eiffel Tower?"
        )
    )
)

val client = RTVIClient(transport, callbacks, options)

client.start().withCallback {
    // ...
}
```

## Resources

<CardGroup cols={2}>
  <Card horizontal title="Demo" icon="play" href="https://github.com/pipecat-ai/pipecat-client-android-gemini-live-websocket-demo">
    Simple Chatbot Demo
  </Card>

  <Card horizontal title="Source" icon="github" href="https://github.com/pipecat-ai/pipecat-client-android-transports/">
    Client Transports
  </Card>
</CardGroup>

<Card title="Pipecat Android Client Reference" icon="book-open" href="https://docs-android.rtvi.ai/">
  Complete API documentation for the Pipecat Android client.
</Card>

---

# Source: https://docs.pipecat.ai/server/services/llm/gemini.md

# Source: https://docs.pipecat.ai/client/js/transports/gemini.md

> ## Documentation Index
> Fetch the complete documentation index at: https://docs.pipecat.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# GeminiLiveWebSocketTransport

## Overview

The `GeminiLiveWebsocketTransport` class implements a fully functional [Pipecat `Transport`](./transport), providing a framework for implementing real-time communication directly with the [Gemini Multimodal Live](https://ai.google.dev/api/multimodal-live) service. Like all transports, it handles media device management, audio/video streams, and state management for the connection.

<Note>
  Transports of this type are designed primarily for development and testing
  purposes. For production applications, you will need to build a server
  component with a server-friendly transport, like the
  [DailyTransport](./daily), to securely handle API keys.
</Note>

## Usage

### Basic Setup

```javascript  theme={null}
import {
  GeminiLiveWebsocketTransport,
  GeminiLLMServiceOptions,
} from "@pipecat-ai/gemini-live-websocket-transport";
import { PipecatClient } from "@pipecat-ai/client-js";

const options: GeminiLLMServiceOptions = {
  api_key: "YOUR_API_KEY",
  initial_messages: [
    // Set up initial system and user messages.
    // Without the user message, the bot will not respond immediately
    // and wait for the user to speak first.
    {
      role: "model",
      content: "You are a confused jellyfish.",
    },
    { role: "user", content: "Blub blub!" },
  ],
  generation_config: {
    temperature: 0.7,
    maxOutput_tokens: 1000,
  },
};

const transport = new GeminiLiveWebsocketTransport(options);
let pcClient = new PipecatClient({
  transport: new GeminiLiveWebsocketTransport(options),
  callbacks: {
    // Event handlers
  },
});
pcClient.connect();
```

## API Reference

### Constructor Options

#### `GeminiLLMServiceOptions`

```typescript  theme={null}
interface GeminiLLMServiceOptions {
  api_key: string; // Required: Your Gemini API key
  initial_messages?: Array<{
    // Optional: Initial conversation context
    content: string;
    role: string;
  }>;
  generation_config?: {
    // Optional: Generation parameters
    candidate_count?: number;
    max_output_tokens?: number;
    temperature?: number;
    top_p?: number;
    top_k?: number;
    presence_penalty?: number;
    frequency_penalty?: number;
    response_modalities?: string;
    speech_config?: {
      voice_config?: {
        prebuilt_voice_config?: {
          voice_name: "Puck" | "Charon" | "Kore" | "Fenrir" | "Aoede";
        };
      };
    };
  };
}
```

### TransportConnectionParams

The `GeminiLiveWebsocketTransport` does not take connection parameters. It connects directly to the Gemini Multimodal Live service using the API key provided as part of the initial configuration.

### Events

The GeminiLiveWebSocketTransport implements the various [PipecatClient event handlers](/client/js/api-reference/callbacks). Check out the docs or samples for more info.

## More Information

<CardGroup cols={2}>
  <Card horizontal title="Demo" icon="play" href="https://github.com/pipecat-ai/pipecat-client-web-transports/tree/main/examples/directToLLMTransports">
    Gemini MultiModal Live Basic Demo
  </Card>

  <Card horizontal title="Source" icon="github" href="https://github.com/pipecat-ai/pipecat-client-web-transports/tree/main/transports/gemini-live-websocket-transport">
    `GeminiLiveWebsocketTransport`
  </Card>
</CardGroup>

<Card horizontal title="Package" icon="browser" href="https://www.npmjs.com/package/@pipecat-ai/gemini-live-websocket-transport">
  `@pipecat-ai/realtime-websocket-transport`
</Card>

---

# Source: https://docs.pipecat.ai/server/services/stt/gladia.md

> ## Documentation Index
> Fetch the complete documentation index at: https://docs.pipecat.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Gladia

> Speech-to-text service implementation using Gladia's API

## Overview

`GladiaSTTService` provides real-time speech recognition using Gladia's WebSocket API with support for 99+ languages, custom vocabulary, translation, sentiment analysis, and advanced audio processing features for comprehensive transcription.

<CardGroup cols={2}>
  <Card title="Gladia STT API Reference" icon="code" href="https://reference-server.pipecat.ai/en/latest/api/pipecat.services.gladia.html">
    Pipecat's API methods for Gladia STT integration
  </Card>

  <Card title="Example Implementation" icon="play" href="https://github.com/pipecat-ai/pipecat/blob/main/examples/foundational/07j-interruptible-gladia.py">
    Complete example with interruption handling
  </Card>

  <Card title="Gladia Documentation" icon="book" href="https://docs.gladia.io/api-reference/live-flow">
    Official Gladia documentation and features
  </Card>

  <Card title="Gladia Platform" icon="microphone" href="https://www.gladia.io/">
    Access multilingual transcription and API keys
  </Card>
</CardGroup>

## Installation

To use Gladia services, install the required dependency:

```bash  theme={null}
pip install "pipecat-ai[gladia]"
```

## Prerequisites

### Gladia Account Setup

Before using Gladia STT services, you need:

1. **Gladia Account**: Sign up at [Gladia](https://www.gladia.io/)
2. **API Key**: Generate an API key from your account dashboard
3. **Region Selection**: Choose your preferred region (EU-West or US-West)

### Required Environment Variables

* `GLADIA_API_KEY`: Your Gladia API key for authentication
* `GLADIA_REGION`: Your preferred region (optional, defaults to "eu-west")

---

# Source: https://docs.pipecat.ai/server/services/image-generation/google-imagen.md

> ## Documentation Index
> Fetch the complete documentation index at: https://docs.pipecat.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Google Imagen

> Image generation service implementation using Google's Imagen models

## Overview

`GoogleImageGenService` provides high-quality image generation capabilities using Google's Imagen models. It supports generating multiple images from text prompts with various customization options and advanced prompt understanding for photorealistic and artistic image creation.

<CardGroup cols={2}>
  <Card title="Google Imagen API Reference" icon="code" href="https://reference-server.pipecat.ai/en/latest/api/pipecat.services.google.image.html">
    Pipecat's API methods for Google Imagen integration
  </Card>

  <Card title="Example Implementation" icon="play" href="https://github.com/pipecat-ai/pipecat/blob/main/examples/foundational/03b-still-frame-imagen.py">
    Browse examples using Google Imagen
  </Card>

  <Card title="Google Imagen Documentation" icon="book" href="https://cloud.google.com/vertex-ai/generative-ai/docs/image/overview">
    Official Google Imagen API documentation and guides
  </Card>

  <Card title="Google Cloud Console" icon="image" href="https://console.cloud.google.com/">
    Access Imagen models and manage API keys
  </Card>
</CardGroup>

## Installation

To use Google Imagen services, install the required dependencies:

```bash  theme={null}
pip install "pipecat-ai[google]"
```

## Prerequisites

### Google Cloud Setup

Before using Google Imagen services, you need:

1. **Google Cloud Account**: Set up at [Google Cloud Console](https://console.cloud.google.com/)
2. **API Key**: Generate a Google API key with Vertex AI access
3. **Project Configuration**: Enable Vertex AI API for your project
4. **Model Access**: Ensure access to Imagen generation models

### Required Environment Variables

* `GOOGLE_API_KEY`: Your Google API key for authentication

---

# Source: https://docs.pipecat.ai/server/services/llm/google-vertex.md

> ## Documentation Index
> Fetch the complete documentation index at: https://docs.pipecat.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Google Vertex AI

> LLM service implementation using Google's Vertex AI with OpenAI-compatible interface

## Overview

`GoogleVertexLLMService` provides access to Google's language models through Vertex AI while maintaining an OpenAI-compatible interface. It inherits from `OpenAILLMService` and supports all OpenAI features while connecting to Google's enterprise AI services with enhanced security and compliance.

<CardGroup cols={2}>
  <Card title="Vertex AI LLM API Reference" icon="code" href="https://reference-server.pipecat.ai/en/latest/api/pipecat.services.google.llm_vertex.html">
    Pipecat's API methods for Google Vertex AI integration
  </Card>

  <Card title="Example Implementation" icon="play" href="https://github.com/pipecat-ai/pipecat/blob/main/examples/foundational/14p-function-calling-gemini-vertex-ai.py">
    Browse examples using Vertex AI models
  </Card>

  <Card title="Vertex AI Documentation" icon="book" href="https://cloud.google.com/vertex-ai/generative-ai/docs/model-reference/inference">
    Official Google Vertex AI documentation
  </Card>

  <Card title="Google Cloud Console" icon="microphone" href="https://console.cloud.google.com/vertex-ai">
    Access Vertex AI and manage credentials
  </Card>
</CardGroup>

## Installation

To use Google Vertex AI services, install the required dependencies:

```bash  theme={null}
pip install "pipecat-ai[google]"
```

## Prerequisites

### Google Cloud Setup

Before using Google Vertex AI LLM services, you need:

1. **Google Cloud Account**: Sign up at [Google Cloud Console](https://console.cloud.google.com/)
2. **Project Setup**: Create a project and enable the Vertex AI API
3. **Service Account**: Create a service account with Vertex AI permissions
4. **Authentication**: Set up credentials via service account key or Application Default Credentials

### Required Environment Variables

* `GOOGLE_APPLICATION_CREDENTIALS`: Path to your service account key file (recommended)
* Or use Application Default Credentials for cloud deployments

---

# Source: https://docs.pipecat.ai/server/services/tts/google.md

# Source: https://docs.pipecat.ai/server/services/stt/google.md

> ## Documentation Index
> Fetch the complete documentation index at: https://docs.pipecat.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Google

> Speech-to-text service implementation using Google Cloud's Speech-to-Text V2 API

## Overview

`GoogleSTTService` provides real-time speech recognition using Google Cloud's Speech-to-Text V2 API with support for 125+ languages, multiple models, voice activity detection, and advanced features like automatic punctuation and word-level confidence scores.

<CardGroup cols={2}>
  <Card title="Google STT API Reference" icon="code" href="https://reference-server.pipecat.ai/en/latest/api/pipecat.services.google.stt.html">
    Pipecat's API methods for Google Cloud STT integration
  </Card>

  <Card title="Example Implementation" icon="play" href="https://github.com/pipecat-ai/pipecat/blob/main/examples/foundational/07n-interruptible-google.py">
    Complete example with Google Cloud services
  </Card>

  <Card title="Google Cloud Documentation" icon="book" href="https://cloud.google.com/speech-to-text/v2/docs">
    Official Google Cloud Speech-to-Text documentation
  </Card>

  <Card title="Google Cloud Console" icon="microphone" href="https://console.cloud.google.com/iam-admin/serviceaccounts">
    Create service accounts and manage API access
  </Card>
</CardGroup>

## Installation

To use Google Cloud Speech services, install the required dependency:

```bash  theme={null}
pip install "pipecat-ai[google]"
```

## Prerequisites

### Google Cloud Setup

Before using Google Cloud STT services, you need:

1. **Google Cloud Account**: Sign up at [Google Cloud Console](https://console.cloud.google.com/)
2. **Project Setup**: Create a project and enable the Speech-to-Text API
3. **Service Account**: Create a service account with Speech-to-Text permissions
4. **Authentication**: Set up credentials via service account key or Application Default Credentials

### Required Environment Variables

* `GOOGLE_APPLICATION_CREDENTIALS`: Path to your service account key file (recommended)
* Or use Application Default Credentials for cloud deployments

---

# Source: https://docs.pipecat.ai/server/services/llm/grok.md

> ## Documentation Index
> Fetch the complete documentation index at: https://docs.pipecat.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Grok

> LLM service implementation using Grok's API with OpenAI-compatible interface

## Overview

`GrokLLMService` provides access to Grok's language models through an OpenAI-compatible interface. It inherits from `OpenAILLMService` and supports streaming responses, function calling, and context management with Grok's unique reasoning capabilities.

<CardGroup cols={2}>
  <Card title="Grok LLM API Reference" icon="code" href="https://reference-server.pipecat.ai/en/latest/api/pipecat.services.grok.llm.html">
    Pipecat's API methods for Grok integration
  </Card>

  <Card title="Example Implementation" icon="play" href="https://github.com/pipecat-ai/pipecat/blob/main/examples/foundational/14g-function-calling-grok.py">
    Complete example with function calling
  </Card>

  <Card title="Grok Documentation" icon="book" href="https://docs.x.ai/docs/api-reference#chat-completions">
    Official Grok API documentation and features
  </Card>

  <Card title="X.AI Platform" icon="microphone" href="https://console.x.ai/">
    Access Grok models and manage API keys
  </Card>
</CardGroup>

## Installation

To use Grok services, install the required dependencies:

```bash  theme={null}
pip install "pipecat-ai[grok]"
```

## Prerequisites

### Grok Account Setup

Before using Grok LLM services, you need:

1. **X.AI Account**: Sign up at [X.AI Console](https://console.x.ai/)
2. **API Key**: Generate an API key from your console dashboard
3. **Model Selection**: Choose from available Grok models

### Required Environment Variables

* `XAI_API_KEY`: Your X.AI API key for authentication

---

# Source: https://docs.pipecat.ai/server/services/tts/groq.md

# Source: https://docs.pipecat.ai/server/services/stt/groq.md

# Source: https://docs.pipecat.ai/server/services/llm/groq.md

> ## Documentation Index
> Fetch the complete documentation index at: https://docs.pipecat.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Groq

> LLM service implementation using Groq's API with OpenAI-compatible interface

## Overview

`GroqLLMService` provides access to Groq's language models through an OpenAI-compatible interface. It inherits from `OpenAILLMService` and supports streaming responses, function calling, and context management with ultra-fast inference speeds.

<CardGroup cols={2}>
  <Card title="Groq LLM API Reference" icon="code" href="https://reference-server.pipecat.ai/en/latest/api/pipecat.services.groq.llm.html">
    Pipecat's API methods for Groq integration
  </Card>

  <Card title="Example Implementation" icon="play" href="https://github.com/pipecat-ai/pipecat/blob/main/examples/foundational/14f-function-calling-groq.py">
    Complete example with function calling
  </Card>

  <Card title="Groq Documentation" icon="book" href="https://console.groq.com/docs/api-reference#chat-create">
    Official Groq API documentation and features
  </Card>

  <Card title="Groq Console" icon="microphone" href="https://console.groq.com/">
    Access models and manage API keys
  </Card>
</CardGroup>

## Installation

To use Groq services, install the required dependency:

```bash  theme={null}
pip install "pipecat-ai[groq]"
```

## Prerequisites

### Groq Account Setup

Before using Groq LLM services, you need:

1. **Groq Account**: Sign up at [Groq Console](https://console.groq.com/)
2. **API Key**: Generate an API key from your console dashboard
3. **Model Selection**: Choose from available models with ultra-fast inference

### Required Environment Variables

* `GROQ_API_KEY`: Your Groq API key for authentication

---

# Source: https://docs.pipecat.ai/server/utilities/runner/guide.md

> ## Documentation Index
> Fetch the complete documentation index at: https://docs.pipecat.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Development Runner

> Unified runner for building voice AI bots with Daily, WebRTC, and telephony transports

## Overview

The Pipecat development runner provides a unified way to run voice AI bots across multiple transport types. It handles infrastructure setup - creating Daily rooms, managing WebRTC connections, and routing telephony calls.

## Installation

```bash  theme={null}
pip install pipecat-ai[runner]
```

## What is a Runner?

A runner in Pipecat refers to a "bot runner", an HTTP service that provides a gateway for spawning bots on-demand. It's the component that enables your bot to run by providing it with server infrastructure and connection details like rooms and tokens.

A bot runner typically creates transport sessions (like Daily WebRTC rooms), generates authentication tokens for both bots and users, spawns new bot processes when users request sessions, and manages bot lifecycle and cleanup. Think of it as the bridge between incoming user connections and your bot logic.

## How the Development Runner Works

The development runner operates as a FastAPI web server that automatically discovers and executes your bot code. When you start the runner, it creates the necessary web endpoints and infrastructure for your chosen transport type.

* **WebRTC connections**: It serves a built-in web interface where users can connect directly as well as an endpoint to create new WebRTC sessions
* **Daily integration**: It provides endpoints that create new rooms and tokens and redirect users to join them
* **Telephony providers**: For Twilio, it sets up webhook endpoints that handle incoming calls and establish WebSocket connections for audio streaming

The runner automatically detects which transport type you're using and configures the appropriate infrastructure. It then discovers your bot function and spawns new instances whenever users connect. This means you can focus on writing your bot logic while the runner handles all the server infrastructure, connection management, and transport-specific details.

Your bot code receives runner arguments that contain everything it needs, including Daily room URLs and tokens, WebRTC connections, or WebSocket streams for telephony. The runner abstracts away the complexity of managing these different connection types, providing a unified interface for building bots that work across multiple platforms.

## Pipecat Cloud Ready

The bot runner is designed to be cloud-ready, meaning that you can run the same bot code locally and deployed to Pipecat Cloud without any modifications. It automatically handles the differences in transport setup, providing you with the flexibility to test locally using an open source transport, like SmallWebRTCTransport, but run in production using Daily or telephony transports.

## Building with the Runner

Now let's build a practical example to see how this works. The key insight is that your bot code is structured into two parts: the core bot logic that works with any transport, and the entry point that creates the appropriate transport based on the runner arguments.

Here's the basic structure:

```python  theme={null}
# Your imports
from pipecat.runner.types import RunnerArguments
from pipecat.transports.base_transport import BaseTransport, TransportParams
from pipecat.transports.network.small_webrtc import SmallWebRTCTransport

async def run_bot(transport: BaseTransport):
    """Your core bot logic here:
    - Define services (STT, TTS, LLM)
    - Initialize messages and context
    - Create the pipeline
    - Add event handlers
    - Run the pipeline
    """

    # Your bot logic goes here
    # Define STT, LLM, TTS...
    # ...
    # Run your pipeline
    await runner.run(task)

async def bot(runner_args: RunnerArguments):
    """Main bot entry point compatible for local dev and Pipecat Cloud."""
    transport = SmallWebRTCTransport(
        params=TransportParams(
            audio_in_enabled=True,
            audio_out_enabled=True,
        ),
        webrtc_connection=runner_args.webrtc_connection,
    )

    await run_bot(transport)

if __name__ == "__main__":
    from pipecat.runner.run import main
    main()
```

The `run_bot()` function contains your actual bot logic and is transport-agnostic. The `bot()` function is the entry point that the runner calls - it creates the appropriate transport and passes it to your bot logic. This separation allows the same bot code to work across different transports.

When you run this with `python bot.py`, the development runner starts a web server and opens a browser interface at `http://localhost:7860/client`. Each time someone connects, the runner calls your `bot()` function with WebRTC runner arguments.

## Supporting Multiple Transports

To make your bot work across different platforms, you can detect the transport type from the runner arguments and create the appropriate transport. Here's how to support both Daily and WebRTC:

```python  theme={null}
from pipecat.runner.types import DailyRunnerArguments, RunnerArguments, SmallWebRTCRunnerArguments

async def bot(runner_args: RunnerArguments):
    """Main bot entry point compatible with Pipecat Cloud."""

    transport = None

    if isinstance(runner_args, DailyRunnerArguments):
        from pipecat.transports.daily.transport import DailyParams, DailyTransport

        transport = DailyTransport(
            runner_args.room_url,
            runner_args.token,
            "Pipecat Bot",
            params=DailyParams(
                audio_in_enabled=True,
                audio_out_enabled=True,
            ),
        )

    elif isinstance(runner_args, SmallWebRTCRunnerArguments):
        from pipecat.transports.base_transport import TransportParams
        from pipecat.transports.network.small_webrtc import SmallWebRTCTransport

        transport = SmallWebRTCTransport(
            params=TransportParams(
                audio_in_enabled=True,
                audio_out_enabled=True,
            ),
            webrtc_connection=runner_args.webrtc_connection,
        )

    else:
        logger.error(f"Unsupported runner arguments type: {type(runner_args)}")
        return

    if transport is None:
        logger.error("Failed to create transport")
        return

    await run_bot(transport)

if __name__ == "__main__":
    from pipecat.runner.run import main
    main()
```

Now you can run your bot with different transports:

```bash  theme={null}
python bot.py -t webrtc  # Uses SmallWebRTCRunnerArguments
python bot.py -t daily   # Uses DailyRunnerArguments
```

### Understanding Runner Arguments

Runner arguments are how the runner communicates transport-specific information to your bot. The runner determines which transport to use based on the command-line arguments, then creates the appropriate runner arguments:

* **`DailyRunnerArguments`**: Contains `room_url`, `token` (Optional), `body` (Optional) for joining Daily rooms
* **`SmallWebRTCRunnerArguments`**: Contains `webrtc_connection` for local WebRTC sessions
* **`WebSocketRunnerArguments`**: Contains `websocket` for telephony connections

All runner arguments also include:

* **`handle_sigint`**: Whether the bot should handle SIGINT (Ctrl+C) signals (managed automatically)
* **`handle_sigterm`**: Whether the bot should handle SIGTERM signals (managed automatically)

<Warning>
  `handle_sigint` and `handle_sigterm` are development only features and cannot
  be used when deploying to Pipecat Cloud.
</Warning>

The runner handles all the complex setup - creating Daily rooms, generating tokens, establishing WebSocket connections - and provides your bot with everything it needs through these runner arguments.

Notice how we use lazy imports (`from pipecat.transports.daily.transport import ...`) inside the conditional blocks. This ensures that transport-specific dependencies are only required when that transport is actually used, making your bot more portable.

<Info>
  `RunnerArguments` is the base class for all runner arguments. It provides a
  common interface for the runner to pass transport-specific information to your
  bot.
</Info>

## Environment Detection

When building bots that work both locally and in production, you often need to detect the execution environment to enable different features. The development runner sets the `ENV` environment variable to help with this:

```python  theme={null}
import os

async def bot(runner_args: RunnerArguments):
    # Check if running in local development environment
    is_local = os.environ.get("ENV") == "local"

    # Enable production features only when deployed
    if not is_local:
        from pipecat.audio.filters.krisp_viva_filter import KrispVivaFilter
        krisp_filter = KrispVivaFilter()
    else:
        krisp_filter = None

    transport = DailyTransport(
        runner_args.room_url,
        runner_args.token,
        "Pipecat Bot",
        params=DailyParams(
            audio_in_filter=krisp_filter,  # Krisp VIVA filter only in production
            audio_in_enabled=True,
            audio_out_enabled=True,
        ),
    )
```

### Environment Values

The development runner automatically sets environment variables based on how your bot is running:

* **Local development**: `ENV=local` (set by the development runner)
* **Production/Cloud deployment**: `ENV` is not set or has a different value

This allows you to easily customize behavior between development and production environments:

## All Supported Transports

The development runner supports six transport types, each designed for different use cases:

### WebRTC (`-t webrtc`)

Local WebRTC connections with a built-in browser interface. Perfect for development and testing.

```bash  theme={null}
python bot.py -t webrtc
# Opens http://localhost:7860/client

# ESP32 compatibility mode
python bot.py -t webrtc --esp32 --host 192.168.1.100
```

**Runner Arguments**: `SmallWebRTCRunnerArguments`

* `webrtc_connection`: Pre-configured WebRTC peer connection

### Daily (`-t daily`)

Integration with Daily for production video conferencing with rooms, participant management, and Pipecat client compatibility.

```bash  theme={null}
python bot.py -t daily
# Opens http://localhost:7860 with room creation interface
# Also provides POST /start endpoint for Pipecat clients

# Direct connection for testing (bypasses web server)
python bot.py -d

# Enable PSTN dial-in webhook handling
python bot.py -t daily --dialin
```

**Runner Arguments**: `DailyRunnerArguments`

* `room_url`: Daily room URL to join
* `token`: Authentication token for the room
* `body`: Request data from /start endpoint (dict) or dial-in webhook data

**Available Endpoints**:

* `GET /`: Web interface for creating rooms
* `POST /start`: RTVI-compatible endpoint for programmatic access
* `POST /daily-dialin-webhook`: PSTN dial-in webhook handler (requires `--dialin` flag)

### Telephony (`-t twilio|telnyx|plivo|exotel`)

Phone call integration through telephony providers. Requires a public webhook endpoint and provider credentials.

```bash  theme={null}
# Set up environment variables first
export TWILIO_ACCOUNT_SID=your_account_sid
export TWILIO_AUTH_TOKEN=your_auth_token

# Then run with public proxy
python bot.py -t twilio -x yourproxy.ngrok.io
```

**Environment Variables**:

* Twilio: `TWILIO_ACCOUNT_SID`, `TWILIO_AUTH_TOKEN`
* Telnyx: `TELNYX_API_KEY`
* Plivo: `PLIVO_AUTH_ID`, `PLIVO_AUTH_TOKEN`
* Exotel: None required (no hang-up functionality available)

Environment variables are optional, but when provided will attempt to hang up the call when the session ends.

**Runner Arguments**: `WebSocketRunnerArguments`

* `websocket`: WebSocket connection for audio streaming

The runner automatically detects the telephony provider from incoming WebSocket messages and configures the appropriate serializers and audio settings. Your bot receives a pre-configured FastAPI WebSocket connection ready for telephony audio streaming.

## Command Line Options

The development runner accepts several command-line arguments to customize its behavior:

```bash  theme={null}
python bot.py [OPTIONS]

Options:
  --host TEXT          Server host address (default: localhost)
  --port INTEGER       Server port (default: 7860)
  -t, --transport      Transport type: daily, webrtc, twilio, telnyx, plivo, exotel (default: webrtc)
  -x, --proxy TEXT     Public proxy hostname for telephony webhooks (required for telephony)
  --esp32              Enable SDP munging for ESP32 WebRTC compatibility
  -d, --direct         Connect directly to Daily room for testing (automatically sets transport to daily)
  --dialin             Enable Daily PSTN dial-in webhook handling (requires Daily transport)
  -v, --verbose        Increase logging verbosity
```

### Key Arguments

**`--transport` / `-t`**: Determines which transport infrastructure to set up

* `webrtc`: Local WebRTC with browser interface
* `daily`: Daily.co integration with room management
* `twilio`, `telnyx`, `plivo`, `exotel`: Telephony provider integration

**`--proxy` / `-x`**: Required for most telephony transports (Twilio, Telnyx, Plivo). This should be a publicly accessible hostname (like `yourbot.ngrok.io`) that can receive webhooks from telephony providers. The runner automatically strips protocol prefixes (http\://, https\://) if provided. Not required for Exotel, which uses direct WebSocket connections.

**`--direct` / `-d`**: Special mode for Daily that bypasses the web server and connects your bot directly to a Daily room. Useful for quick testing but not recommended for production use.

**`--dialin`**: Enables the `/daily-dialin-webhook` endpoint for handling Daily PSTN dial-in calls. Only works with Daily transport (`-t daily`). This endpoint receives webhook data from Daily when a phone call dials into your configured phone number, creates a SIP-enabled room, and spawns your bot.

**`--esp32`**: Enables SDP (Session Description Protocol) modifications needed for ESP32 WebRTC compatibility. Must be used with a specific IP address via `--host`.

### Environment Variables

Different transports require different environment variables:

**Daily**:

* `DAILY_API_KEY`: Daily API key for creating rooms and tokens (required for dial-in)
* `DAILY_SAMPLE_ROOM_URL` (Optional): Existing room URL to use
* `DAILY_API_URL` (Optional): Daily API URL (defaults to [https://api.daily.co/v1](https://api.daily.co/v1))

**Telephony**:

* `TWILIO_ACCOUNT_SID`, `TWILIO_AUTH_TOKEN`: Twilio credentials
* `PLIVO_AUTH_ID`, `PLIVO_AUTH_TOKEN`: Plivo credentials
* `TELNYX_API_KEY`: Telnyx API key

The runner automatically uses these environment variables when creating transport sessions and authentication tokens.

## Simplifying with the Transport Utility

While the manual approach gives you full control, the `create_transport` utility provides a much cleaner way to handle multiple transports. Instead of writing conditional logic for each transport type, you define transport configurations upfront and let the utility handle the selection:

```python  theme={null}
from pipecat.runner.utils import create_transport

# Define transport configurations using factory functions
transport_params = {
    "daily": lambda: DailyParams(
        audio_in_enabled=True,
        audio_out_enabled=True,
    ),
    "webrtc": lambda: TransportParams(
        audio_in_enabled=True,
        audio_out_enabled=True,
    ),
    "twilio": lambda: FastAPIWebsocketParams(
        audio_in_enabled=True,
        audio_out_enabled=True,
        # add_wav_header and serializer handled automatically
    ),
}

async def bot(runner_args):
    """Simplified bot entry point using the transport utility."""
    transport = await create_transport(runner_args, transport_params)
    await run_bot(transport)
```

The utility automatically:

* Detects the telephony provider from WebSocket messages
* Configures the appropriate serializer (Twilio, Telnyx, Plivo, or Exotel)
* Sets up authentication using environment variables
* Handles WebRTC and Daily transport creation

Now your bot supports all six transport types with just two lines of code in the bot() function.

## RTVI Integration

The development runner provides RTVI (Real-Time Voice Interface) compatible endpoints for the Pipecat client SDKs. This allows you to use the Pipecat client libraries locally during development as well as when deployed to Pipecat Cloud.

### The `/start` Endpoint

For Daily transports, the runner automatically creates a `/start` POST endpoint that:

1. **Creates Daily rooms and tokens** using your `DAILY_API_KEY`
2. **Spawns bot instances** with request `body` data
3. **Returns connection details** in RTVI-compatible format

```bash  theme={null}
# Start the Daily runner
python bot.py -t daily

# The /start endpoint is now available at:
# POST http://localhost:7860/start
```

**Request Format:**

```json  theme={null}
{
  "createDailyRoom": true,
  "dailyRoomProperties": {
    "start_video_off": true,
    "start_audio_off": false
  },
  "body": {
    "custom_data": "your_value",
    "user_id": "user123"
  }
}
```

<Note>`dailyRoomProperties` are not yet handled by the runner.</Note>

**Response Format:**

```json  theme={null}
{
  "dailyRoom": "https://domain.daily.co/room-name",
  "dailyToken": "eyJ0eXAiOiJKV1QiLCJhbGciOiJIUzI1NiJ9..."
}
```

### Accessing Request Data in Your Bot

The `body` field from the `/start` request is passed to your bot through `DailyRunnerArguments.body`:

```python  theme={null}
async def bot(runner_args: DailyRunnerArguments):
    # Access custom data from the /start request
    custom_data = runner_args.body.get("custom_data")
    user_id = runner_args.body.get("user_id")

    print(f"Bot started for user: {user_id}")
    print(f"Custom data: {custom_data}")

    # Your bot logic here
    transport = DailyTransport(runner_args.room_url, runner_args.token, "Bot")
    await run_bot(transport)
```

### RTVI Client Example

You can use RTVI client libraries to connect to your local development runner:

```javascript  theme={null}
import { PipecatClient } from "@pipecat-ai/client-js";
import { DailyTransport } from "@pipecat-ai/daily-transport";

const client = new PipecatClient({
  transport: new DailyTransport(),
  enableMic: true,
  enableCam: false,
});

// Start a session with custom data
await client.connect({ endpoint: "http://localhost:7860/start" });
```

<Info>
  The `/start` endpoint is only available for Daily transports (`-t daily`).
  WebRTC and telephony transports use different connection methods.
</Info>

## Daily Dial-In Webhook

The development runner can handle Daily PSTN dial-in webhooks when started with the `--dialin` flag. This allows you to test phone call integrations locally before deploying to production.

### Enabling Dial-In Support

```bash  theme={null}
python bot.py -t daily --dialin
# Webhook endpoint available at:
# POST http://localhost:7860/daily-dialin-webhook
```

### How It Works

When a phone call dials into your Daily phone number:

1. **Daily sends a webhook** to your configured endpoint with call details
2. **The runner creates a SIP-enabled room** with appropriate configuration
3. **Your bot is spawned** with dial-in context in `runner_args.body`
4. **The call is connected** to the room where your bot is waiting

### Webhook Payload

Daily sends the following data to the `/daily-dialin-webhook` endpoint:

```json  theme={null}
{
  "From": "+15551234567",
  "To": "+15559876543",
  "callId": "uuid-call-id",
  "callDomain": "uuid-call-domain",
  "sipHeaders": {}
}
```

### Accessing Dial-In Data in Your Bot

The dial-in webhook data is passed to your bot through `DailyRunnerArguments.body`. You need to parse it and configure the Daily transport with dial-in settings:

```python  theme={null}
from pipecat.runner.types import DailyDialinRequest, RunnerArguments
from pipecat.transports.daily.transport import DailyDialinSettings, DailyParams, DailyTransport

async def bot(runner_args: RunnerArguments):
    # Parse the dial-in request from the runner
    request = DailyDialinRequest.model_validate(runner_args.body)

    # Configure Daily transport with dial-in settings
    daily_dialin_settings = DailyDialinSettings(
        call_id=request.dialin_settings.call_id,
        call_domain=request.dialin_settings.call_domain,
    )

    transport = DailyTransport(
        runner_args.room_url,
        runner_args.token,
        "Daily PSTN Dial-in Bot",
        params=DailyParams(
            api_key=request.daily_api_key,
            api_url=request.daily_api_url,
            dialin_settings=daily_dialin_settings,
            audio_in_enabled=True,
            audio_out_enabled=True,
        ),
    )

    # Log caller information if available
    if request.dialin_settings.From:
        print(f"Handling call from: {request.dialin_settings.From}")

    # Your bot logic here
    await run_bot(transport)
```

### Configuring Daily Phone Numbers

To test dial-in locally, you need to:

1. **Configure a Daily phone number** in your Daily dashboard
2. **Set the webhook URL** to your public endpoint (e.g., via ngrok)
3. **Ensure `DAILY_API_KEY` is set** in your environment

```bash  theme={null}
# Example with ngrok
ngrok http 7860

# Configure webhook URL in Daily dashboard:
# https://your-subdomain.ngrok.io/daily-dialin-webhook
```

<Note>
  The `--dialin` flag requires the Daily transport (`-t daily`) and a valid
  `DAILY_API_KEY` environment variable.
</Note>

## Quick Reference

| Transport     | Command                                             | Access                                                       | Environment Variables                               |
| ------------- | --------------------------------------------------- | ------------------------------------------------------------ | --------------------------------------------------- |
| WebRTC        | `python bot.py`                                     | [http://localhost:7860/client](http://localhost:7860/client) | None                                                |
| Daily         | `python bot.py -t daily`                            | [http://localhost:7860](http://localhost:7860)               | `DAILY_API_KEY`, `DAILY_SAMPLE_ROOM_URL` (Optional) |
| Daily Direct  | `python bot.py -d`                                  | Direct connection                                            | `DAILY_API_KEY`, `DAILY_SAMPLE_ROOM_URL` (Optional) |
| Daily Dial-In | `python bot.py -t daily --dialin`                   | Phone calls via Daily PSTN                                   | `DAILY_API_KEY` (Required)                          |
| Twilio        | `python bot.py -t twilio -x proxy.ngrok.io`         | Phone calls                                                  | `TWILIO_ACCOUNT_SID`, `TWILIO_AUTH_TOKEN`           |
| Telnyx        | `python bot.py -t telnyx -x proxy.ngrok.io`         | Phone calls                                                  | `TELNYX_API_KEY`                                    |
| Plivo         | `python bot.py -t plivo -x proxy.ngrok.io`          | Phone calls                                                  | `PLIVO_AUTH_ID`, `PLIVO_AUTH_TOKEN`                 |
| Exotel        | `python bot.py -t exotel`                           | Phone calls                                                  | None                                                |
| ESP32 WebRTC  | `python bot.py -t webrtc --esp32 --host <ESP32_IP>` | ESP32 WebRTC connection                                      | None                                                |

## Examples

For practical examples of using the development runner with different transports, check out the following:

<Card title="Runner Examples" icon="code" href="https://github.com/pipecat-ai/pipecat-examples/tree/main/runner-examples">
  Explore the examples for different ways to use the development runner with
  various transports.
</Card>

---

# Source: https://docs.pipecat.ai/server/services/tts/hathora.md

# Source: https://docs.pipecat.ai/server/services/stt/hathora.md

> ## Documentation Index
> Fetch the complete documentation index at: https://docs.pipecat.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Hathora

> Speech-to-text service implementations hosted on Hathora

## Overview

Hathora is a hosting provider for several models for voice AI, which can be utilized under the single `HathoraSTTService`.

<CardGroup cols={2}>
  <Card title="Hathora STT API Reference" icon="code" href="https://reference-server.pipecat.ai/en/latest/api/pipecat.services.hahotra.stt.html">
    Pipecat's API methods for Hathora-hosted STT models
  </Card>

  <Card title="Example Implementation" icon="play" href="https://github.com/pipecat-ai/pipecat/blob/main/examples/foundational/07ag-interruptible-hathora.py">
    Complete example using Hathora-hosted models
  </Card>

  <Card title="Hathora Models Documentation" icon="book" href="https://models.hathora.dev/">
    Official Hathora documentation and features
  </Card>
</CardGroup>

## Installation

To use Hathora services, install the required dependencies:

```bash  theme={null}
pip install "pipecat-ai[hathora]"
```

## Prerequisites

### Hathora Account Setup

Before using Hathora STT services, you need:

1. **Hathora Account**: Sign up at [Hathora Models Console](https://models.hathora.dev/)
2. **API Key**: Generate an API token from your [Tokens page](https://models.hathora.dev/tokens)

### Hathora Model Specifier

The `HathoraSTTService` accepts a `model: str` parameter which corresponds to the model you would like to use.

You can find available specifiers [here](https://models.hathora.dev/)

---

# Source: https://docs.pipecat.ai/server/pipeline/heartbeats.md

> ## Documentation Index
> Fetch the complete documentation index at: https://docs.pipecat.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Pipeline Heartbeats

> Monitor pipeline health with heartbeat frames

## Overview

Pipeline heartbeats provide a way to monitor the health of your pipeline by sending periodic heartbeat frames through the system. When enabled, the pipeline will send heartbeat frames every second and monitor their progress through the pipeline.

## Enabling Heartbeats

Heartbeats can be enabled by setting `enable_heartbeats` to `True` in the `PipelineParams`:

```python  theme={null}
from pipecat.pipeline.task import PipelineParams, PipelineTask

pipeline = Pipeline([...])
params = params=PipelineParams(enable_heartbeats=True)
task = PipelineTask(pipeline, params)
```

## How It Works

When heartbeats are enabled:

1. The pipeline sends a `HeartbeatFrame` every second
2. The frame traverses through all processors in the pipeline, from source to sink
3. The pipeline monitors how long it takes for heartbeat frames to complete their journey
4. If a heartbeat frame isn't received within 5 seconds, a warning is logged

## Monitoring Output

The system will log:

* Trace-level logs showing heartbeat processing time
* Warning messages if heartbeats aren't received within the monitoring window

Example warning message:

```
WARNING    PipelineTask#1: heartbeat frame not received for more than 5.0 seconds
```

## Use Cases

Heartbeat monitoring is useful for:

* Detecting pipeline stalls or blockages
* Monitoring processing latency through the pipeline
* Identifying performance issues in specific processors
* Ensuring the pipeline remains responsive

## Configuration

The heartbeat system uses two key timing constants:

* `HEARTBEAT_SECONDS = 1.0` - Interval between heartbeat frames
* `HEARTBEAT_MONITOR_SECONDS = 10.0` - Time before warning if no heartbeat received

<Note>
  These values are currently fixed but may be configurable in future versions.
</Note>

---

# Source: https://docs.pipecat.ai/server/services/video/heygen.md

# Source: https://docs.pipecat.ai/server/services/transport/heygen.md

> ## Documentation Index
> Fetch the complete documentation index at: https://docs.pipecat.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# HeyGenTransport

> AI avatar video generation service for creating interactive conversational avatars

## Overview

`HeyGenTransport` enables your Pipecat bot to join the same virtual room as a HeyGen avatar and human participants. The transport integrates with the HeyGen [LiveAvatar](https://www.liveavatar.com/) platform to create interactive AI-powered video avatars that respond naturally in real-time conversations. The service handles bidirectional audio/video streaming, avatar animations, voice activity detection, and conversation interruptions to deliver engaging conversational AI experiences with lifelike visual presence.

When used, the Pipecat bot connects to a LiveKit room alongside the HeyGen avatar and user. The bot receives audio input from participants, processes it through your pipeline, and sends TTS audio to the HeyGen avatar for synchronized video rendering.

<CardGroup cols={2}>
  <Card title="HeyGen Trasnport API Reference" icon="code" href="https://reference-server.pipecat.ai/en/latest/api/pipecat.transports.heygen.html">
    Pipecat's API methods for HeyGen video integration
  </Card>

  <Card title="Example Implementation" icon="play" href="https://github.com/pipecat-ai/pipecat/blob/main/examples/foundational/43-heygen-transport.py">
    Complete example with interactive avatar
  </Card>

  <Card title="HeyGen Documentation" icon="book" href="https://docs.liveavatar.com/docs/getting-started">
    Official HeyGen API documentation and guides
  </Card>

  <Card title="HeyGen Platform" icon="microphone" href="https://www.liveavatar.com/">
    Access interactive avatars and API keys
  </Card>
</CardGroup>

## Installation

To use HeyGen services, install the required dependency:

```bash  theme={null}
pip install "pipecat-ai[heygen]"
```

## Prerequisites

### HeyGen Account Setup

Before using HeyGen video services, you need:

1. **HeyGen Account**: Sign up at [HeyGen Platform](https://app.liveavatar.com/signin)
2. **API Key**: Generate an API key from your account dashboard
3. **Avatar Selection**: Choose from available interactive avatars
4. **Streaming Setup**: Configure real-time avatar streaming capabilities

### Required Environment Variables

* `HEYGEN_LIVE_AVATAR_API_KEY`: Your HeyGen LiveAvatar API key for authentication

---

# Source: https://docs.pipecat.ai/deployment/pipecat-cloud/security/hipaa.md

> ## Documentation Index
> Fetch the complete documentation index at: https://docs.pipecat.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# HIPAA

> HIPAA compliance for Pipecat Cloud

Pipecat Cloud provides HIPAA-compliant infrastructure for healthcare applications that need to process Protected Health Information (PHI).

## Infrastructure Compliance

Pipecat Cloud's infrastructure meets HIPAA controls for:

* Data processing
* Data transmission
* Secure handling of PHI

## Data Logging with HIPAA Controls

When HIPAA controls are enabled for your account:

**No application data is logged** - This includes conversation history and any PHI that may be contained in your application logs.

**Rationale**: Pipecat follows the HIPAA "minimum necessary" rule by logging only what's essential to run the service, rather than storing your PHI data.

**Alternative logging**: Configure an independent logging solution to route your application logs to your own secure log storage system.

## Business Associate Agreement (BAA)

Daily can sign a Business Associate Agreement covering:

* Pipecat Cloud
* Daily WebRTC infrastructure
* Daily transcription services

A single BAA covers all these services.

## Getting Started

Contact us at [help@daily.co](mailto:help@daily.co) to:

* Enable HIPAA controls for your account
* Sign a Business Associate Agreement
* Learn more about HIPAA compliance features

---

# Source: https://docs.pipecat.ai/client/react/hooks.md

> ## Documentation Index
> Fetch the complete documentation index at: https://docs.pipecat.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Hooks

> React hooks for accessing Pipecat client functionality

The Pipecat React SDK provides hooks for accessing client functionality, managing media devices, and handling events.

## usePipecatClient

Provides access to the `PipecatClient` instance originally passed to `PipecatClientProvider`.

```jsx  theme={null}
import { usePipecatClient } from "@pipecat-ai/client-react";

function MyComponent() {
  const pcClient = usePipecatClient();

  await pcClient.startBotAndConnect({
    endpoint: '/api/start',
    requestData: {
      // Any custom data your /start endpoint requires
    }
  });
}
```

## useRTVIClientEvent

Allows subscribing to RTVI client events. It is advised to wrap handlers with `useCallback`.

```jsx  theme={null}
import { useCallback } from "react";
import { RTVIEvent, TransportState } from "@pipecat-ai/client-js";
import { useRTVIClientEvent } from "@pipecat-ai/client-react";

function EventListener() {
  useRTVIClientEvent(
    RTVIEvent.TransportStateChanged,
    useCallback((transportState: TransportState) => {
      console.log("Transport state changed to", transportState);
    }, [])
  );
}
```

**Arguments**

<ParamField path="event" type="RTVIEvent" required />

<ParamField path="handler" type="function" required />

## usePipecatClientMediaDevices

Manage and list available media devices.

```jsx  theme={null}
import { usePipecatClientMediaDevices } from "@pipecat-ai/client-react";

function DeviceSelector() {
  const {
    availableCams,
    availableMics,
    selectedCam,
    selectedMic,
    updateCam,
    updateMic,
  } = usePipecatClientMediaDevices();

  return (
    <>
      <select
        name="cam"
        onChange={(ev) => updateCam(ev.target.value)}
        value={selectedCam?.deviceId}
      >
        {availableCams.map((cam) => (
          <option key={cam.deviceId} value={cam.deviceId}>
            {cam.label}
          </option>
        ))}
      </select>
      <select
        name="mic"
        onChange={(ev) => updateMic(ev.target.value)}
        value={selectedMic?.deviceId}
      >
        {availableMics.map((mic) => (
          <option key={mic.deviceId} value={mic.deviceId}>
            {mic.label}
          </option>
        ))}
      </select>
    </>
  );
}
```

## usePipecatClientMediaTrack

Access audio and video tracks.

```jsx  theme={null}
import { usePipecatClientMediaTrack } from "@pipecat-ai/client-react";

function MyTracks() {
  const localAudioTrack = usePipecatClientMediaTrack("audio", "local");
  const botAudioTrack = usePipecatClientMediaTrack("audio", "bot");
}
```

**Arguments**

<ParamField path="trackType" type="'audio' | 'video'" required />

<ParamField path="participantType" type="'bot' | 'local'" required />

## usePipecatClientTransportState

Returns the current transport state.

```jsx  theme={null}
import { usePipecatClientTransportState } from "@pipecat-ai/client-react";

function ConnectionStatus() {
  const transportState = usePipecatClientTransportState();
}
```

## usePipecatClientCamControl

Controls the local participant's camera state.

```jsx  theme={null}
import { usePipecatClientCamControl } from "@pipecat-ai/client-react";
function CamToggle() {
  const { enableCam, isCamEnabled } = usePipecatClientCamControl();

  return (
    <button onClick={() => enableCam(!isCamEnabled)}>
      {isCamEnabled ? "Disable Camera" : "Enable Camera"}
    </button>
  );
}
```

## usePipecatClientMicControl

Controls the local participant's microphone state.

```jsx  theme={null}
import { usePipecatClientMicControl } from "@pipecat-ai/client-react";
function MicToggle() {
  const { enableMic, isMicEnabled } = usePipecatClientMicControl();

  return (
    <button onClick={() => enableMic(!isMicEnabled)}>
      {isMicEnabled ? "Disable Microphone" : "Enable Microphone"}
    </button>
  );
}
```

---

# Source: https://docs.pipecat.ai/server/services/tts/hume.md

> ## Documentation Index
> Fetch the complete documentation index at: https://docs.pipecat.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Hume

> Text-to-speech service using Hume AI's expressive Octave models with word timestamps

## Overview

Hume provides expressive text-to-speech synthesis using their Octave models, which adapt pronunciation, pitch, speed, and emotional style based on context. `HumeTTSService` offers real-time streaming with word-level timestamps, custom voice support, and advanced synthesis controls including acting instructions, speed adjustment, and trailing silence configuration.

<CardGroup cols={2}>
  <Card title="Hume TTS API Reference" icon="code" href="https://reference-server.pipecat.ai/en/latest/api/pipecat.services.hume.tts.html">
    Pipecat's API methods for Hume TTS integration
  </Card>

  <Card title="Example Implementation" icon="play" href="https://github.com/pipecat-ai/pipecat/blob/main/examples/foundational/07ze-interruptible-hume.py">
    Complete example with word timestamps and interruption handling
  </Card>

  <Card title="Hume Documentation" icon="book" href="https://dev.hume.ai/docs/text-to-speech-tts/overview">
    Official Hume TTS API documentation and features
  </Card>

  <Card title="Voice Library" icon="microphone" href="https://dev.hume.ai/docs/text-to-speech-tts/voices">
    Browse and manage available voices
  </Card>
</CardGroup>

## Installation

To use Hume services, install the required dependencies:

```bash  theme={null}
pip install "pipecat-ai[hume]"
```

## Prerequisites

### Hume Account Setup

Before using Hume TTS services, you need:

1. **Hume Account**: Sign up at [Hume AI](https://www.hume.ai/)
2. **API Key**: Generate an API key from your account dashboard
3. **Voice Selection**: Choose voice IDs from the voice library or create custom voices

### Required Environment Variables

* `HUME_API_KEY`: Your Hume API key for authentication

---

# Source: https://docs.pipecat.ai/server/utilities/filters/identify-filter.md

> ## Documentation Index
> Fetch the complete documentation index at: https://docs.pipecat.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# IdentityFilter

> Processor that passes all frames through without modification

## Overview

`IdentityFilter` is a simple pass-through processor that forwards all frames without any modification or filtering. It acts as a transparent layer in your pipeline, allowing all frames to flow through unchanged.

<Tip>
  Check out Observers for an option that delivers similar functionality but
  doesn't require a processor to reside in the Pipeline.
</Tip>

## Constructor Parameters

The `IdentityFilter` constructor accepts no specific parameters beyond those inherited from `FrameProcessor`.

## Functionality

When a frame passes through the processor, it is immediately forwarded in the same direction with no changes. This applies to all frame types and both directions (upstream and downstream).

## Use Cases

While functionally equivalent to having no filter at all, `IdentityFilter` can be useful in several scenarios:

* Testing `ParallelPipeline` configurations to ensure frames aren't duplicated
* Acting as a placeholder where a more complex filter might be added later
* Monitoring frame flow in pipelines by adding logging in subclasses
* Creating a base class for more complex conditional filters

## Usage Example

```python  theme={null}
from pipecat.processors.filters import IdentityFilter

# Create an identity filter
pass_through = IdentityFilter()

# Add to pipeline
pipeline = Pipeline([
    source,
    pass_through,  # All frames pass through unchanged
    destination
])
```

## Frame Flow

```mermaid  theme={null}
graph LR
    A[Input Frame] --> B[IdentityFilter] --> C[Output Frame]
```

## Notes

* Simplest possible filter implementation
* Passes all frames through without modification
* Useful in testing parallel pipelines
* Can serve as a placeholder or base class
* Zero overhead in normal operation

---

# Source: https://docs.pipecat.ai/cli/init.md

> ## Documentation Index
> Fetch the complete documentation index at: https://docs.pipecat.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# init

> Scaffold a new Pipecat project with an interactive setup wizard

Create a new Pipecat project interactively with guided setup for bot type, transport, AI services, and deployment options.

**Usage:**

```shell  theme={null}
pipecat init [OPTIONS]
```

**Options:**

<ParamField path="--output / -o" type="string">
  Output directory where files will be saved. Defaults to current directory.
</ParamField>

## Interactive Setup

The CLI guides you through selecting:

* **Bot type and client framework** - Phone, web (Next.js, Vite, Vanilla JS), or mobile
* **Transport provider** - Daily, Twilio, Telnyx, Plivo, Exotel
* **Pipeline mode** - Cascade or Realtime
* **AI services** - STT, LLM, and TTS providers
* **Optional features** - Additional capabilities for your bot
* **Deployment target** - Local development or Pipecat Cloud

## Examples

### Basic Usage

```shell  theme={null}
# Initialize in current directory
pipecat init
```

### Specify Output Directory

```shell  theme={null}
# Initialize in a specific directory
pipecat init --output my-bot
```

## Generated Project Structure

```
mybot/
├── server/                 # Python bot server
│   ├── bot.py              # Main bot implementation
│   ├── pyproject.toml      # Python dependencies
│   ├── .env.example        # Environment variables template
│   ├── Dockerfile          # Container image (if cloud enabled)
│   └── pcc-deploy.toml     # Deployment config (if cloud enabled)
├── client/                 # Web client (if generated)
│   ├── src/
│   ├── package.json
│   └── ...
├── .gitignore
└── README.md               # Project setup instructions
```

---

# Source: https://docs.pipecat.ai/server/utilities/turn-management/interruption-strategies.md

> ## Documentation Index
> Fetch the complete documentation index at: https://docs.pipecat.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Interruption Strategies

> Configure when users can interrupt the bot to prevent unwanted interruptions from brief affirmations

<Warning>
  DEPRECATED Interruption strategies have been deprecated in favor of [User Turn
  Strategies](/server/utilities/turn-management/user-turn-strategies).
</Warning>

## Overview

Interruption strategies allow you to control when users can interrupt the bot during speech. By default, any user speech immediately interrupts the bot, but this can be problematic when users engage in backchanneling—brief vocal responses like "yeah", "okay", or "mm-hmm" that indicate they're listening without intending to interrupt.

With interruption strategies, you can require users to meet specific criteria (such as speaking a minimum number of words or reaching a certain audio volume) before their speech will interrupt the bot, creating a more natural conversation flow.

<Tip>
  Want to try it out? Check out the [interruption strategies foundational
  demo](https://github.com/pipecat-ai/pipecat/blob/main/examples/foundational/42-interruption-config.py)
</Tip>

## Configuration

Interruption strategies are configured via the `interruption_strategies` parameter in `PipelineParams`. When specified, the normal immediate interruption behavior is replaced with conditional interruption based on your criteria.

<ParamField path="interruption_strategies" type="List[BaseInterruptionStrategy]" default="[]">
  List of interruption strategies to apply. When multiple strategies are
  provided, the first one that evaluates to true will trigger the interruption.
  If empty, normal interruption behavior applies.
</ParamField>

## Base Strategy Interface

All interruption strategies inherit from `BaseInterruptionStrategy`, which provides a common interface for evaluating interruption conditions.

<ParamField path="append_audio(audio, sample_rate)" type="async method">
  Appends audio data to the strategy for analysis. Not all strategies handle
  audio.
</ParamField>

<ParamField path="append_text(text)" type="async method">
  Appends text to the strategy for analysis. Not all strategies handle text.
</ParamField>

<ParamField path="should_interrupt()" type="async method">
  Called when the user stops speaking to determine if interruption should occur
  based on accumulated audio and/or text.
</ParamField>

<ParamField path="reset()" type="async method">
  Resets accumulated text and/or audio data.
</ParamField>

## Available Strategies

### MinWordsInterruptionStrategy

Requires users to speak a minimum number of words before interrupting the bot.

<ParamField path="min_words" type="int" required>
  Minimum number of words the user must speak to interrupt the bot. Must be
  greater than 0.
</ParamField>

```python  theme={null}
from pipecat.audio.interruptions.min_words_interruption_strategy import MinWordsInterruptionStrategy

strategy = MinWordsInterruptionStrategy(min_words=3)
```

## How It Works

When interruption strategies are configured:

1. **Bot not speaking**: User speech interrupts immediately (normal behavior)
2. **Bot speaking**: User speech and audio are collected and fed to strategies
3. **User stops speaking**: Strategies are evaluated in order
4. **First match wins**: The first strategy that returns `True` triggers interruption
5. **No matches**: User speech is discarded

The system automatically handles both audio and text input:

* Audio frames (`InputAudioRawFrame`) are fed to `append_audio()`
* Transcription text is fed to `append_text()`
* Strategies can use either or both data types

## Usage Examples

### Basic Word Count Interruption

Require users to speak at least 3 words to interrupt the bot:

```python  theme={null}
from pipecat.audio.interruptions.min_words_interruption_strategy import MinWordsInterruptionStrategy
from pipecat.pipeline.task import PipelineParams, PipelineTask

task = PipelineTask(
    pipeline,
    params=PipelineParams(
        allow_interruptions=True,
        interruption_strategies=[MinWordsInterruptionStrategy(min_words=3)]
    )
)
```

### Multiple Strategies with Priority

Strategies are evaluated in order, with the first match triggering interruption:

```python  theme={null}
# Prioritize word count, then volume (hypothetical future strategy)
task = PipelineTask(
    pipeline,
    params=PipelineParams(
        allow_interruptions=True,
        interruption_strategies=[
            MinWordsInterruptionStrategy(min_words=2),        # Check first
            # VolumeInterruptionStrategy(min_volume=0.8),     # Your custom strategy
        ]
    )
)
```

## Behavior Comparison

| Scenario                                      | Without Strategy         | With MinWordsInterruptionStrategy(min\_words=3) |
| --------------------------------------------- | ------------------------ | ----------------------------------------------- |
| User says "okay" while bot speaks             | ✅ Interrupts immediately | ❌ Ignored (only 1 word)                         |
| User says "yes that's right" while bot speaks | ✅ Interrupts immediately | ✅ Interrupts (3 words)                          |
| User speaks while bot is silent               | ✅ Processed immediately  | ✅ Processed immediately                         |

## Notes

* Interruption strategies only affect behavior when the bot is actively speaking
* When the bot is not speaking, user input is processed immediately regardless of strategy configuration
* The `allow_interruptions` parameter must be `True` for interruption strategies to work
* User speech that doesn't meet interruption criteria is discarded, not queued
* Strategies are evaluated in order - first match wins
* Both audio and text data are automatically fed to strategies based on their implementation
* Word counting uses simple whitespace splitting for word boundaries

---

# Source: https://docs.pipecat.ai/server/services/serializers/introduction.md

# Source: https://docs.pipecat.ai/server/introduction.md

# Source: https://docs.pipecat.ai/server/frameworks/rtvi/introduction.md

# Source: https://docs.pipecat.ai/getting-started/introduction.md

# Source: https://docs.pipecat.ai/deployment/pipecat-cloud/introduction.md

# Source: https://docs.pipecat.ai/client/react/introduction.md

# Source: https://docs.pipecat.ai/client/react-native/introduction.md

# Source: https://docs.pipecat.ai/client/js/introduction.md

# Source: https://docs.pipecat.ai/client/ios/introduction.md

# Source: https://docs.pipecat.ai/client/introduction.md

# Source: https://docs.pipecat.ai/client/c++/introduction.md

# Source: https://docs.pipecat.ai/client/android/introduction.md

> ## Documentation Index
> Fetch the complete documentation index at: https://docs.pipecat.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# SDK Introduction

> Build Android applications with Pipecat's Kotlin client library

The Pipecat Android SDK provides a Kotlin implementation for building voice and multimodal AI applications on Android. It handles:

* Real-time audio and video streaming
* Bot communication and state management
* Media device handling
* Event handling

## Installation

Add the dependency for your chosen transport to your `build.gradle` file. For example, to use the Daily transport:

```gradle  theme={null}
implementation "ai.pipecat:daily-transport:1.0.3"
```

## Example

Here's a simple example using Daily as the transport layer.

```kotlin  theme={null}
val callbacks = object : PipecatEventCallbacks() {

    override fun onBackendError(message: String) {
        Log.e(TAG, "Error from backend: $message")
    }

    // ...
}

val options = PipecatClientOptions(callbacks = callbacks)

val client: PipecatClientDaily = PipecatClient(DailyTransport(context), options)

// Kotlin coroutines:
client.startBotAndConnect(startBotParams).await()

// Or, callbacks:
client.startBotAndConnect(startBotParams).withCallback {
    // ...
}
```

## Documentation

<CardGroup cols={2}>
  <Card title="API Reference" icon="book" href="https://docs-android.rtvi.ai/">
    Complete SDK API documentation
  </Card>

  <Card title="Daily Transport" icon="network-wired" href="./transports/daily">
    WebRTC implementation using Daily
  </Card>
</CardGroup>

---

# Source: https://docs.pipecat.ai/server/services/tts/inworld.md

> ## Documentation Index
> Fetch the complete documentation index at: https://docs.pipecat.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Inworld

> Text-to-speech service using Inworld AI's TTS APIs

## Overview

Inworld provides high-quality, low-latency speech synthesis via two implementation types: `InworldTTSService` for real-time, minimal-latency use-cases through websockets and `InworldHttpTTSService` for streaming and non-streaming use-cases over HTTP. Featuring support for 12+ languages, timestamps, custom pronunciation and instant voice cloning.

<CardGroup cols={2}>
  <Card title="Inworld TTS API Reference" icon="code" href="https://reference-server.pipecat.ai/en/latest/api/pipecat.services.inworld.tts.html">
    Pipecat's API methods for Inworld TTS integration
  </Card>

  <Card title="Example Implementation (Websockets)" icon="play" href="https://github.com/pipecat-ai/pipecat/blob/main/examples/foundational/07zb-interruptible-inworld.py">
    Complete example with Inworld TTS
  </Card>

  <Card title="Inworld Documentation" icon="book" href="https://docs.inworld.ai/docs/tts/tts">
    Official Inworld TTS API documentation
  </Card>

  <Card title="Inworld Portal" icon="microphone" href="https://platform.inworld.ai/">
    Create and manage voice models
  </Card>
</CardGroup>

## Installation

To use Inworld services, no additional dependencies are required beyond the base installation:

```bash  theme={null}
pip install "pipecat-ai"
```

## Prerequisites

### Inworld Account Setup

Before using Inworld TTS services, you need:

1. **Inworld Account**: Sign up at [Inworld Studio](https://studio.inworld.ai/)
2. **API Key**: Generate an API key from your account dashboard
3. **Voice Selection**: Choose from available voice models

### Required Environment Variables

* `INWORLD_API_KEY`: Your Inworld API key for authentication

---

# Source: https://docs.pipecat.ai/server/utilities/extensions/ivr.md

# Source: https://docs.pipecat.ai/guides/fundamentals/ivr.md

> ## Documentation Index
> Fetch the complete documentation index at: https://docs.pipecat.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# IVR Navigation

> Automatically navigate phone system menus using AI-powered decision making

## Overview

The `IVRNavigator` enables your bot to automatically navigate Interactive Voice Response (IVR) phone systems to reach specific goals. Instead of manually programming navigation paths, you provide an end goal and the bot handles the complex decision-making required to traverse phone menus using DTMF tones and conversational responses.

## How IVR Navigation Works

The IVRNavigator combines several intelligent capabilities:

1. **Goal-oriented navigation**: You specify what you want to accomplish (e.g., "reach billing support")
2. **Automatic classification**: Detects whether incoming audio is an IVR system or human conversation
3. **Smart decision making**: Analyzes menu options and selects the best path toward your goal
4. **Multi-modal responses**: Uses both DTMF tones for menu selection, natural language for prompts, and waits when no input is appropriate
5. **Status tracking**: Monitors progress and reports completion, waiting, or stuck states

## Navigation Outcomes

The `IVRNavigator` can reach several outcomes during navigation:

### Completed ✅

The navigator successfully reaches its goal and is ready for the next step. Common actions:

* [Terminate the pipeline](/guides/learn/pipeline-termination) if the goal is complete
* Transfer the call to a human agent
* Allow your bot to start a conversation with the reached department

```python  theme={null}
@ivr_navigator.event_handler("on_ivr_status_changed")
async def handle_ivr_status(processor, status):
    if status == IVRStatus.COMPLETED:
        logger.info("Successfully navigated to target department")
        # Start conversation
        # Replace the context with new messages for the upcoming conversation
        messages = [{"role": "system", "content": "You are a helpful customer service assistant."}]
        await task.queue_frame(LLMMessagesUpdateFrame(messages))
        # Adjust VAD stop_secs for conversation
        vad_params = VADParams(stop_secs=0.8)
        await task.queue_frame(VADParamsUpdateFrame(vad_params))
```

### Stuck ⚠️

The navigator cannot find a path forward in the IVR system. This might happen when:

* Required information (account numbers, PINs) isn't available
* The menu options don't align with the stated goal
* The system encounters errors or invalid selections

```python  theme={null}
@ivr_navigator.event_handler("on_ivr_status_changed")
async def handle_ivr_status(processor, status):
    if status == IVRStatus.STUCK:
        logger.warning("IVR navigation stuck - terminating call")
        # Log the issue and clean up
        await log_navigation_failure()
        await task.queue_frame(EndFrame())
```

## Flexible Entry Points

One of the `IVRNavigator`'s key features is flexible entry point handling. When you dial a phone number, you might encounter either:

* An IVR system with menu options
* A direct connection to a human

The navigator automatically detects which scenario occurs and emits appropriate events:

### Human Conversation Detected

When a human answers instead of an IVR system, an `on_conversation_detected` event is emitted. You can handle that event to transition to a conversation.

```python  theme={null}
@ivr_navigator.event_handler("on_conversation_detected")
async def on_conversation_detected(processor, conversation_history):
    # Set up conversation prompt and preserve conversation history
    messages = [
        {
            "role": "system",
            "content": "You are an assistant calling to check on the prescription for John Smith, date of birth 01/01/1990.",
        }
    ]

    # Add preserved conversation history if available
    if conversation_history:
        messages.extend(conversation_history)

    await task.queue_frame(LLMMessagesUpdateFrame(messages=messages, run_llm=True))
```

Note that the `on_conversation_detected` event also emits a `conversation_history` parameter that contains the previous conversation history. This allows you to build a prompt that includes your conversation system prompt plus any conversation history up to that point in time.

### IVR System Detected

When an IVR system is detected, the IVR Navigator automatically transitions into navigation mode:

* **System prompt**: Updates to use your specified navigation goal
* **VAD timing**: Adjusts to `stop_secs=2.0` (or your custom `ivr_vad_params`) to allow time for complete menu announcements
* **Navigation logic**: Begins analyzing menu options and making decisions toward your goal

No additional code is required. The navigator handles this transition automatically.

**Optional Event Handling**

If you need to log the detection or perform custom actions, you can handle the `on_ivr_status_changed` event:

```python  theme={null}
@ivr_navigator.event_handler("on_ivr_status_changed")
async def on_ivr_status_changed(processor, status):
    if status == IVRStatus.DETECTED:
        logger.info("IVR system detected - beginning navigation")
        # Optional: Add analytics tracking, custom setup, etc.
```

## Basic Implementation

### Step 1: Create the IVR Navigator

```python  theme={null}
from pipecat.extensions.ivr.ivr_navigator import IVRNavigator
from pipecat.audio.vad.vad_analyzer import VADParams

# Define your navigation goal
ivr_goal = "Navigate to the billing department to discuss my account balance"

# Create navigator with extended response time for IVR systems
ivr_vad_params = VADParams(stop_secs=2.0)  # Longer wait for IVR menus

ivr_navigator = IVRNavigator(
    llm=your_llm_service,
    ivr_prompt=ivr_goal,
    ivr_vad_params=ivr_vad_params
)
```

### Step 2: Set Up Event Handlers

```python  theme={null}
from pipecat.frames.frames import LLMMessagesUpdateFrame
from pipecat.extensions.ivr.ivr_navigator import IVRStatus

@ivr_navigator.event_handler("on_conversation_detected")
async def on_conversation_detected(processor, conversation_history):
    """Handle when a human conversation is detected instead of IVR"""
    logger.info("Human conversation detected")

    # Set up conversation context
    messages = [
        {"role": "system", "content": "You are a customer service representative."}
    ]
    if conversation_history:
        messages.extend(conversation_history)

    await task.queue_frame(LLMMessagesUpdateFrame(messages=messages, run_llm=True))

@ivr_navigator.event_handler("on_ivr_status_changed")
async def on_ivr_status_changed(processor, status):
    """Handle IVR navigation status changes"""
    if status == IVRStatus.COMPLETED:
        logger.info("IVR navigation completed successfully")
        # Your success handling logic here

    elif status == IVRStatus.STUCK:
        logger.warning("IVR navigation got stuck")
        # Your error handling logic here
        await handle_navigation_failure()
```

### Step 3: Add to Pipeline

Add the IVR Navigator to your pipeline in the place where you would normally add the LLM. The IVR Navigator contains your LLM and will perform the same functions as an LLM would, but in addition it will navigate the IVR system.

```python  theme={null}
from pipecat.pipeline.pipeline import Pipeline

pipeline = Pipeline([
    transport.input(),
    stt_service,
    ivr_navigator,  # Add the navigator to your pipeline
    tts_service,
    transport.output()
])
```

## VAD Parameter Optimization

The IVRNavigator automatically optimizes Voice Activity Detection (VAD) parameters for different scenarios:

### IVR Navigation Mode

* **Default**: `stop_secs=2.0`
* **Purpose**: Allows time to hear complete menu options before responding
* **Result**: Higher navigation success rates

### Conversation Mode

* **Recommended**: `stop_secs=0.8`
* **Purpose**: Enables natural conversation flow with quick responses
* **Implementation**: Push `VADParamsUpdateFrame` when transitioning to conversation

## Next Steps

<CardGroup cols={2}>
  <Card title="Pipeline Termination" icon="stop" href="/guides/learn/pipeline-termination">
    Learn how to properly terminate pipelines when IVR navigation completes
  </Card>

  <Card title="Pipecat Flows" icon="flow" href="/guides/features/pipecat-flows">
    Integrate structured conversation flows after successful IVR navigation
  </Card>
</CardGroup>

The IVRNavigator provides a powerful foundation for automating phone system interactions, allowing your bots to handle the complex task of menu navigation while you focus on the core conversation logic.

---

# Source: https://docs.pipecat.ai/server/utilities/audio/koala-filter.md

> ## Documentation Index
> Fetch the complete documentation index at: https://docs.pipecat.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# KoalaFilter

> Audio noise reduction filter using Koala AI technology from Picovoice

## Overview

`KoalaFilter` is an audio processor that reduces background noise in real-time audio streams using Koala Noise Suppression technology from Picovoice. It inherits from `BaseAudioFilter` and processes audio frames to improve audio quality by removing unwanted noise.

To use Koala, you need a Picovoice access key. Get started at [Picovoice Console](https://console.picovoice.ai/signup).

## Installation

The Koala filter requires additional dependencies:

```bash  theme={null}
pip install "pipecat-ai[koala]"
```

You'll also need to set up your Koala access key as an environment variable: `KOALA_ACCESS_KEY`

## Constructor Parameters

<ParamField path="access_key" type="str" required>
  Picovoice access key for using the Koala noise suppression service
</ParamField>

## Input Frames

<ParamField path="FilterEnableFrame" type="Frame">
  Specific control frame to toggle filtering on/off

  ```python  theme={null}
  from pipecat.frames.frames import FilterEnableFrame

  # Disable noise reduction
  await task.queue_frame(FilterEnableFrame(False))

  # Re-enable noise reduction
  await task.queue_frame(FilterEnableFrame(True))
  ```
</ParamField>

## Usage Example

```python  theme={null}
from pipecat.audio.filters.koala_filter import KoalaFilter

transport = DailyTransport(
    room_url,
    token,
    "Respond bot",
    DailyParams(
        audio_in_filter=KoalaFilter(access_key=os.getenv("KOALA_ACCESS_KEY")), # Enable Koala noise reduction
        audio_in_enabled=True,
        audio_out_enabled=True,
    ),
)
```

## Notes

* Requires Picovoice access key
* Supports real-time audio processing
* Handles 16-bit PCM audio format
* Can be dynamically enabled/disabled
* Maintains audio quality while reducing noise
* Efficient processing for low latency
* Automatically handles audio frame buffering
* Sample rate must match Koala's required sample rate

---

# Source: https://docs.pipecat.ai/server/utilities/audio/krisp-filter.md

# KrispFilter

> Audio noise reduction filter using Krisp AI technology

<Warning>
  The `KrispFilter` is deprecated and will be removed in a future version.
  Please use the `KrispVivaFilter`, which is the latest model from Krisp. See
  the [Krisp VIVA guide](/server/utilities/audio/krisp-viva-filter) for more
  information.
</Warning>

## Overview

`KrispFilter` is an audio processor that reduces background noise in real-time audio streams using Krisp AI technology. It inherits from `BaseAudioFilter` and processes audio frames to improve audio quality by removing unwanted noise.

To use Krisp, you need a Krisp SDK license. Get started at [Krisp.ai](https://krisp.ai/developers/).

<Tip>
  Looking for help getting started with Krisp and Pipecat? Checkout our [Krisp
  noise cancellation guide](/guides/features/krisp).
</Tip>

## Installation

The Krisp filter requires additional dependencies:

```bash  theme={null}
pip install "pipecat-ai[krisp]"
```

## Environment Variables

You need to provide the path to the Krisp model. This can either be done by setting the `KRISP_MODEL_PATH` environment variable or by setting the `model_path` in the constructor.

## Constructor Parameters

<ParamField path="sample_type" type="str" default="PCM_16">
  Audio sample type format
</ParamField>

<ParamField path="channels" type="int" default="1">
  Number of audio channels
</ParamField>

<ParamField path="model_path" type="str" default="None">
  Path to the Krisp model file.

  You can set the `model_path` directly. Alternatively, you can set the `KRISP_MODEL_PATH` environment variable to the model file path.
</ParamField>

## Input Frames

<ParamField path="FilterEnableFrame" type="Frame">
  Specific control frame to toggle filtering on/off

  ```python  theme={null}
  from pipecat.frames.frames import FilterEnableFrame

  # Disable noise reduction
  await task.queue_frame(FilterEnableFrame(False))

  # Re-enable noise reduction
  await task.queue_frame(FilterEnableFrame(True))
  ```
</ParamField>

## Usage Example

```python  theme={null}
from pipecat.audio.filters.krisp_filter import KrispFilter

transport = DailyTransport(
    room_url,
    token,
    "Respond bot",
    DailyParams(
        audio_in_filter=KrispFilter(), # Enable Krisp noise reduction
        audio_in_enabled=True,
        audio_out_enabled=True,
        vad_analyzer=SileroVADAnalyzer(),
    ),
)
```

## Audio Flow

```mermaid  theme={null}
graph TD
    A[AudioRawFrame] --> B[KrispFilter]
    B[KrispFilter] --> C[VAD]
    C[VAD] --> D[STT]
```

## Notes

* Requires Krisp SDK and model file to be available
* Supports real-time audio processing
* Supports additional features like background voice removal
* Handles PCM\_16 audio format
* Thread-safe for pipeline processing
* Can be dynamically enabled/disabled
* Maintains audio quality while reducing noise
* Efficient processing for low latency


---

> To find navigation and other pages in this documentation, fetch the llms.txt file at: https://docs.pipecat.ai/llms.txt

---

# Source: https://docs.pipecat.ai/deployment/pipecat-cloud/guides/krisp-noise-cancellation.md

# Krisp Noise Cancellation

> Eliminate background noise and voices for more reliable agent interactions

<Warning>
  The `KrispFilter` is deprecated and will be removed in a future version.
  Please use the `KrispVivaFilter`, which is the latest model from Krisp. See
  the [Krisp VIVA guide](/deployment/pipecat-cloud/guides/krisp-viva) for more
  information.
</Warning>

## Overview

Voice agents are vulnerable to background noise, especially other voices that trigger false interruptions. When this happens, your bot stops mid-sentence unnecessarily, creating a poor user experience. Pipecat Cloud's Krisp integration processes incoming audio to remove ambient noise and background voices before they reach your agent, resulting in cleaner audio input with fewer false interruptions and better transcription.

<Info>
  Krisp processes the audio before it reaches your agent's speech recognition
  and VAD systems, helping to ensure that only the intended user's voice
  triggers interruptions.
</Info>

## Enabling Krisp

Enabling Krisp with Pipecat Cloud involves the following steps:

### 1. Use the Pipecat Base Image

Krisp dependencies are pre-installed in the official Pipecat base image. You don't need to add any Krisp-related packages to your `requirements.txt` file.

<Note>
  Krisp support is available in the `dailyco/pipecat-base` image starting with
  version `0.0.6`.
</Note>

### 2. Add the Krisp Filter to Your Transport

In your bot code, import and add the `KrispFilter` to your transport:

```python  theme={null}
from pipecat.audio.filters.krisp_filter import KrispFilter
from pipecat.audio.vad.silero import SileroVADAnalyzer
from pipecat.transports.daily.transport import DailyParams, DailyTransport

transport = DailyTransport(
    room_url,
    token,
    "Voice AI Bot",
    DailyParams(
        audio_in_enabled=True,
        audio_in_filter=KrispFilter(),  # Add Krisp filter here
        audio_out_enabled=True,
        vad_analyzer=SileroVADAnalyzer(),
    ),
)
```

### 3. Enable Krisp in Your Deployment

You can enable Krisp when deploying your agent through:

<Tabs>
  <Tab title="CLI">
    ```bash  theme={null}
    pipecat cloud deploy --enable-krisp true
    ```
  </Tab>

  <Tab title="Config File">
    In your `pcc-deploy.toml`:

    ```toml  theme={null}
    agent_name = "voice-starter"
    image = "your-repo/voice-starter:0.1"
    secret_set = "voice-starter-secrets"
    enable_krisp = true

    [scaling]
    min_agents = 0

    ```
  </Tab>

  <Tab title="Dashboard">
    In the Pipecat Cloud dashboard:

    1. Navigate to your agent
    2. Go to Settings > Agent Configuration
    3. Toggle "Krisp Noise Cancellation" on
    4. Save your changes
  </Tab>
</Tabs>

<Check>
  Once configured, your agent will process incoming audio through Krisp's noise
  cancellation.
</Check>

## Testing Your Integration

To verify Krisp is working:

1. Deploy your agent with Krisp enabled
2. Connect to your agent in an environment with background noise; try turning on a TV or podcast
3. Observe whether background noises and voices affect your agent's behavior

## Pricing

Krisp processing is billed based on active session minutes:

| Usage Tier                      | Price               |
| ------------------------------- | ------------------- |
| First 100,000 minutes per month | Free                |
| Additional minutes              | \$0.0002 per minute |

<Note>
  For agents deployed with Krisp enabled, active session minutes count as Krisp
  usage, accruing towards your monthly bill.
</Note>

## Local Development

Krisp cannot be used in local development environments, as it requires a proprietary SDK and model which can only be distributed on Pipecat Cloud. When running locally be sure to not import the KrispFilter module and don't apply it to your transport.

## Troubleshooting

If you're experiencing issues with Krisp noise cancellation:

* Verify `enable_krisp` is set to `true` in your deployment configuration
* Confirm you've added the `KrispFilter()` to your transport configuration
* Check your logs for any Krisp-related error messages
* Confirm that you have not added any Krisp-related dependencies to your `requirements.txt` file, as they are not needed and may cause conflicts.

```
```


---

> To find navigation and other pages in this documentation, fetch the llms.txt file at: https://docs.pipecat.ai/llms.txt

---

# Source: https://docs.pipecat.ai/server/utilities/audio/krisp-viva-filter.md

> ## Documentation Index
> Fetch the complete documentation index at: https://docs.pipecat.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# KrispVivaFilter

> Audio noise reduction filter using Krisp VIVA model

## Overview

`KrispVivaFilter` is an audio processor that reduces background noise in real-time audio streams using Krisp VIVA SDK. It inherits from `BaseAudioFilter` and processes audio frames to improve audio quality by removing unwanted noise using Krisp's proprietary noise suppression algorithms.

To use Krisp, you need a Krisp SDK license. Get started at [Krisp.ai](https://krisp.ai/developers/).

## Installation

See the [Krisp guide](/guides/features/krisp-viva) to learn how to install the Krisp VIVA SDK.

## Environment Variables

You need to provide the path to the Krisp model file (.kef extension). This can either be done by setting the `KRISP_VIVA_MODEL_PATH` environment variable or by setting the `model_path` in the constructor.

## Constructor Parameters

<ParamField path="model_path" type="str" default="None">
  Path to the Krisp model file (.kef extension).

  You can set the `model_path` directly. Alternatively, you can set the `KRISP_VIVA_MODEL_PATH` environment variable to the model file path.
</ParamField>

<ParamField path="noise_suppression_level" type="int" default="100">
  Noise suppression level for the filter
</ParamField>

## Supported Sample Rates

The filter supports the following sample rates:

* 8000 Hz
* 16000 Hz
* 24000 Hz
* 32000 Hz
* 44100 Hz
* 48000 Hz

## Input Frames

<ParamField path="FilterEnableFrame" type="Frame">
  Specific control frame to toggle filtering on/off

  ```python  theme={null}
  from pipecat.frames.frames import FilterEnableFrame

  # Disable noise reduction
  await task.queue_frame(FilterEnableFrame(False))

  # Re-enable noise reduction
  await task.queue_frame(FilterEnableFrame(True))
  ```
</ParamField>

## Usage Example

```python  theme={null}
from pipecat.audio.filters.krisp_viva_filter import KrispVivaFilter
from pipecat.transports.daily.transport import DailyParams, DailyTransport

transport = DailyTransport(
    room_url,
    token,
    "Respond bot",
    DailyParams(
        audio_in_enabled=True,
        audio_in_filter=KrispVivaFilter(), # Enable Krisp noise reduction
        audio_out_enabled=True,
    ),
)
```

---

# Source: https://docs.pipecat.ai/guides/features/krisp-viva.md

# Source: https://docs.pipecat.ai/deployment/pipecat-cloud/guides/krisp-viva.md

> ## Documentation Index
> Fetch the complete documentation index at: https://docs.pipecat.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Krisp VIVA Noise Cancellation

> Eliminate background noise and voices for more reliable agent interactions using Krisp's VIVA noise cancellation model

## Overview

Voice agents are vulnerable to background noise, especially other voices that trigger false interruptions. When this happens, your bot stops mid-sentence unnecessarily, creating a poor user experience. Pipecat Cloud's Krisp integration processes incoming audio to remove ambient noise and background voices before they reach your agent, resulting in cleaner audio input with fewer false interruptions and better transcription.

<Info>
  Krisp processes the audio before it reaches your agent's speech recognition
  and VAD systems, helping to ensure that only the intended user's voice
  triggers interruptions.
</Info>

## Enabling Krisp VIVA

Enabling Krisp VIVA with Pipecat Cloud involves the following steps:

### 1. Use the Pipecat Base Image

Krisp VIVA dependencies are pre-installed in the official Pipecat base image. You don't need to add any Krisp VIVA-related packages to your `requirements.txt` file or `pyproject.toml` file.

<Note>
  Krisp VIVA support is available in the `dailyco/pipecat-base` image starting
  with version `0.1.8`.
</Note>

### 2. Add the KrispVivaFilter() to Your Transport

In your bot code, import and add the `KrispVivaFilter()` to your transport:

```python  theme={null}
from pipecat.audio.filters.krisp_viva_filter import KrispVivaFilter
from pipecat.transports.daily.transport import DailyParams, DailyTransport

transport = DailyTransport(
    room_url,
    token,
    "Voice AI Bot",
    DailyParams(
        audio_in_enabled=True,
        audio_in_filter=KrispVivaFilter(),  # Add Krisp VIVA filter here
        audio_out_enabled=True,
    ),
)
```

### 3. Enable Krisp VIVA in Your Deployment

You can enable Krisp VIVA when deploying your agent through:

<Tabs>
  <Tab title="Config File">
    In your `pcc-deploy.toml`:

    ```toml  theme={null}
    agent_name = "voice-starter"
    image = "your-repo/voice-starter:0.1"
    image_credentials = "my-dockerhub-secret"
    secret_set = "voice-starter-secrets"

    [krisp_viva]
    audio_filter = "tel"

    [scaling]
    min_agents = 0
    ```
  </Tab>

  <Tab title="CLI">
    ```bash  theme={null}
    pipecat cloud deploy --krisp-viva-audio-filter tel
    ```
  </Tab>

  <Tab title="REST API">
    When creating or updating an agent via the REST API, include the `krispViva` parameter:

    ```bash  theme={null}
    curl -X POST https://api.pipecat.daily.co/v1/agents \
      -H "Authorization: Bearer YOUR_PRIVATE_API_KEY" \
      -H "Content-Type: application/json" \
      -d '{
        "serviceName": "voice-starter",
        "image": "your-repo/voice-starter:0.1",
        "secretSet": "voice-starter-secrets",
        "krispViva": {
          "audioFilter": "tel"
        },
        "autoScaling": {
          "minAgents": 1,
          "maxAgents": 10
        }
      }'
    ```

    To update an existing agent:

    ```bash  theme={null}
    curl -X POST https://api.pipecat.daily.co/v1/agents/voice-starter \
      -H "Authorization: Bearer YOUR_PRIVATE_API_KEY" \
      -H "Content-Type: application/json" \
      -d '{
        "krispViva": {
          "audioFilter": "tel"
        }
      }'
    ```

    See the [REST API reference](/deployment/pipecat-cloud/rest-reference/endpoint/agent-create) for more details.
  </Tab>

  <Tab title="Dashboard">
    In the Pipecat Cloud dashboard:

    1. Navigate to your agent
    2. Go to Settings > Agent Configuration
    3. Select your desired model from the dropdown
    4. Save your changes which will automatically deploy your agent with the selected model
  </Tab>
</Tabs>

Supported values are:

* `tel`: Supports up to 16kHz audio
* `pro`: Supports up to 32kHz audio

<Tip>
  By default, Pipecat processes input audio at 16kHz, making the `tel` value
  appropriate for most use cases.
</Tip>

## Testing Your Integration

To verify Krisp is working:

1. Deploy your agent with Krisp enabled
2. Connect to your agent in an environment with background noise; try turning on a TV or podcast
3. Observe whether background noises and voices affect your agent's behavior

## Pricing

Krisp processing is billed based on active session minutes:

| Usage Tier                     | Price               |
| ------------------------------ | ------------------- |
| First 10,000 minutes per month | Free                |
| Additional minutes             | \$0.0015 per minute |

<Note>
  For agents deployed with Krisp enabled, active session minutes count as Krisp
  usage, accruing towards your monthly bill.
</Note>

## Local Development

Krisp cannot be used in local development environments, as it requires a proprietary SDK and model which can only be distributed on Pipecat Cloud. When running locally be sure to not import the KrispFilter module and don't apply it to your transport.

## Troubleshooting

If you're experiencing issues with Krisp noise cancellation:

* Verify Krisp VIVA is enabled in your deployment configuration
* Confirm you've added the `KrispVivaFilter()` to your transport configuration
* Check your logs for any Krisp-related error messages
* Confirm that you have **not** added any Krisp-related dependencies to your `requirements.txt` or `pyproject.toml` files, as they are not needed and may cause conflicts.

---

# Source: https://docs.pipecat.ai/guides/features/krisp.md

# Noise cancellation with Krisp

> Learn how to integrate Krisp noise cancellation into your Pipecat application

<Warning>
  The `KrispFilter` is deprecated and will be removed in a future version.
  Please use the `KrispVivaFilter`, which is the latest model from Krisp. See
  the [Krisp VIVA guide](/guides/features/krisp-viva) for more information.
</Warning>

## Overview

This guide will walk you through setting up and using Krisp's noise reduction capabilities in your Pipecat application. Krisp provides professional-grade noise cancellation that can significantly improve audio quality in real-time communications.

To use Krisp's noise cancellation, you'll need to obtain their SDK and models through a Krisp developer account. Our Pipecat Krisp module simplifies the integration process, which we'll cover in this guide.

## Walkthrough

### Get Access to Krisp SDK and Models

1. Create a Krisp developers account at [krisp.ai/developers](https://krisp.ai/developers)
2. Download the [Krisp Desktop SDK (v7)](https://sdk.krisp.ai/sdk/desktop) that matches your platform:
   * Linux: `Desktop SDK v7.0.2: Linux`
   * macOS (ARM): `Desktop SDK v7.0.1: Mac ARM`
   * macOS (Intel): `Desktop SDK v7.0.1: Mac Intel`
   * Windows(ARM): `Desktop SDK v7.0.2: Windows ARM`
   * Windows (x64): `Desktop SDK v7.0.2: Windows`
3. Download the corresponding models. We recommend trying the `Background Voice Cancellation` model. Recommended model for each platform:
   * Linux (ARM): `hs.c6.f.s.de56df.kw`
   * Mac (ARM): `hs.c6.f.s.de56df.kw`
   * Linux (x86\_64): `hs.c6.f.s.de56df.kw`
   * Mac (x86\_64): `hs.c6.f.s.de56df.kw`
   * Windows (x86\_64): `hs.c6.f.s.de56df.kw`

<Frame><img src="https://mintcdn.com/daily/PVk58z7DekUaD0vv/images/krisp-portal.png?fit=max&auto=format&n=PVk58z7DekUaD0vv&q=85&s=5dc5cd957815ebaac7c0a95bd6fc5746" alt="Krisp SDK Portal" data-og-width="2802" width="2802" data-og-height="2348" height="2348" data-path="images/krisp-portal.png" data-optimize="true" data-opv="3" srcset="https://mintcdn.com/daily/PVk58z7DekUaD0vv/images/krisp-portal.png?w=280&fit=max&auto=format&n=PVk58z7DekUaD0vv&q=85&s=a5a71131093407c0f73ba9c577f6e2bd 280w, https://mintcdn.com/daily/PVk58z7DekUaD0vv/images/krisp-portal.png?w=560&fit=max&auto=format&n=PVk58z7DekUaD0vv&q=85&s=bed7afb172e3db79d4886d74e7ada182 560w, https://mintcdn.com/daily/PVk58z7DekUaD0vv/images/krisp-portal.png?w=840&fit=max&auto=format&n=PVk58z7DekUaD0vv&q=85&s=165a851cb70505988eadaba352b43fcc 840w, https://mintcdn.com/daily/PVk58z7DekUaD0vv/images/krisp-portal.png?w=1100&fit=max&auto=format&n=PVk58z7DekUaD0vv&q=85&s=b6290177c64d4a7af7e4947b83cc135a 1100w, https://mintcdn.com/daily/PVk58z7DekUaD0vv/images/krisp-portal.png?w=1650&fit=max&auto=format&n=PVk58z7DekUaD0vv&q=85&s=d9a18f3a2dcb9684e9154a7712cb622f 1650w, https://mintcdn.com/daily/PVk58z7DekUaD0vv/images/krisp-portal.png?w=2500&fit=max&auto=format&n=PVk58z7DekUaD0vv&q=85&s=0253d0235e27a1590b37d0a1eb25b808 2500w" /></Frame>

### Install build dependencies

The `pipecat-ai-krisp` module is a python wrapper around Krisp's C++ SDK. To build the module, you'll need to install the following dependencies:

<Tabs>
  <Tab title="macOS">
    ```bash  theme={null}
    # Using Homebrew
    brew install cmake pybind11
    ```
  </Tab>

  <Tab title="Ubuntu/Debian">
    ```bash  theme={null}
    # Using apt
    sudo apt-get update
    sudo apt-get install cmake python3-dev pybind11-dev g++
    ```
  </Tab>

  <Tab title="Windows">
    ```powershell  theme={null}
    # Using Chocolatey
    choco install cmake
    # pybind11 will be installed via pip during the build process

    # OR using Visual Studio
    # 1. Install Visual Studio with C++ development tools
    # 2. Install CMake from https://cmake.org/download/
    # pybind11 will be installed via pip during the build process
    ```
  </Tab>
</Tabs>

<Note>
  For Windows users: Make sure you have Visual Studio installed with C++
  development tools, or alternatively, have the Visual C++ Build Tools
  installed.
</Note>

### Install the pipecat-ai-krisp module

In your Pipecat repo, activate your virtual environment:

<Tabs>
  <Tab title="macOS/Linux">
    ```bash  theme={null}
    python3 -m venv venv
    source venv/bin/activate
    ```
  </Tab>

  <Tab title="Windows">
    ```powershell  theme={null}
    # Using PowerShell
    python -m venv venv
    .\venv\Scripts\Activate.ps1

    # OR using Command Prompt
    python -m venv venv
    .\venv\Scripts\activate.bat

    # OR using Git Bash
    python -m venv venv
    source venv/Scripts/activate
    ```
  </Tab>
</Tabs>

Export the path to your Krisp SDK and model files:

<Tabs>
  <Tab title="macOS/Linux">
    ```bash  theme={null}
    # Add to your .env file or shell configuration
    export KRISP_SDK_PATH=/PATH/TO/KRISP/SDK
    export KRISP_MODEL_PATH=/PATH/TO/KRISP/MODEL.kef
    ```
  </Tab>

  <Tab title="Windows">
    ```powershell  theme={null}
    # Using PowerShell
    $env:KRISP_SDK_PATH = "C:\PATH\TO\KRISP\SDK"
    $env:KRISP_MODEL_PATH = "C:\PATH\TO\KRISP\MODEL.kef"

    # OR using Command Prompt
    set KRISP_SDK_PATH=C:\PATH\TO\KRISP\SDK
    set KRISP_MODEL_PATH=C:\PATH\TO\KRISP\MODEL.kef
    ```
  </Tab>
</Tabs>

<Tip>
  When selecting a `KRISP_MODEL_PATH`, ensure that you're selecting the actual
  model file, not just the directory. The path should look something like this:

  <Tabs>
    <Tab title="ARM Linux">
      ```bash  theme={null}
      export KRISP_MODEL_PATH=./krisp/hs.c6.f.s.de56df.kw
      ```
    </Tab>

    <Tab title="Mac ARM">
      ```bash export theme={null}
      KRISP_MODEL_PATH=./krisp/outbound-bvc-models-fp16/hs.c6.f.s.de56df.bucharest.kef
      ```
    </Tab>

    <Tab title="Windows">
      ```powershell  theme={null}
      $env:KRISP_MODEL_PATH="C:\krisp\outbound-bvc-models-fp32\hs.c6.f.s.de56df.bucharest.kef"
      ```
    </Tab>
  </Tabs>
</Tip>

Next, install the `pipecat-ai[krisp]` module, which will automatically build the `pipecat-ai-krisp` python wrapper module:

```bash  theme={null}
pip install "pipecat-ai[krisp]"
```

### Test the integration

You can now test the Krisp integration. The easiest way to do this is to run the foundational example: [07p-interruptible-krisp.py](https://github.com/pipecat-ai/pipecat/blob/main/examples/foundational/07p-interruptible-krisp.py).

<Tip>
  You can run a foundational example by running the following command:

  ```bash  theme={null}
  python examples/foundational/07p-interruptible-krisp.py -u YOUR_DAILY_ROOM_URL
  ```
</Tip>

#### Important for macOS users

If you're running on macOS you may receive a security warning about running the script. This is expected. You can allow access by going to `System Settings > Privacy & Security` then click `Allow Anyway` to permit the example to run.

<Frame><img src="https://mintcdn.com/daily/PVk58z7DekUaD0vv/images/allow-krisp.png?fit=max&auto=format&n=PVk58z7DekUaD0vv&q=85&s=a93947edd11c47b2e1be8879a35e8284" alt="Security & Privacy" data-og-width="1654" width="1654" data-og-height="1474" height="1474" data-path="images/allow-krisp.png" data-optimize="true" data-opv="3" srcset="https://mintcdn.com/daily/PVk58z7DekUaD0vv/images/allow-krisp.png?w=280&fit=max&auto=format&n=PVk58z7DekUaD0vv&q=85&s=c2d115817943bc4689fa31a3e00c18a9 280w, https://mintcdn.com/daily/PVk58z7DekUaD0vv/images/allow-krisp.png?w=560&fit=max&auto=format&n=PVk58z7DekUaD0vv&q=85&s=2c1c9cba0cc342070c3d19d74e038982 560w, https://mintcdn.com/daily/PVk58z7DekUaD0vv/images/allow-krisp.png?w=840&fit=max&auto=format&n=PVk58z7DekUaD0vv&q=85&s=96bfd405f776480b4f1ee8f12916f249 840w, https://mintcdn.com/daily/PVk58z7DekUaD0vv/images/allow-krisp.png?w=1100&fit=max&auto=format&n=PVk58z7DekUaD0vv&q=85&s=ef7019662e66a027a44d74bf812f591a 1100w, https://mintcdn.com/daily/PVk58z7DekUaD0vv/images/allow-krisp.png?w=1650&fit=max&auto=format&n=PVk58z7DekUaD0vv&q=85&s=080f04365097880083f4d8bcda2dfdbb 1650w, https://mintcdn.com/daily/PVk58z7DekUaD0vv/images/allow-krisp.png?w=2500&fit=max&auto=format&n=PVk58z7DekUaD0vv&q=85&s=353d57481f15078b24d8f44ae46683f7 2500w" /></Frame>

After allowing and re-running, you may get a pop-up asking for permission. Select `Open Anyway` to run the script.

<Frame><img src="https://mintcdn.com/daily/PVk58z7DekUaD0vv/images/open-anyway-krisp.png?fit=max&auto=format&n=PVk58z7DekUaD0vv&q=85&s=6aeec83402972fb028e23a053d12d200" alt="Open Anyway" data-og-width="744" width="744" data-og-height="912" height="912" data-path="images/open-anyway-krisp.png" data-optimize="true" data-opv="3" srcset="https://mintcdn.com/daily/PVk58z7DekUaD0vv/images/open-anyway-krisp.png?w=280&fit=max&auto=format&n=PVk58z7DekUaD0vv&q=85&s=e2132b4a915c4e9b05bdd8f76985bd9b 280w, https://mintcdn.com/daily/PVk58z7DekUaD0vv/images/open-anyway-krisp.png?w=560&fit=max&auto=format&n=PVk58z7DekUaD0vv&q=85&s=4cd16fb3141cc97629aa650def4ec8eb 560w, https://mintcdn.com/daily/PVk58z7DekUaD0vv/images/open-anyway-krisp.png?w=840&fit=max&auto=format&n=PVk58z7DekUaD0vv&q=85&s=23d79f2041851fe136fe7296460bd46b 840w, https://mintcdn.com/daily/PVk58z7DekUaD0vv/images/open-anyway-krisp.png?w=1100&fit=max&auto=format&n=PVk58z7DekUaD0vv&q=85&s=3fdceddc530a82a15eb7fe52fbd9c05f 1100w, https://mintcdn.com/daily/PVk58z7DekUaD0vv/images/open-anyway-krisp.png?w=1650&fit=max&auto=format&n=PVk58z7DekUaD0vv&q=85&s=2cad0cfb25587da019f7803976037f86 1650w, https://mintcdn.com/daily/PVk58z7DekUaD0vv/images/open-anyway-krisp.png?w=2500&fit=max&auto=format&n=PVk58z7DekUaD0vv&q=85&s=93ee05306a137495afbc09e5eff71a3c 2500w" /></Frame>

## Usage Example

Here's a basic example of how to add Krisp noise reduction to your Pipecat pipeline:

```python  theme={null}
from pipecat.audio.filters.krisp_filter import KrispFilter
from pipecat.transports.services.daily import DailyParams, DailyTransport

# Add to transport configuration
transport = DailyTransport(
    room_url,
    token,
    "Audio Bot",
    DailyParams(
        audio_in_filter=KrispFilter(),  # Enable Krisp noise reduction
        audio_in_enabled=True,
        audio_out_enabled=True,
        vad_analyzer=SileroVADAnalyzer(),
    ),
)
```

## Troubleshooting

Common issues and solutions:

1. **Missing Dependencies**

```bash  theme={null}
Error: Missing module: pipecat_ai_krisp
```

Solution: Ensure you've installed with the krisp extra: `pip install "pipecat-ai[krisp]"`

2. **Model Path Not Found**

```bash  theme={null}
Error: Model path for KrispAudioProcessor must be provided
```

Solution: Set the `KRISP_MODEL_PATH` environment variable or provide it in the constructor

3. **SDK Path Issues**

```bash  theme={null}
Error: Cannot find Krisp SDK
```

Solution: Verify `KRISP_SDK_PATH` points to a valid Krisp SDK installation

## Additional Resources

* [KrispFilter Reference Documentation](/server/utilities/audio/krisp-filter)
* [Example Application](https://github.com/pipecat-ai/pipecat/blob/main/examples/foundational/07p-interruptible-krisp.py)


---

> To find navigation and other pages in this documentation, fetch the llms.txt file at: https://docs.pipecat.ai/llms.txt

---

# Source: https://docs.pipecat.ai/server/services/transport/livekit.md

> ## Documentation Index
> Fetch the complete documentation index at: https://docs.pipecat.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# LiveKitTransport

> WebRTC transport implementation using LiveKit for real-time audio communication

## Overview

`LiveKitTransport` provides real-time audio communication capabilities using LiveKit's open-source WebRTC platform. It supports bidirectional audio streaming, data messaging, participant management, and room event handling for conversational AI applications with the flexibility of self-hosted or cloud infrastructure.

<CardGroup cols={2}>
  <Card title="LiveKitTransport API Reference" icon="code" href="https://reference-server.pipecat.ai/en/latest/api/pipecat.transports.livekit.transport.html">
    Pipecat's API methods for LiveKit integration
  </Card>

  <Card title="Example Implementation" icon="play" href="https://github.com/pipecat-ai/pipecat/blob/main/examples/foundational/04b-transports-livekit.py">
    Complete LiveKit voice agent example
  </Card>

  <Card title="LiveKit Documentation" icon="book" href="https://docs.livekit.io/reference/">
    Official LiveKit API documentation and guides
  </Card>

  <Card title="LiveKit Cloud" icon="external-link" href="https://cloud.livekit.io/login">
    Sign up for hosted LiveKit service
  </Card>
</CardGroup>

## Installation

To use LiveKitTransport, install the required dependencies:

```bash  theme={null}
pip install "pipecat-ai[livekit]"
```

## Prerequisites

### LiveKit Setup

Before using LiveKitTransport, you need:

1. **LiveKit Server**: Set up self-hosted or use [LiveKit Cloud](https://cloud.livekit.io/)
2. **API Credentials**: Generate API key and secret from your LiveKit project
3. **Room Management**: Create rooms using LiveKit API or SDK
4. **Access Tokens**: Generate JWT tokens for room authentication

### Required Environment Variables

* `LIVEKIT_API_KEY`: Your LiveKit API key for authentication
* `LIVEKIT_API_SECRET`: Your LiveKit API secret for token generation
* `LIVEKIT_URL`: Your LiveKit server URL (wss\://your-domain.livekit.cloud)

### Key Features

* **Open Source**: Self-hosted or cloud-hosted WebRTC infrastructure
* **Multi-participant Support**: Handle multiple participants in rooms
* **Data Channels**: Real-time messaging alongside audio streams
* **Room Management**: Dynamic participant and room lifecycle management
* **Flexible Deployment**: Self-hosted or managed cloud options

## Usage

LiveKitTransport connects your Pipecat bot to LiveKit rooms where it can communicate with participants through audio and data channels. The transport handles room joining, participant events, and media streaming automatically.

See the [complete example](https://github.com/pipecat-ai/pipecat/blob/main/examples/foundational/04b-transports-livekit.py) for a full implementation including:

* LiveKit room creation and token generation
* Transport configuration with participant management
* Pipeline integration with audio processing
* Event handling for room lifecycle management

---

# Source: https://docs.pipecat.ai/server/utilities/observers/llm-observer.md

> ## Documentation Index
> Fetch the complete documentation index at: https://docs.pipecat.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# LLM Log Observer

> Logging LLM activity in Pipecat

The `LLMLogObserver` provides detailed logging of Large Language Model (LLM) activity within your Pipecat pipeline. It tracks the entire lifecycle of LLM interactions, from initial prompts to final responses.

## Frame Types Monitored

The observer tracks the following frame types (only from/to LLM service):

* **LLMFullResponseStartFrame**: When the LLM begins generating a response
* **LLMFullResponseEndFrame**: When the LLM completes its response
* **LLMTextFrame**: Individual text chunks generated by the LLM
* **FunctionCallInProgressFrame**: Function/tool calls made by the LLM
* **LLMMessagesFrame**: Input messages sent to the LLM (deprecated in favor of context frames)
* **LLMContextFrame**: Updated context sent to the LLM
* **FunctionCallResultFrame**: Results returned from function calls

## Usage

```python  theme={null}
from pipecat.observers.loggers.llm_log_observer import LLMLogObserver

task = PipelineTask(
    pipeline,
    params=PipelineParams(
        observers=[LLMLogObserver()],
    ),
)
```

## Log Output Format

The observer uses emojis and consistent formatting for easy log reading:

* 🧠 \[Source] → LLM START/END RESPONSE
* 🧠 \[Source] → LLM GENERATING: \[text]
* 🧠 \[Source] → LLM FUNCTION CALL: \[details]
* 🧠 → \[Destination] LLM MESSAGES FRAME: \[messages]
* 🧠 → \[Destination] LLM CONTEXT FRAME: \[context]

All log entries include timestamps for precise timing analysis.

---

# Source: https://docs.pipecat.ai/server/utilities/service-switchers/llm-switcher.md

> ## Documentation Index
> Fetch the complete documentation index at: https://docs.pipecat.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# LLMSwitcher

> Dynamically switch between different LLM services with support for ad-hoc inference and unified function registration

## Overview

`LLMSwitcher` is a specialized version of `ServiceSwitcher` designed specifically for managing multiple LLM services. Beyond basic service switching, it provides convenient methods for running ad-hoc inferences and registering function handlers across all LLMs simultaneously.

This is particularly useful when you need to switch between different LLM providers based on task complexity, cost optimization, or specific model capabilities while maintaining a consistent function calling interface.

## Constructor

```python  theme={null}
from pipecat.pipeline.llm_switcher import LLMSwitcher
from pipecat.pipeline.service_switcher import ServiceSwitcherStrategyManual

switcher = LLMSwitcher(
    llms=[openai_service, gemini_service],
    strategy_type=ServiceSwitcherStrategyManual
)
```

<ParamField path="llms" type="List[LLMService]" required>
  List of LLM service instances to switch between.
</ParamField>

<ParamField path="strategy_type" type="Type[ServiceSwitcherStrategy]" required>
  The strategy class to use for switching logic. Pass the class itself, not an
  instance.
</ParamField>

## Properties

<ParamField path="llms" type="List[LLMService]">
  The list of LLM services managed by this switcher.
</ParamField>

<ParamField path="active_llm" type="Optional[LLMService]">
  The currently active LLM service, or None if no LLMs are configured.
</ParamField>

## Methods

### run\_inference()

Run a one-shot inference with the currently active LLM, outside of the normal pipeline flow.

```python  theme={null}
result = await llm_switcher.run_inference(context=llm_context)
```

<ParamField path="context" type="LLMContext" required>
  The LLM context containing conversation history and messages.
</ParamField>

**Returns**: `Optional[str]` - The LLM's response as a string, or None if no LLM is active.

### register\_function()

Register a function handler with all LLMs in the switcher, regardless of which is currently active.

```python  theme={null}
llm_switcher.register_function(
    function_name="get_weather",
    handler=handle_weather,
    cancel_on_interruption=True
)
```

<ParamField path="function_name" type="Optional[str]" required>
  The name of the function to handle. Use None for a catch-all handler that
  processes all function calls.
</ParamField>

<ParamField path="handler" type="Callable" required>
  The async function handler. Should accept a single `FunctionCallParams`
  parameter.
</ParamField>

<ParamField path="start_callback" type="Optional[Callable]" deprecated>
  Legacy callback function (deprecated). Put initialization code at the top of
  your handler instead.
</ParamField>

<ParamField path="cancel_on_interruption" type="bool" default="True">
  Whether to cancel this function call when a user interruption occurs.
</ParamField>

## Usage Examples

### Basic LLM Switching

```python  theme={null}
from pipecat.pipeline.llm_switcher import LLMSwitcher
from pipecat.pipeline.service_switcher import ServiceSwitcherStrategyManual
from pipecat.services.openai.llm import OpenAILLMService
from pipecat.services.google.llm import GoogleLLMService
from pipecat.frames.frames import ManuallySwitchServiceFrame

# Create LLM services
openai = OpenAILLMService(api_key=os.getenv("OPENAI_API_KEY"))
gemini = GoogleLLMService(api_key=os.getenv("GOOGLE_API_KEY"))

# Create switcher
llm_switcher = LLMSwitcher(
    llms=[openai, gemini],
    strategy_type=ServiceSwitcherStrategyManual
)

# Use in pipeline
pipeline = Pipeline([
    transport.input(),
    stt,
    context_aggregator.user(),
    llm_switcher,
    tts,
    transport.output(),
    context_aggregator.assistant()
])

# Switch to cheaper model for simple tasks
await task.queue_frame(ManuallySwitchServiceFrame(service=gpt35))
```

---

# Source: https://docs.pipecat.ai/server/utilities/frame/llm-text-processor.md

> ## Documentation Index
> Fetch the complete documentation index at: https://docs.pipecat.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# LLMTextProcessor

> A processor for aggregating LLMTextFrames into logical units before passing them to downstream services

## Overview

`LLMTextProcessor` is a processor designed to aggregate `LLMTextFrame`s into coherent text units before passing them to downstream services, such as TTS. By utilizing text aggregators, it ensures that text is properly segmented and structured, enhancing the quality of subsequent processing. This processor expects `LLMTextFrame`s as input and outputs `AggregatedTextFrame`s containing the aggregated text.

<Note>
  If an `LLMTextProcessor` is in use, the `text_aggregator` parameter of TTS services will be ignored, as text aggregation is handled upstream.
</Note>

The benefit of pre-aggregating LLM text frames is that it allows for more controlled and meaningful text synthesis. Downstream services can operate on complete sentences or logical text blocks. For TTS services, this means being able to customize how certain types of text are spoken (e.g., spelling out phone numbers, stripping out url protocols, or inserting other tts-specific annotations) or even skipping over certain text segments entirely (e.g., code snippets or markup).  For other services, such as RTVI, it allows for sending these logical text units as separate `bot-output` messages, supporting custom client-side handling and rendering (e.g. collapsible code blocks, clickable links, etc.).

<CardGroup cols={2}>
  <Card title="LLMTextProcessor API Reference" icon="code" href="https://reference-server.pipecat.ai/en/latest/api/pipecat.processors.aggregators.llm_text_processor.html">
    Pipecat's API methods for LLMTextProcessor
  </Card>

  <Card title="Example Implementation" icon="play" href="https://github.com/pipecat-ai/pipecat-examples/blob/main/code-helper">
    Complete example with LLMTextProcessor and custom TTS and RTVI handling
  </Card>
</CardGroup>

## Constructor Parameters

<ParamField path="text_aggregator" type="BaseTextAggregator" default="None">
  An instance of a text aggregator (e.g., `PatternPairAggregator` or a custom aggregator type) used to aggregate incoming text from `LLMTextFrame`s.
  If `None`, a `SimpleTextAggregator` will be used by default, aggregating text based on sentence boundaries.
</ParamField>

## Usage

The `LLMTextProcessor` should be integrated into your pipeline after the LLM service and before any services that consume text, such as TTS. It processes incoming `LLMTextFrame`s, aggregates their text content, and outputs `AggregatedTextFrame`s.

<Tip>
  For more usage examples, check out the docs for the [PatternPairAggregator](/server/utilities/text/pattern-pair-aggregator#usage-examples).
</Tip>

```python  theme={null}
from pipecat.processors.aggregators.llm_text_processor import LLMTextProcessor

...

llm_text_aggregator = PatternPairAggregator()
llm_text_aggregator.add_pattern(
    type="code",
    start_pattern="<code>",
    end_pattern="</code>",
    action=MatchAction.AGGREGATE,
)
llm_text_processor = LLMTextProcessor(text_aggregator=llm_text_aggregator)

...

# Pipeline - The following pipeline is typical for a STT->LLM->TTS bot + RTVI
#            with the addition of the LLMTextProcessor to handle special text segments.
pipeline = Pipeline(
    [
        transport.input(),
        rtvi,
        stt,
        transcript_processor.user(),
        context_aggregator.user(),
        llm,
        llm_text_processor,
        tts,
        transport.output(),
        transcript_processor.assistant(),
        context_aggregator.assistant(),
    ]
)
```

---

# Source: https://docs.pipecat.ai/guides/learn/llm.md

> ## Documentation Index
> Fetch the complete documentation index at: https://docs.pipecat.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# LLM Inference

> Learn how to configure language models to generate intelligent responses in your voice AI pipeline

**LLM services** are responsible for chat completions and tool calling based on the provided context (conversation history). The LLM responds by streaming tokens via `LLMTextFrame`s, which are used by subsequent processors to create audio output for the bot.

## Pipeline Placement

The LLM instance should be placed after the user context aggregator and before any downstream services that depend on the LLM's output stream:

```python  theme={null}
pipeline = Pipeline([
    transport.input(),
    stt,                           # Creates TranscriptionFrames
    context_aggregator.user(),     # Processes user context → creates LLMContextFrame
    llm,                           # Processes context → streams LLMTextFrames
    tts,                           # Processes LLMTextFrames → creates TTSAudioRawFrames
    transport.output(),
    context_aggregator.assistant(),
])
```

**Frame flow:**

* **Input**: Receives `LLMContextFrame` containing conversation history
* **Processing**:
  * Analyzes context and generates streaming response
  * Handles function calls if tools are available
  * Tracks token usage for metrics
* **Output**:
  * Denotes the start of the streaming response by pushing an `LLMFullResponseStartFrame`
  * Streams `LLMTextFrame`s containing response tokens to downstream processors (enables real-time TTS processing)
  * Ends with an `LLMFullResponseEndFrame` to mark the completion of the response
* **Function calls:**
  * `FunctionCallsStartedFrame`: Indicates function execution beginning
  * `FunctionCallInProgressFrame`: Indicates a function is currently executing
  * `FunctionCallResultFrame`: Contains results from executed functions

## Supported LLM Services

Pipecat supports a wide range of LLM providers to fit different needs, performance requirements, and budgets:

### Text-Based LLMs

Most LLM services are built on the OpenAI chat completion specification for compatibility:

<CardGroup cols={2}>
  <Card title="OpenAI" icon="openai">
    GPT models with the original chat completion API
  </Card>

  <Card title="Anthropic" icon="anthropic">
    Claude models with advanced reasoning capabilities
  </Card>

  <Card title="Google Gemini" icon="google">
    Multimodal capabilities with competitive performance
  </Card>

  <Card title="AWS Bedrock" icon="aws">
    Enterprise-grade hosting for various foundation models
  </Card>
</CardGroup>

**Compatible APIs**: Any OpenAI-spec compatible service can be used via the `base_url` parameter.

### Speech-to-Speech Models

For lower latency, some providers offer direct speech-to-speech models:

* **OpenAI Realtime**: Direct speech input/output with GPT models
* **Gemini Live**: Real-time speech conversations with Gemini
* **AWS Nova Sonic**: Speech-optimized models on Bedrock

For a complete list of supported LLM services, see the Supported Services page:

<Card title="Supported LLM Services" icon="list" href="/server/services/supported-services#large-language-models">
  View the complete list of supported language model providers
</Card>

## LLM Service Architecture

### BaseOpenAILLMService

Many LLM services use the OpenAI chat completion specification. Pipecat provides a `BaseOpenAILLMService` that most providers extend, enabling easy switching between compatible services:

```python  theme={null}
from pipecat.services.openai.llm import OpenAILLMService

# Native OpenAI
llm = OpenAILLMService(api_key=os.getenv("OPENAI_API_KEY"))

# OpenAI-compatible service via base_url
llm = OpenAILLMService(
    api_key=os.getenv("OTHER_API_KEY"),
    base_url="https://api.other-provider.com/v1"  # Custom endpoint
)
```

This architecture allows you to quickly plug in different LLM services without changing your pipeline code.

## LLM Configuration

### Service-Specific Configuration

Each LLM service has its own configuration options. For example, configuring OpenAI with various parameters:

```python  theme={null}
from pipecat.services.openai.base_llm import BaseOpenAILLMService
from pipecat.services.openai.llm import OpenAILLMService

llm = OpenAILLMService(
    api_key=os.getenv("OPENAI_API_KEY"),
    model="gpt-4",
    params=BaseOpenAILLMService.InputParams(
        temperature=0.7,            # Response creativity (0.0-2.0)
        max_completion_tokens=150,  # Maximum response length
        frequency_penalty=0.5,      # Reduce repetition (0.0-2.0)
        presence_penalty=0.5,       # Encourage topic diversity (0.0-2.0)
    ),
)
```

For detailed configuration options specific to each provider:

<Card title="Individual LLM Services" icon="settings" href="/server/services/supported-services#large-language-models">
  Explore configuration options for each supported LLM provider
</Card>

### Base Class Configuration

All LLM services inherit from the `LLMService` base class with shared configuration options:

```python  theme={null}
llm = YourLLMService(
    # Service-specific options...
    run_in_parallel=True,  # Whether function calls run in parallel (default: True)
)
```

**Key options:**

* **`run_in_parallel`**: Controls whether function calls execute simultaneously or sequentially
  * `True` (default): Faster execution when multiple functions are called
  * `False`: Sequential execution for dependent function calls

## Event Handlers

LLM services provide event handlers for monitoring completion lifecycle:

```python  theme={null}
@llm.event_handler("on_completion_timeout")
async def on_completion_timeout(service):
    logger.warning("LLM completion timed out")
    # Handle timeout (retry, fallback, etc.)

@llm.event_handler("on_function_calls_started")
async def on_function_calls_started(service, function_calls):
    logger.info(f"Starting {len(function_calls)} function calls")
    # Optionally notify user that bot is "thinking"
    await tts.queue_frame(TTSSpeakFrame("Let me check on that."))
```

**Available events:**

* **`on_completion_timeout`**: Triggered when LLM requests timeout
* **`on_function_calls_started`**: Triggered when function calls are initiated

These handlers enable you to provide user feedback and implement error recovery strategies.

## Function Calling

LLMs can call external functions to access real-time data and perform actions beyond their training data. This enables capabilities like checking weather, querying databases, or controlling external APIs.

Function calls and their results are automatically stored in the conversation context by the context aggregator.

<Card title="Function Calling" icon="code" href="/guides/learn/function-calling">
  Learn how to enable LLMs to interact with external services and APIs
</Card>

## Key Takeaways

* **Pipeline placement matters** - LLM goes after user context, before TTS
* **Token streaming enables real-time responses** - no waiting for complete generation
* **OpenAI compatibility** enables easy provider switching
* **Function calling extends capabilities** beyond training data
* **Configuration affects behavior** - tune temperature, penalties, and limits
* **Services are modular** - swap providers without changing pipeline code

## What's Next

Now that you understand LLM configuration, let's explore how function calling enables your bot to interact with external services and real-time data.

<Card title="Function Calling" icon="arrow-right" href="/guides/learn/function-calling">
  Learn how to enable LLMs to interact with external services and APIs
</Card>

---

# Source: https://docs.pipecat.ai/server/services/tts/lmnt.md

> ## Documentation Index
> Fetch the complete documentation index at: https://docs.pipecat.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# LMNT

> Text-to-speech service implementation using LMNT's streaming API

## Overview

`LMNTTTSService` provides real-time text-to-speech synthesis through LMNT's WebSocket-based streaming API optimized for conversational AI. The service offers ultra-low latency with high-quality voice models and supports multiple languages with automatic interruption handling.

<CardGroup cols={2}>
  <Card title="LMNT TTS API Reference" icon="code" href="https://reference-server.pipecat.ai/en/latest/api/pipecat.services.lmnt.tts.html">
    Pipecat's API methods for LMNT TTS integration
  </Card>

  <Card title="Example Implementation" icon="play" href="https://github.com/pipecat-ai/pipecat/blob/main/examples/foundational/07k-interruptible-lmnt.py">
    Complete example with voice synthesis
  </Card>

  <Card title="LMNT Documentation" icon="book" href="https://docs.lmnt.com/api-reference/speech/streaming">
    Official LMNT streaming speech API documentation
  </Card>

  <Card title="Voice Library" icon="microphone" href="https://app.lmnt.com/">
    Browse and create custom voices
  </Card>
</CardGroup>

## Installation

To use LMNT services, install the required dependencies:

```bash  theme={null}
pip install "pipecat-ai[lmnt]"
```

## Prerequisites

### LMNT Account Setup

Before using LMNT TTS services, you need:

1. **LMNT Account**: Sign up at [LMNT Console](https://app.lmnt.com/)
2. **API Key**: Generate an API key from your account dashboard
3. **Voice Selection**: Choose from available voice models or create custom voices

### Required Environment Variables

* `LMNT_API_KEY`: Your LMNT API key for authentication

---

# Source: https://docs.pipecat.ai/deployment/pipecat-cloud/fundamentals/logging.md

> ## Documentation Index
> Fetch the complete documentation index at: https://docs.pipecat.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Logging and Observability

> Obtaining logs and metrics from your agents and sessions

<Warning>
  This section of the documentation is currently work in progress. Please check
  back soon for updates.
</Warning>

## Configuring log level

You can control the Pipecat logging level for your deployed agents using the `PIPECAT_LOG_LEVEL` environment variable. This can be set as a [secret](/deployment/pipecat-cloud/fundamentals/secrets) or directly in your deployment configuration.

Available log levels:

| Level     | Description                                                          |
| --------- | -------------------------------------------------------------------- |
| `TRACE`   | The most detailed information for debugging (includes Frame logging) |
| `DEBUG`   | Verbose output for debugging (default)                               |
| `INFO`    | General operational information                                      |
| `WARNING` | Warning messages for potential issues                                |
| `ERROR`   | Error messages only                                                  |

## Agent logs

Agent logs are available via both the CLI and Dashboard. You can view logs for a specific agent by running the following command:

```bash  theme={null}
pipecat cloud agent logs my-agent
```

This command accept various filters to help you narrow down the logs you are looking for. For example, you can filter logs by severity level:

```bash  theme={null}
pipecat cloud agent logs my-agent -l ERROR
```

### Session logging

We recommend using the `loguru` library for logging within your agent. This will ensure any logging within your agent associated to the session it is running in.

```python  theme={null}
from loguru import logger

async def bot():
	logger.info("Hello, world!") # will be associated with the session id
```

If you are handling logging manually, you can obtain the active session ID from the `PipecatRunnerArguments` object (or subclass alternative) passed to your `bot()` method:

```python  theme={null}
from pipecat.runner.types import PipecatRunnerArguments

async def bot(args: PipecatRunnerArguments):
	session_id = args.session_id
```

<Note>
  See [the Session Arguments reference](../sdk-reference/session-arguments) for
  more additional SessionArgument types.
</Note>

## CPU and memory metrics

Pipecat Cloud tracks CPU and memory usage for each session, which can be helpful for troubleshooting performance issues. You can view these metrics in two ways:

### Dashboard

Navigate to your agent in the Pipecat Cloud dashboard, then go to **Sessions** and click on a specific **Session ID** to view CPU and memory usage graphs.

### CLI

Use the `sessions` command with a specific session ID to see CPU and memory usage with sparkline visualizations and percentile summaries:

```bash  theme={null}
pipecat cloud agent sessions my-agent --id <session-id>
```

See the [CLI reference](/cli/cloud/agent#sessions) for more details.

---

# Source: https://docs.pipecat.ai/deployment/pipecat-cloud/guides/managed-api-keys.md

> ## Documentation Index
> Fetch the complete documentation index at: https://docs.pipecat.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Managed API Keys

> Using managed API keys for your Pipecat Cloud agents

## Overview

Pipecat Cloud provides API keys for some Pipecat services that you can use directly in your applications. These managed keys give you access to competitive pricing, higher rate limits, pre-signed BAAs and DPAs for compliance, and integrated billing with your Pipecat Cloud account.

<Note>
  Managed keys are currently in beta. We are planning to add more services based
  on demand. If you're interested in us supporting a specific service, let us
  know by emailing [pipecat-ai@daily.co](mailto:pipecat-ai@daily.co).
</Note>

## Supported Services

* **Deepgram Flux (STT)**

## Pricing

Usage of managed keys is billed through your Pipecat Cloud account. For detailed pricing information, visit our [pricing page](https://www.daily.co/pricing/pipecat-cloud/).

## How to Use Managed Keys

### Enable in Deployment Configuration

When deploying your agent, you need to enable managed keys in your configuration.

**Using pcc-deploy.toml:**

```toml  theme={null}
enable_managed_keys = true
```

**Using the CLI:**

```bash  theme={null}
pipecat cloud deploy --enable-managed-keys true
```

<Info>
  This setting is configured per agent deployment. Any changes require
  redeploying your agent to take effect.
</Info>

### Update Your Code

When initializing a supported service in your Pipecat application, use the special string `"PIPECATCLOUD"` as the API key. This tells Pipecat Cloud to use the managed key when your agent is deployed.

To ensure your code works both locally (with your own API key) and when deployed (with managed keys), use this pattern:

```python  theme={null}
import os
from pipecat.services.deepgram.flux.stt import DeepgramFluxSTTService

tts = DeepgramFluxSTTService(api_key=os.getenv("DEEPGRAM_API_KEY", "PIPECATCLOUD"))
```

This approach:

* Uses your local `DEEPGRAM_API_KEY` environment variable during development
* Automatically switches to the managed key when deployed to Pipecat Cloud

<Note>
  The `PIPECATCLOUD` key is only available for Deepgram Flux (STT) when there is
  no value for `DEEPGRAM_API_KEY` in your corresponding secrets set.
</Note>

## Getting Help

If you have any questions about using managed keys, reach out to us on [Discord](https://discord.gg/pipecat). Our community and team are happy to help!

---

# Source: https://docs.pipecat.ai/server/utilities/text/markdown-text-filter.md

> ## Documentation Index
> Fetch the complete documentation index at: https://docs.pipecat.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# MarkdownTextFilter

> Converts Markdown-formatted text to TTS-friendly plain text while preserving structure

## Overview

`MarkdownTextFilter` transforms Markdown-formatted text into plain text that's suitable for text-to-speech (TTS) systems. It intelligently removes formatting elements while preserving the content structure, including proper spacing and list formatting.

This filter is especially valuable for LLM-generated content, which often includes Markdown formatting that would sound unnatural if read aloud by a TTS system.

## Constructor

```python  theme={null}
filter = MarkdownTextFilter(params=InputParams())
```

<ParamField path="params" type="InputParams">
  Configuration parameters for the filter
</ParamField>

### Input Parameters

Configure the filter behavior with these options:

<ParamField path="enable_text_filter" type="bool" default="True">
  Whether the filter is active (when False, text passes through unchanged)
</ParamField>

<ParamField path="filter_code" type="bool" default="False">
  Whether to remove code blocks from the output
</ParamField>

<ParamField path="filter_tables" type="bool" default="False">
  Whether to remove Markdown tables from the output
</ParamField>

## Features

The filter handles these Markdown elements:

* **Basic Formatting**: Removes `*italic*`, `**bold**`, and other formatting markers
* **Code**: Removes inline code ticks and optionally removes code blocks
* **Lists**: Preserves numbered lists while removing Markdown formatting
* **Tables**: Optionally removes Markdown tables
* **Whitespace**: Carefully preserves meaningful whitespace for natural speech
* **HTML**: Removes HTML tags and converts entities to their plain text equivalents

## Usage Examples

### Basic Usage with TTS Service

```python  theme={null}
from pipecat.utils.text.markdown_text_filter import MarkdownTextFilter
from pipecat.services.cartesia.tts import CartesiaTTSService

# Create the filter
md_filter = MarkdownTextFilter()

# Use with TTS service
tts = CartesiaTTSService(
    api_key=os.getenv("CARTESIA_API_KEY"),
    voice_id="voice_id_here",
    text_filter=md_filter
)
```

### Custom Configuration

```python  theme={null}
# Create filter that removes code blocks and tables
md_filter = MarkdownTextFilter(
    params=MarkdownTextFilter.InputParams(
        filter_code=True,
        filter_tables=True
    )
)
```

## What Gets Removed

| Markdown Feature           | Example                  | Result       |
| -------------------------- | ------------------------ | ------------ |
| Bold                       | `**important**`          | `important`  |
| Italic                     | `*emphasized*`           | `emphasized` |
| Headers                    | `## Section`             | `Section`    |
| Code (inline)              | `` `code` ``             | `code`       |
| Code blocks (when enabled) | ` ```python\ncode\n``` ` | ` `          |
| Tables (when enabled)      | `\|A\|B\|\n\|--\|--\|`   | ` `          |
| HTML tags                  | `<em>text</em>`          | `text`       |
| Repeated characters        | `!!!!!!!`                | `!`          |

## Notes

* Preserves sentence structure and readability
* Maintains whitespace that affects speech prosody
* Handles streaming text with partial Markdown elements
* Efficiently converts HTML entities to plain text characters
* Smart handling of code blocks and tables with state tracking
* Integrates directly with TTS services in the Pipecat pipeline

---

# Source: https://docs.pipecat.ai/server/utilities/mcp/mcp.md

> ## Documentation Index
> Fetch the complete documentation index at: https://docs.pipecat.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# MCPClient

> Service to connect to MCP (Model Context Protocol) servers

## Overview

MCP is an open standard for enabling AI agents to interact with external data and tools. `MCPClient` provides a way to access and call tools via MCP. For example, instead of writing bespoke function call implementations for an external API, you may use an MCP server that provides a bridge to the API. *Be aware there may be security implications.* See [MCP documenation](https://github.com/modelcontextprotocol) for more details.

## Installation

To use `MCPClient`, install the required dependencies:

```bash  theme={null}
pip install "pipecat-ai[mcp]"
```

You may also need to set environment variables as required by the specific MCP server to which you are connecting.

## Configuration

### Constructor Parameters

You can connect to your MCP server via Stdio or SSE transport. See [here](https://modelcontextprotocol.io/docs/concepts/transports#built-in-transport-types) for more documentation on MCP transports.

<ParamField path="server_params" type="str | StdioServerParameters" required>
  You can provide either:

  * URL: "[https://your.mcp.server/sse](https://your.mcp.server/sse)"
  * StdioServerParameters, which are defined as:

  ```python  theme={null}
    StdioServerParameters(
          command="python",  # Executable
          args=["example_server.py"],  # Optional command line arguments
          env=None,  # Optional environment variables
      )
  ```
</ParamField>

### Input Parameters

See more information regarding server params [here](https://github.com/modelcontextprotocol/python-sdk?tab=readme-ov-file#writing-mcp-clients).

## Usage Example

### MCP Stdio Transport Implementation

```python  theme={null}

# Import MCPClient and StdioServerParameters
...
from mcp import StdioServerParameters
from pipecat.services.mcp_service import MCPClient
...

# Initialize an LLM
llm = ...

# Initialize and configure MCPClient with server parameters
mcp = MCPClient(
        server_params=StdioServerParameters(
            command=shutil.which("npx"),
            args=["-y", "@name/mcp-server-name@latest"],
            env={"ENV_API_KEY": "<env_api_key>"},
        )
    )

# Create tools schema from the MCP server and register them with llm
tools = await mcp.register_tools(llm)

# Create context with system message and tools
# Tip: Let the LLM know it has access to tools from an MCP server by including it in the system prompt.
context = LLMContext(
    messages=[
        {
            "role": "system",
            "content": "You are a helpful assistant in a voice conversation. You have access to MCP tools. Keep responses concise."
        }
    ],
    tools=tools
)
```

### MCP SSE Transport Implementation

```python  theme={null}

# Import MCPClient
...
from pipecat.services.mcp_service import MCPClient
...

# Initialize an LLM
llm = ...

# Initialize and configure MCPClient with MCP SSE server url
mcp = MCPClient(server_params="https://your.mcp.server/sse")

# Create tools schema from the MCP server and register them with llm
tools = await mcp.register_tools(llm)

# Create context with system message and tools
# Tip: Let the LLM know it has access to tools from an MCP server by including it in the system prompt.
context = LLMContext(
    messages=[
        {
            "role": "system",
            "content": "You are a helpful assistant in a voice conversation. You have access to MCP tools. Keep responses concise."
        }
    ],
    tools=tools
)
```

## Methods

<ResponseField name="register_tools" type="async method">
  Converts MCP tools to Pipecat-friendly function definitions and registers the
  functions with the llm.
</ResponseField>

```python  theme={null}
async def register_tools(self, llm) -> ToolsSchema:
```

## Additional documentation

<Note>
  See [MCP's docs](https://github.com/modelcontextprotocol/python-sdk) for MCP
  related updates.
</Note>

---

# Source: https://docs.pipecat.ai/server/services/memory/mem0.md

> ## Documentation Index
> Fetch the complete documentation index at: https://docs.pipecat.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Mem0

> Long-term conversation memory service powered by Mem0

## Overview

`Mem0MemoryService` provides long-term memory capabilities for conversational agents by integrating with Mem0's API. It automatically stores conversation history and retrieves relevant past context based on the current conversation, enhancing LLM responses with persistent memory across sessions.

<CardGroup cols={2}>
  <Card title="Mem0 Memory API Reference" icon="code" href="https://reference-server.pipecat.ai/en/latest/api/pipecat.services.mem0.memory.html">
    Pipecat's API methods for Mem0 memory integration
  </Card>

  <Card title="Example Implementation" icon="play" href="https://github.com/pipecat-ai/pipecat/blob/main/examples/foundational/37-mem0.py">
    Browse examples using Mem0 memory
  </Card>

  <Card title="Mem0 Documentation" icon="book" href="https://docs.mem0.ai/">
    Official Mem0 API documentation and guides
  </Card>

  <Card title="Mem0 Platform" icon="microphone" href="https://mem0.ai">
    Access memory services and manage API keys
  </Card>
</CardGroup>

## Installation

To use Mem0 memory services, install the required dependencies:

```bash  theme={null}
pip install "pipecat-ai[mem0]"
```

## Prerequisites

### Mem0 Account Setup

Before using Mem0 memory services, you need:

1. **Mem0 Account**: Sign up at [Mem0 Platform](https://mem0.ai)
2. **API Key**: Generate an API key from your account dashboard
3. **Host Configuration**: Set up your Mem0 host endpoint
4. **User Management**: Configure user IDs for memory association

### Required Environment Variables

* `MEM0_API_KEY`: Your Mem0 API key for authentication

### Configuration Options

* **User ID**: Unique identifier for associating memories with specific users
* **Agent ID**: Identifier for the agent using the memory service
* **Run ID**: Identifier for specific conversation sessions
* **Memory Retrieval**: Configure how past context is retrieved and used

### Key Features

* **Persistent Memory**: Long-term conversation history across sessions
* **Context Retrieval**: Automatic retrieval of relevant past conversations
* **User Association**: Memory tied to specific users for personalization
* **Session Management**: Track conversations across different runs and agents

---

# Source: https://docs.pipecat.ai/client/js/api-reference/messages.md

> ## Documentation Index
> Fetch the complete documentation index at: https://docs.pipecat.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Custom Messaging

The Pipecat JavaScript client can send and receive arbitrary messages to/from the server running the bot. This page outlines and demonstrates both client and server code for passing and responding to custom messages as well as providing arbitrary data at connection time.

## Connection-Time Configuration

Oftentimes clients need to provide configuration data to the server when starting the bot. This can include things like preferred language, user preferences, initial messages, or any other data that the server needs to know about the client. This must occur before the bot is started and therefore is not part of the RTVI standard, but rather a custom implementation. That said, the `PipecatClient` makes it easy to send this data as part of the `startBot()` and `startBotAndConnect()` methods, by passing an API endpoint object with the `requestData` property. Your server endpoint can then handle this data as needed. In the example below, we demonstrate sending an initial prompt and preferred language to the server when connecting.

<CodeGroup>
  ```javascript client theme={null}
  try {
      pcClient.startBotAndConnect({
          endpoint: '/api/start', // Your server endpoint to start the bot
          requestData: {
              initial_prompt: "You are a pirate captain",
              preferred_language: 'en-US'
          }
      });
  } catch (error) {
      console.error("Error starting the bot:", error);
  }
  ```

  ```python FastAPI endpoint theme={null}

  def validate_request_data(body: Dict[str, Any]) -> Tuple[str, str]:
      """Validate and extract prompt and language from request data."""
      if not isinstance(body, dict):
          raise ValueError("Request body must be a dictionary")

      prompt = body.get("initial_prompt", "You are a pirate captain")
      lang = body.get("preferred_language", "en-US")

      if not isinstance(prompt, str) or not isinstance(lang, str):
          raise ValueError("Both initial_prompt and preferred_language must be strings")

      return prompt, lang

  @app.post("/api/start")
  async def start(request: Request) -> Dict[Any, Any]:
      """Startup endpoint that creates a room, starts the bot, and returns
         connection credentials.

      This endpoint is called by clients to kick off a session and establish
      a connection.

      Returns:
          Dict[Any, Any]: Authentication bundle containing room_url and token

      Raises:
          HTTPException: If room creation, token generation, or bot startup fails
      """
      body = await request.json()
      try:
          prompt, lang = validate_request_data(body)
      except ValueError as e:
          raise HTTPException(status_code=400, info=f"Invalid request data: {e}")

      print("Creating room for RTVI connection", body)
      room_url, token = await create_room_and_token()
      print(f"Room URL: {room_url}")

      # Start the bot process
      try:
          proc = subprocess.Popen(
              [f"python3 -m bot -u {room_url} -t {token} -p {prompt} -l {lang}"],
              shell=True,
              bufsize=1,
              cwd=os.path.dirname(os.path.abspath(__file__)),
          )
          bot_procs[proc.pid] = (proc, room_url)
      except Exception as e:
          raise HTTPException(status_code=500, detail=f"Failed to start subprocess: {e}")

      # Return the authentication bundle in format expected by DailyTransport
      return {"url": room_url, "token": token}
  ```

  ```python bot theme={null}
  import argparse
  import asyncio

  def extract_arguments():
      parser = argparse.ArgumentParser(description="Example")
      parser.add_argument(
          "-u", "--room-url", type=str, default=os.getenv("DAILY_SAMPLE_ROOM_URL", "")
      )
      parser.add_argument(
          "-t", "--token", type=str, default=os.getenv("DAILY_SAMPLE_ROOM_TOKEN", None)
      )
      parser.add_argument(
          "-p", "--prompt", type=str, default="You are a pirate captain"
      )
      parser.add_argument("-l", "--language", type=str, default="en-US")
      return parser.parse_args()

  async def main():
      args = extract_arguments()
      print(f"room_url: {args.room_url}")

      daily_transport = DailyTransport(
          args.room_url,
          args.token,
          "Chatbot",
          DailyParams(
              audio_in_enabled=True,
              audio_out_enabled=True,
          ),
      )

      llm = GeminiLiveLLMService(
          api_key=os.getenv("GOOGLE_API_KEY"),
          voice_id="Puck",
          system_instruction=SYSTEM_INSTRUCTION,
          params=InputParams(
              language=args.language,  # Pass preferred language to LLM
          ),
      )

      messages = [ { role: "system", content: args.prompt } ]
      context = LLMContext(messages)
      user_aggregator, assistant_aggregator = LLMContextAggregatorPair(
          context,
          user_params=LLMUserAggregatorParams(
              vad_analyzer=SileroVADAnalyzer(params=VADParams(stop_secs=0.2)),
          ),
      )

      pipeline = Pipeline(
          [
              daily_transport.input(),
              user_aggregator,
              llm,
              daily_transport.output(),
              assistant_aggregator,
          ]
      )

      task = PipelineTask(pipeline)

      @rtvi.event_handler("on_client_ready")
      async def on_client_ready(rtvi):
          logger.debug("Client ready")
          # Kick off the conversation
          await task.queue_frames([LLMRunFrame()])

      @transport.event_handler("on_client_disconnected")
      async def on_client_disconnected(transport, client):
          logger.debug(f"Client disconnected")
          await task.cancel()

      runner = PipelineRunner(handle_sigint=False)

      await runner.run(task)

  if __name__ == "__main__":
      asyncio.run(main())
  ```
</CodeGroup>

## Sending Custom Messages to the Server

Once connected, you can send custom messages to the server using the `sendClientMessage` method. This is useful for triggering specific actions or sending data that the server needs to process.

<CodeGroup>
  ```javascript client theme={null}
  try {
      pcClient.sendClientMessage('set-language', { language: 'en-US' });
  } catch (error) {
      console.error("Error sending message to server:", error);
  }
  ```

  ```python bot theme={null}
      task = PipelineTask(
          pipeline,
          params,
          observers=[RTVIObserver(rtvi)],
      )

      @rtvi.event_handler("on_client_message")
      async def on_client_message(rtvi, msg):
          print("RTVI client message:", msg.type, msg.data)
          if msg.type == "set-language":
              language = msg.data.get("language", "en-US")
              await task.queue_frames([STTUpdateSettingsFrame(language=language)])

  ### Alternatively, if your message requires asynchronous processing or storing of
  ### state, you may want to handle it from inside a FrameProcessor, listen for a
  ### RTVIClientMessageFrame and push a RTVIServerResponseFrame
  class CustomFrameProcessor(FrameProcessor):
      async def process_frame(self, frame: Frame, direction: FrameDirection):
          await super().process_frame(frame, direction)
          if isinstance(frame, RTVIClientMessageFrame):
              print("RTVI client message:", frame.msg_id, frame.type, frame.data)
              if frame.type == "set-language":
                  language = frame.data.get("language", "en-US")
                  await self.push_frame(STTUpdateSettingsFrame(language=language))
                  return
          await self.push_frame(frame, direction)
  ```
</CodeGroup>

## Requesting Information from the Server

You can also request information from the server using the `sendClientRequest` method. This is useful for querying the server for specific data or triggering and action and getting a success/failure response.

<CodeGroup>
  ```javascript client theme={null}
  try {
      const response = await pcClient.sendClientRequest('get-language');
      console.log("Current language:", response.language);
  } catch (error) {
      console.error("Error requesting data from server:", error);
  }
  ```

  ```python bot theme={null}
  @rtvi.event_handler("on_client_message")
  async def on_client_message(rtvi, msg):
      print("RTVI client message:", msg.type, msg.data)
      if msg.type == "get-language":
          await rtvi.send_server_response(msg, {"language": get_current_language()})
      else:
          await rtvi.send_error_response(msg, "Unknown request type")

  ### Alternatively, if your message requires asynchronous processing or storing of
  ### state, you may want to handle it from inside a FrameProcessor, listen for a
  ### RTVIClientMessageFrame and push a RTVIServerResponseFrame
  class CustomFrameProcessor(FrameProcessor):
      async def process_frame(self, frame: Frame, direction: FrameDirection):
          await super().process_frame(frame, direction)
          if isinstance(frame, RTVIClientMessageFrame):
              print("RTVI client message:", frame.msg_id, frame.type, frame.data)
              if frame.type == "get-language":
                  data = {"language": get_current_language()}
                  await self.push_frame(
                      RTVIServerResponseFrame(
                          client_msg=frame,
                          data=data,
                      ),
                  )
                  return
              else:
                  await self.push_frame(
                      RTVIServerResponseFrame(
                          client_msg=frame,
                          error="Unknown request type"
                      )
                  )
          await self.push_frame(frame, direction)
  ```
</CodeGroup>

## Handling Custom Messages from the Server

You can handle custom messages sent from the server using the `onServerMessage` callback. This allows you to process messages that the server sends back to the client, such as notifications or updates.

<CodeGroup>
  ```javascript client theme={null}
  pcClient.onServerMessage((message) => {
      console.log("Received message from server:", message);
      if (message.data.msg === 'language-updated') {
          console.log("Language updated to:", message.data.language);
      }
  });
  ```

  ```python bot theme={null}
  ## From inside an Observer, call `send_server_message` directly on your rtvi instance
  class CustomObserver(BaseObserver):
      async def on_push_frame(self, data: FramePushed):
          if isinstance(frame, STTUpdateSettingsFrame):
              for key, value in settings.items():
                  if key == "language":
                      await rtvi.send_server_message({
                          "msg": "language-updated",
                          "language": value
                      })

  ### Alternatively, from inside a FrameProcessor, push a RTVIServerMessageFrame
  class CustomFrameProcessor(FrameProcessor):
      async def process_frame(self, frame: Frame, direction: FrameDirection):
          await super().process_frame(frame, direction)
          if isinstance(frame, STTUpdateSettingsFrame):
              for key, value in settings.items():
                  if key == "language":
                      await self.push_frame(
                          RTVIServerMessageFrame(
                              data={
                                  "msg": "language-updated",
                                  "language": value
                              }
                          )
                      )
          await self.push_frame(frame, direction)
  ```
</CodeGroup>

---

# Source: https://docs.pipecat.ai/guides/fundamentals/metrics.md

> ## Documentation Index
> Fetch the complete documentation index at: https://docs.pipecat.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Metrics

> Learn how to monitor performance and LLM/TTS usage with Pipecat.

When developing real-time, multimodal AI applications, monitoring two key
factors is crucial: performance (latency) and LLM/TTS usage. Performance impacts
user experience, while usage can affect operational costs. Pipecat offers
built-in metrics for both, which can be enabled with straightforward
configuration options.

## Enabling performance metrics

Set `enable_metrics=True` in `PipelineParams` when creating a task:

```python Example config theme={null}
task = PipelineTask(
            pipeline,
            params=PipelineParams(
                ...
                enable_metrics=True,
                ...
            ),
        )
```

Once enabled, Pipecat logs the following metrics:

| Metric          | Description                                     |
| --------------- | ----------------------------------------------- |
| TTFB            | Time To First Byte in seconds                   |
| Processing Time | Time taken by the service to respond in seconds |

```console Sample output theme={null}
AnthropicLLMService#0 TTFB: 0.8378312587738037
CartesiaTTSService#0 processing time: 0.0005071163177490234
CartesiaTTSService#0 TTFB: 0.17177796363830566
AnthropicLLMService#0 processing time: 2.4927797317504883
```

### Limiting TTFB responses

If you only want the **first** TTFB measurement for each service, you can
optionally pass `report_only_initial_ttfb=True` in `PipelineParams`:

```python Example config theme={null}
task = PipelineTask(
            pipeline,
            params=PipelineParams(
                ...
                enable_metrics=True,
                report_only_initial_ttfb=True,
                ...
            ),
        )
```

> **Note:** `enable_metrics=True` is required for this setting to have an
> effect.

## Enabling LLM/TTS Usage Metrics

Set `enable_usage_metrics=True` in PipelineParams when creating a task:

```python Example config theme={null}
task = PipelineTask(
            pipeline,
            params=PipelineParams(
                ...
                enable_usage_metrics=True,
                ...
            ),
        )
```

Pipecat will log the following as applicable:

| Metric    | Description                                 |
| --------- | ------------------------------------------- |
| LLM Usage | Number of prompt and completion tokens used |
| TTS Usage | Number of characters processed              |

```console Sample output theme={null}
CartesiaTTSService#0 usage characters: 65
AnthropicLLMService#0 prompt tokens: 104, completion tokens: 53
```

> **Note:** Usage metrics are recorded per interaction and do not represent
> running totals.

## Capturing Metrics Data

When metrics are enabled, Pipecat will emit a `MetricsFrame` for each interaction. The `MetricsFrame` contains:

* TTFB data
* Processing time
* Token usage for LLMs
* Character usage for TTS
* Smart Turn prediction data

You can access the metrics data by either adding a custom [FrameProcessor](/guides/fundamentals/custom-frame-processor) to your pipeline or adding an [observer](/server/utilities/observers/observer-pattern) to monitor `MetricsFrame`s.

### Example: Using a Custom FrameProcessor

Create a custom FrameProcessor to handle metrics data. Here's an example Metrics Processor that can be added to your pipeline after the TTS processor.

```python  theme={null}
from pipecat.frames.frames import MetricsFrame
from pipecat.metrics.metrics import (
    LLMUsageMetricsData,
    ProcessingMetricsData,
    TTFBMetricsData,
    TTSUsageMetricsData,
)
from pipecat.processors.frame_processor import FrameDirection, FrameProcessor

class MetricsLogger(FrameProcessor):
    async def process_frame(self, frame: Frame, direction: FrameDirection):
        await super().process_frame(frame, direction)

        if isinstance(frame, MetricsFrame):
            for d in frame.data:
                if isinstance(d, TTFBMetricsData):
                    print(f"!!! MetricsFrame: {frame}, ttfb: {d.value}")
                elif isinstance(d, ProcessingMetricsData):
                    print(f"!!! MetricsFrame: {frame}, processing: {d.value}")
                elif isinstance(d, LLMUsageMetricsData):
                    tokens = d.value
                    print(
                        f"!!! MetricsFrame: {frame}, tokens: {tokens.prompt_tokens}, characters: {tokens.completion_tokens}"
                    )
                elif isinstance(d, TTSUsageMetricsData):
                    print(f"!!! MetricsFrame: {frame}, characters: {d.value}")
        await self.push_frame(frame, direction)
```

---

# Source: https://docs.pipecat.ai/client/react/migration-guide.md

# Source: https://docs.pipecat.ai/client/react-native/migration-guide.md

# Source: https://docs.pipecat.ai/client/migration-guide.md

# Source: https://docs.pipecat.ai/client/js/migration-guide.md

# Source: https://docs.pipecat.ai/client/ios/migration-guide.md

# Source: https://docs.pipecat.ai/client/android/migration-guide.md

> ## Documentation Index
> Fetch the complete documentation index at: https://docs.pipecat.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# RTVIClient Migration Guide for Android

> A Guide to migrating from RTVIClient to PipecatClient for Android

This guide covers migrating from the `RTVIClient` to the new `PipecatClient` in an Android application. The new client introduces simplified configuration, modern messaging patterns, and improved function call handling. For an overview of the changes, see the top-level [RTVIClient Migration Guide](/client/migration-guide).

## Key Changes

### 1. Package and Class Names

**Old**

```kotlin  theme={null}
import ai.pipecat.client.RTVIClient
import ai.pipecat.client.RTVIClientOptions
import ai.pipecat.client.RTVIClientParams
import ai.pipecat.client.RTVIEventCallbacks
```

**New**

```kotlin  theme={null}
import ai.pipecat.client.PipecatClient
import ai.pipecat.client.PipecatClientOptions
// RTVIClientParams has been removed
import ai.pipecat.client.PipecatEventCallbacks

// Type alias for a Pipecat client using the Daily transport
import ai.pipecat.client.daily.PipecatClientDaily
```

### 2. Client and Transport Configuration

Previously, the `RTVIClient` constructor accepted a transport factory parameter, whereas in v1.0.0 the transport is passed in directly.

Parameters such as the connection URL are no longer passed in at construction time. These should be given to `startBot()`, `startBotAndConnect()`, or `connect()`.

**Old**

```kotlin  theme={null}
val options = RTVIClientOptions(
    params = RTVIClientParams(
        baseUrl = baseUrl,
        endpoints = RTVIURLEndpoints(),
    )
)

val client = RTVIClient(DailyTransport.Factory(context), callbacks, options)
```

**New**

```kotlin  theme={null}
val options = PipecatClientOptions(callbacks = callbacks)
val client = PipecatClient(DailyTransport(context), options)
```

### 3. Connection

Previously, the `connect()` method did a POST request to your bot's HTTP endpoint, which returned the connection details (such as room URL and token), before making the actual connection.

This functionality has been split out into `startBot()`, which provides more flexibility in how the request is performed.

A helper method `startBotAndConnect()` unifies the two steps and is the equivalent of the original `connect()` method. You can call `connect()` directly if you wish to skip the POST step.

**Old**

```kotlin  theme={null}
// Uses parameters passed into the constructor
client.connect().await()
```

**New**

```kotlin  theme={null}
// Parameters now passed in at connection time
client.startBotAndConnect(
    APIRequest(
        endpoint = url,
        requestData = Value.Object()
    )
).await()
```

### 4. Function Call Handling

Helpers like `LLMHelper` have been removed. You now register function call handlers directly on the `PipecatClient`.

```kotlin  theme={null}
client.registerFunctionCallHandler("get_my_current_location") { data, onResult ->
    thread {
        val location = getCurrentLocation(data.args)
        onResult(location)
    }
}
```

### 5. Pipeline Configuration Initialization

Previously, you could provide a pipeline configuration as part of the `RTVIClient` constructor, and it was expected to be in a specific format. Now, if you would like to pass any initial pipeline configurations, you do so as `requestData` added to the endpoint you provide to `startBot()` or `startBotAndConnect()`. In both cases, you would need server-side code to parse and apply these settings, but now you can define the structure and what pieces of configuration you want to send.

Check out [this section of docs](../js/api-reference/messages#connection-time-configuration) for an example, complete with server-side code showing how to initialize the pipeline configuration at connection time.

### 6. LLM Context Updates

Previously, context updates were done via helpers. Now, use `appendToContext()`:

```kotlin  theme={null}
client.appendToContext(
    message = LLMContextMessage(
        role = LLMContextMessage.Role.User,
        content = "Tell me a joke.",
        runImmediately = true
    )
)
```

### 7. Pipeline Configuration Updates

Previously, the client supported updating the pipeline configuration using a specific method that took a configuration object in a generic format. Dynamic and predefined configuration updates, however, are a security concern, allowing clients to override settings and potentially abuse API keys. For this reason, it has been removed and most configuration updates need to be handled custom by your application. To do so, you should take advantage of the client-server messaging system, which allows you to send messages to the server and handle responses. This way, you can implement your own logic for updating configurations securely.

New messaging types replace old action/config helpers:

```kotlin  theme={null}
client.sendClientMessage(msgType = "set-voice", data = Value.Str("Janice")).await()

val llmVendor = client.sendClientRequest(msgType = "get-llm-vendor").await().data?.content
Log.i("Example", "LLM Vendor: $llmVendor")
```

Check out [this section of docs](../js/api-reference/messages#sending-custom-messages-to-the-server) for a more complete example, along with an example on making a request (`sendClientRequest()`) to wait for a response.

### 8. Disconnecting

```kotlin  theme={null}
client.unregisterFunctionCallHandler()
client.disconnect().await()
```

## Breaking Changes

1. **Constructor Params Removed**: No pipeline or endpoint config in constructor.
2. **Helpers Removed**: RTVIClientHelper, LLMHelper are gone.
3. **Configs Removed**: All configuration-related methods, events, and types have been removed: `getConfig()`, `updateConfig()`, `describeConfig()`, `onConfigUpdated`, `onConfigDescribed`, etc.
4. **Actions Removed**: All actions-related methods, events, and types have been removed: `action()`, `describeActions()`, `onActionsAvailable`, etc.
5. **New Messaging Methods**:

   * `appendToContext()`
   * `sendClientRequest()`
   * `sendClientMessage()`
   * `disconnectBot()`
   * `registerFunctionCallHandler()` / `unregisterFunctionCallHandler()` / `unregisterAllFunctionCallHandlers()`

---

# Source: https://docs.pipecat.ai/server/services/tts/minimax.md

> ## Documentation Index
> Fetch the complete documentation index at: https://docs.pipecat.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# MiniMax

> Text-to-speech service implementation using MiniMax T2A API

## Overview

`MiniMaxTTSService` provides high-quality text-to-speech synthesis using MiniMax's T2A (Text-to-Audio) API with streaming capabilities, emotional voice control, and support for multiple languages. The service offers various models optimized for different use cases, from low-latency to high-definition audio quality.

<CardGroup cols={2}>
  <Card title="MiniMax TTS API Reference" icon="code" href="https://reference-server.pipecat.ai/en/latest/api/pipecat.services.minimax.tts.html">
    Pipecat's API methods for MiniMax TTS integration
  </Card>

  <Card title="Example Implementation" icon="play" href="https://github.com/pipecat-ai/pipecat/blob/main/examples/foundational/07y-interruptible-minimax.py">
    Complete example with emotional voice settings
  </Card>

  <Card title="MiniMax Documentation" icon="book" href="https://www.minimax.io/platform/document/T2A%20V2?key=66719005a427f0c8a5701643">
    Official MiniMax T2A API documentation
  </Card>

  <Card title="MiniMax Platform" icon="microphone" href="https://www.minimax.io/platform/">
    Access voice models and API credentials
  </Card>
</CardGroup>

## Installation

To use MiniMax services, no additional dependencies are required beyond the base installation:

```bash  theme={null}
pip install "pipecat-ai"
```

## Prerequisites

### MiniMax Account Setup

Before using MiniMax TTS services, you need:

1. **MiniMax Account**: Sign up at [MiniMax Platform](https://www.minimax.io/platform/)
2. **API Credentials**: Get your API key and Group ID from the platform
3. **Voice Selection**: Choose from available voice models and emotional settings

### Required Environment Variables

* `MINIMAX_API_KEY`: Your MiniMax API key for authentication
* `MINIMAX_GROUP_ID`: Your MiniMax group ID

---

# Source: https://docs.pipecat.ai/deployment/platforms/modal.md

> ## Documentation Index
> Fetch the complete documentation index at: https://docs.pipecat.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Modal

> Deploy Pipecat applications to Modal

[Modal](https://www.modal.com) is well-suited for Pipecat deployments because it handles container orchestration, scaling, and cold starts efficiently. This makes it a good choice for production Pipecat bots that need reliable performance.

This guide walks through the Modal example included in the Pipecat repository, which follows the same [deployment pattern](../pattern).

<Card title="Modal example" icon="github" href="https://github.com/pipecat-ai/pipecat-examples/tree/main/deployment/modal-example">
  View the complete Modal deployment example in our GitHub repository
</Card>

<Frame>
    <img
      src="https://mintcdn.com/daily/2bYrACcmgvvzC075/deployment/images/modal.jpg?fit=max&auto=format&n=2bYrACcmgvvzC075&q=85&s=a3c0c2dfc5fc2d755ecfd0367dfa6016"
      alt="Diagram of the deployment architecture of Modal and Pipecat
  example"
      data-og-width="1545"
      width="1545"
      data-og-height="794"
      height="794"
      data-path="deployment/images/modal.jpg"
      data-optimize="true"
      data-opv="3"
      srcset="https://mintcdn.com/daily/2bYrACcmgvvzC075/deployment/images/modal.jpg?w=280&fit=max&auto=format&n=2bYrACcmgvvzC075&q=85&s=d5214756446b55479255ff566731c1ea 280w, https://mintcdn.com/daily/2bYrACcmgvvzC075/deployment/images/modal.jpg?w=560&fit=max&auto=format&n=2bYrACcmgvvzC075&q=85&s=a9b57986a465e6e82d0f81a6ae279ec9 560w, https://mintcdn.com/daily/2bYrACcmgvvzC075/deployment/images/modal.jpg?w=840&fit=max&auto=format&n=2bYrACcmgvvzC075&q=85&s=0b40626412faa2afa740fd07c029defe 840w, https://mintcdn.com/daily/2bYrACcmgvvzC075/deployment/images/modal.jpg?w=1100&fit=max&auto=format&n=2bYrACcmgvvzC075&q=85&s=291766e70af63cc0e65f1c90c2e538ee 1100w, https://mintcdn.com/daily/2bYrACcmgvvzC075/deployment/images/modal.jpg?w=1650&fit=max&auto=format&n=2bYrACcmgvvzC075&q=85&s=912ccfdf4ceaf304069d805f58772b65 1650w, https://mintcdn.com/daily/2bYrACcmgvvzC075/deployment/images/modal.jpg?w=2500&fit=max&auto=format&n=2bYrACcmgvvzC075&q=85&s=a3d70e9d962ce7fa2654e93d02fcfb24 2500w"
    />
</Frame>

## Install the Modal CLI

<Card title="Set up Modal" icon="rocket" href="https://modal.com/docs/guide#getting-started">
  Follow Modal's official instructions for creating an account and setting up
  the CLI
</Card>

## Deploy a self-serve LLM

1. Deploy Modal's OpenAI-compatible LLM service:

   ```bash  theme={null}
   git clone https://github.com/modal-labs/modal-examples
   cd modal-examples
   modal deploy 06_gpu_and_ml/llm-serving/vllm_inference.py
   ```

<Info>
  Refer to Modal's guide and example for [Deploying an OpenAI-compatible LLM
  service with vLLM](https://modal.com/docs/examples/vllm_inference) for more
  details.
</Info>

2. Take note of the endpoint URL from the previous step, which will look like:
   ```
   https://{your-workspace}--example-vllm-openai-compatible-serve.modal.run
   ```
   You'll need this for the `bot_vllm.py` file in the next section.

<Note>
  The default Modal LLM example uses Llama-3.1 and will shut down after 15
  minutes of inactivity. Cold starts take 5-10 minutes. To prepare the service,
  we recommend visiting the `/docs` endpoint (`https://<Modal workspace>--example-vllm-openai-compatible-serve.modal.run/docs`)
  for your deployed LLM and wait for it to fully load before connecting your client.
</Note>

## Deploy FastAPI App and Pipecat pipeline to Modal

1. Setup environment variables:

```bash  theme={null}
cd server
cp env.example .env
# Modify .env to provide your service API Keys
```

<Info>
  Alternatively, you can configure your Modal app to use
  [secrets](https://modal.com/docs/guide/secrets).
</Info>

2. Update the `modal_url` in `server/src/bot_vllm.py` to point to the URL you received from the self-serve LLM deployment in the previous step.

3. From within the `server` directory, test the app locally:

```bash  theme={null}
modal serve app.py
```

4. Deploy to production:

```bash  theme={null}
modal deploy app.py
```

5. Note the endpoint URL produced from this deployment. It will look like:

```bash  theme={null}
https://{your-workspace}--pipecat-modal-fastapi-app.modal.run
```

You'll need this URL for the client's `app.js` configuration mentioned in its README.

## Launch your bots on Modal

### Option 1: Direct Link

Simply click on the URL displayed after running the server or deploy step to launch an agent and be redirected to a Daily room to talk with the launched bot. This will use the OpenAI pipeline.

### Option 2: Connect via an RTVI Client

Follow the instructions provided in the [client folder's README](https://github.com/pipecat-ai/pipecat-examples/tree/main/deployment/modal-example/client/javascript) for building and running a custom client that connects to your Modal endpoint. The provided client includes a dropdown for choosing which bot pipeline to run.

## Navigating your LLM, server, and Pipecat logs

On your [Modal dashboard](https://modal.com/apps), you should have two Apps listed under Live Apps:

1. `example-vllm-openai-compatible`: This App contains the containers and logs used to run your self-hosted LLM. There will be just one App Function listed: `serve`. Click on this function to view logs for your LLM.
2. `pipecat-modal`: This App contains the containers and logs used to run your `connect` endpoints and Pipecat pipelines. It will list two App Functions:
   1. `fastapi_app`: This function is running the endpoints that your client will interact with and initiate starting a new pipeline (`/`, `/connect`, `/status`). Click on this function to see logs for each endpoint hit.
   2. `bot_runner`: This function handles launching and running a bot pipeline. Click on this function to get a list of all pipeline runs and access each run's logs.

## Modal & Pipecat Tips

* In most other Pipecat examples, we use `Popen` to launch the pipeline process from the `/connect` endpoint. In this example, we use a Modal function instead. This allows us to run the pipelines using a separately defined Modal image as well as run each pipeline in an isolated container.
* For the FastAPI and most common Pipecat Pipeline containers, a default `debian_slim` CPU-only should be all that's required to run. GPU containers are needed for self-hosted services.
* To minimize cold starts of the pipeline and reduce latency for users, set `min_containers=1` on the Modal Function that launches the pipeline to ensure at least one warm instance of your function is always available.

## Next steps

<Card title="Explore Modal's LLM Examples" icon="layer-group" href="https://modal.com/docs/examples/vllm_inference">
  For next steps on running a self-hosted LLM and reducing latency, check out
  all of Modal's LLM examples
</Card>

---

# Source: https://docs.pipecat.ai/server/services/vision/moondream.md

> ## Documentation Index
> Fetch the complete documentation index at: https://docs.pipecat.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Moondream

> Vision service implementation using Moondream for local image analysis and question answering

## Overview

`MoondreamService` provides local image analysis and question-answering capabilities using the Moondream model. It runs entirely on your local machine, supporting various hardware acceleration options including CUDA, Intel XPU, and Apple MPS for privacy-focused computer vision applications.

<CardGroup cols={2}>
  <Card title="Moondream Vision API Reference" icon="code" href="https://reference-server.pipecat.ai/en/latest/api/pipecat.services.moondream.vision.html">
    Pipecat's API methods for Moondream vision integration
  </Card>

  <Card title="Example Implementation" icon="play" href="https://github.com/pipecat-ai/pipecat/blob/main/examples/foundational/12-describe-video.py">
    Browse examples using Moondream vision
  </Card>

  <Card title="Moondream Documentation" icon="book" href="https://www.moondream.ai/">
    Official Moondream model documentation
  </Card>

  <Card title="Hugging Face Model" icon="microphone" href="https://huggingface.co/vikhyatk/moondream2">
    Access Moondream model on Hugging Face
  </Card>
</CardGroup>

## Installation

To use Moondream services, install the required dependencies:

```bash  theme={null}
pip install "pipecat-ai[moondream]"
```

## Prerequisites

### Local Model Setup

Before using Moondream vision services, you need:

1. **Model Download**: First run will automatically download the Moondream model from Hugging Face
2. **Hardware Configuration**: Set up CUDA, Intel XPU, or Apple MPS for optimal performance
3. **Storage Space**: Ensure sufficient disk space for model files
4. **Memory Requirements**: Adequate RAM/VRAM for model inference

### Hardware Acceleration

The service automatically detects and uses the best available hardware:

* **Intel XPU**: Requires intel\_extension\_for\_pytorch
* **NVIDIA CUDA**: For GPU acceleration
* **Apple Metal (MPS)**: For Apple Silicon optimization
* **CPU**: Fallback option for any system

### Configuration Options

* **Model Selection**: Choose Moondream model version and revision
* **Hardware Override**: Force CPU usage if needed
* **Local Processing**: Complete privacy with no external API calls

<Tip>
  No API keys required - Moondream runs entirely locally for complete privacy
  and control.
</Tip>

---

# Source: https://docs.pipecat.ai/server/services/tts/neuphonic.md

> ## Documentation Index
> Fetch the complete documentation index at: https://docs.pipecat.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Neuphonic

> Text-to-speech service implementation using Neuphonic's API

## Overview

Neuphonic provides high-quality text-to-speech synthesis with two service implementations: `NeuphonicTTSService` (WebSocket-based) with real-time streaming and interruption support, and `NeuphonicHttpTTSService` (HTTP-based) with server-sent events. `NeuphonicTTSService` is recommended for interactive applications requiring low latency.

<CardGroup cols={2}>
  <Card title="Neuphonic TTS API Reference" icon="code" href="https://reference-server.pipecat.ai/en/latest/api/pipecat.services.neuphonic.tts.html">
    Pipecat's API methods for Neuphonic TTS integration
  </Card>

  <Card title="Example Implementation" icon="play" href="https://github.com/pipecat-ai/pipecat/blob/main/examples/foundational/07v-interruptible-neuphonic.py">
    Complete example with WebSocket streaming
  </Card>

  <Card title="Neuphonic Documentation" icon="book" href="https://docs.neuphonic.com/api-reference/tts/websocket">
    Official Neuphonic TTS API documentation
  </Card>

  <Card title="Voice Library" icon="microphone" href="https://docs.neuphonic.com/">
    Browse available voices and features
  </Card>
</CardGroup>

## Installation

To use Neuphonic services, install the required dependencies:

```bash  theme={null}
pip install "pipecat-ai[neuphonic]"
```

## Prerequisites

### Neuphonic Account Setup

Before using Neuphonic TTS services, you need:

1. **Neuphonic Account**: Sign up at [Neuphonic](https://docs.neuphonic.com/)
2. **API Key**: Generate an API key from your account dashboard
3. **Voice Selection**: Choose from available voice models

### Required Environment Variables

* `NEUPHONIC_API_KEY`: Your Neuphonic API key for authentication

---

# Source: https://docs.pipecat.ai/getting-started/next-steps.md

> ## Documentation Index
> Fetch the complete documentation index at: https://docs.pipecat.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Continue Learning

> Continue your Pipecat journey

Now that you've run your first Pipecat bot, here's how to continue learning and build your own applications.

## Choose Your Path

<CardGroup cols={2}>
  <Card title="Learn Core Concepts" icon="book-open" href="/guides/learn/overview">
    Understand pipelines, processors, transports, and how to build custom AI
    applications from the ground up.
  </Card>

  <Card title="Explore Examples" icon="code" href="https://github.com/pipecat-ai/pipecat-examples">
    Dive into complete examples including multimodal bots, creative
    applications, and enterprise integrations.
  </Card>
</CardGroup>

## What's Your Goal?

Choose your path based on what you want to accomplish:

### 🚀 Build a Production Application

Ready to create your own bot? Use the CLI to scaffold a project tailored to your needs.

<Card title="Build Your Next Bot" icon="wand-magic-sparkles" href="/getting-started/build-your-next-bot">
  Scaffold phone or web/mobile projects with the Pipecat CLI
</Card>

### 🧠 Understand How Pipecat Works

Master the fundamentals to build custom solutions and debug effectively.

<CardGroup cols={2}>
  <Card title="Core Concepts" icon="book-open" href="/guides/learn/overview">
    Learn about pipelines, processors, transports, and context management
  </Card>

  <Card title="Common Features" icon="puzzle-piece" href="/guides/fundamentals/custom-frame-processor">
    Function calling, audio recording, transcripts, and custom processors
  </Card>
</CardGroup>

### 💡 Get Inspired

Explore real-world examples and recipes for common use cases.

<CardGroup cols={2}>
  <Card title="Browse Examples" icon="code" href="https://github.com/pipecat-ai/pipecat-examples">
    30+ production-ready examples including multimodal bots and games
  </Card>

  <Card title="Recipes" icon="book-open" href="/recipes">
    Common patterns and solutions for specific use cases
  </Card>
</CardGroup>

### 🚀 Deploy to Production

Scale your bot with managed hosting or self-hosted infrastructure.

<Card title="Deployment Options" icon="rocket" href="/deployment/overview">
  Compare Pipecat Cloud, Fly.io, Modal, and other platforms
</Card>

## Need Help?

<CardGroup cols={2}>
  <Card title="Discord Community" icon="discord" href="https://discord.gg/pipecat">
    Get support and share projects with other developers
  </Card>

  <Card title="GitHub Issues" icon="github" href="https://github.com/pipecat-ai/pipecat/issues">
    Report bugs or request features
  </Card>
</CardGroup>

---

# Source: https://docs.pipecat.ai/server/services/llm/nim.md

> ## Documentation Index
> Fetch the complete documentation index at: https://docs.pipecat.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# NVIDIA NIM

> LLM service implementation using NVIDIA's NIM (NVIDIA Inference Microservice) API with OpenAI-compatible interface

## Overview

`NvidiaLLMService` provides access to NVIDIA's NIM language models through an OpenAI-compatible interface. It inherits from `OpenAILLMService` and supports streaming responses, function calling, and context management, with special handling for NVIDIA's incremental token reporting and enterprise deployment.

<CardGroup cols={2}>
  <Card title="NVIDIA NIM LLM API Reference" icon="code" href="https://reference-server.pipecat.ai/en/latest/api/pipecat.services.nim.llm.html">
    Pipecat's API methods for NVIDIA NIM integration
  </Card>

  <Card title="Example Implementation" icon="play" href="https://github.com/pipecat-ai/pipecat/blob/main/examples/foundational/14j-function-calling-nim.py">
    Complete example with function calling
  </Card>

  <Card title="NVIDIA NIM Documentation" icon="book" href="https://docs.nvidia.com/nim/">
    Official NVIDIA NIM documentation and setup
  </Card>

  <Card title="NVIDIA Developer Portal" icon="microphone" href="https://developer.nvidia.com/">
    Access NIM services and manage API keys
  </Card>
</CardGroup>

## Installation

To use NVIDIA NIM services, install the required dependencies:

```bash  theme={null}
pip install "pipecat-ai[nvidia]"
```

## Prerequisites

### NVIDIA NIM Setup

Before using NVIDIA NIM LLM services, you need:

1. **NVIDIA Developer Account**: Sign up at [NVIDIA Developer Portal](https://developer.nvidia.com/)
2. **API Key**: Generate an NVIDIA API key for NIM services
3. **Model Selection**: Choose from available NIM-hosted models
4. **Enterprise Setup**: Configure NIM for on-premises deployment if needed

### Required Environment Variables

* `NVIDIA_API_KEY`: Your NVIDIA API key for authentication

---

# Source: https://docs.pipecat.ai/server/utilities/filters/null-filter.md

> ## Documentation Index
> Fetch the complete documentation index at: https://docs.pipecat.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# NullFilter

> Processor that blocks all frames except system frames

## Overview

`NullFilter` is a filtering processor that blocks all frames from passing through, with the exception of system frames and end frames which are required for proper pipeline operation.

## Constructor Parameters

The `NullFilter` constructor accepts no specific parameters beyond those inherited from `FrameProcessor`.

## Functionality

When a frame passes through the processor:

* If the frame is a `SystemFrame` or `EndFrame`, it is passed through
* All other frame types are blocked and do not continue through the pipeline

This filter effectively acts as a barrier that allows only the essential system frames required for pipeline initialization, shutdown, and management.

## Use Cases

`NullFilter` is useful in several scenarios:

* Temporarily disabling parts of a pipeline without removing components
* Creating dead-end branches in parallel pipelines
* Testing pipeline behavior with blocked communication
* Implementing conditional pipelines where certain paths should be blocked

## Usage Example

```python  theme={null}
from pipecat.processors.filters import NullFilter

# Create a null filter that blocks all non-system frames
blocker = NullFilter()

# Add to pipeline
pipeline = Pipeline([
    source,
    blocker,  # Blocks all regular frames
    destination  # Will only receive system frames
])
```

## Frame Flow

```mermaid  theme={null}
graph TD
    A[Input Frames] --> B[NullFilter]
    B --> C{System/End Frame?}
    C -->|Yes| D[Output Frame]
    C -->|No| E[Blocked]
```

## Notes

* Blocks all regular frames in both directions
* Only allows system frames and end frames to pass through
* Useful for testing, debugging, and creating conditional pipelines
* Minimal overhead as it performs simple type checking
* Can be used to temporarily disable parts of a pipeline

---

# Source: https://docs.pipecat.ai/server/utilities/observers/observer-pattern.md

> ## Documentation Index
> Fetch the complete documentation index at: https://docs.pipecat.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Observer Pattern

> Understanding and implementing observers in Pipecat

The Observer pattern in Pipecat allows non-intrusive monitoring of frames as they flow through the pipeline. Observers can watch frame traffic without affecting the pipeline's core functionality.

<Warning>DEPRECATED: The old observer pattern with individual parameters (`on_push_frame(src, dst, frame, direction, timestamp)`) is deprecated. Use the new pattern with data objects (`on_push_frame(data: FramePushed)`) instead.</Warning>

## Base Observer

All observers must inherit from `BaseObserver` and can implement two main methods:

* `on_push_frame(data: FramePushed)`: Called when a frame is pushed from one processor to another
* `on_process_frame(data: FrameProcessed)`: Called when a frame is being processed by a processor

```python  theme={null}
from pipecat.observers.base_observer import BaseObserver, FramePushed, FrameProcessed

class CustomObserver(BaseObserver):
    async def on_push_frame(self, data: FramePushed):
        # Your frame observation logic here
        pass
    
    async def on_process_frame(self, data: FrameProcessed):
        # Your frame processing observation logic here
        pass
```

## Available Observers

Pipecat provides several built-in observers:

* **LLMLogObserver**: Logs LLM activity and responses
* **TranscriptionLogObserver**: Logs speech-to-text transcription events
* **RTVIObserver**: Converts internal frames to RTVI protocol messages for server to client messaging

## Using Multiple Observers

You can attach multiple observers to a pipeline task. Each observer will be notified of all frames:

```python  theme={null}
task = PipelineTask(
    pipeline,
    params=PipelineParams(
        observers=[LLMLogObserver(), TranscriptionLogObserver(), CustomObserver()],
    ),
)
```

## Example: Debug Observer

Here's an example observer that logs interruptions and bot speaking events:

```python  theme={null}
from pipecat.observers.base_observer import BaseObserver, FramePushed, FrameProcessed
from pipecat.frames.frames import (
    StartInterruptionFrame,
    BotStartedSpeakingFrame,
    BotStoppedSpeakingFrame,
)
from pipecat.processors.frame_processor import FrameDirection
from loguru import logger

class DebugObserver(BaseObserver):
    """Observer to log interruptions and bot speaking events to the console.

    Logs all frame instances of:
    - StartInterruptionFrame
    - BotStartedSpeakingFrame
    - BotStoppedSpeakingFrame

    This allows you to see the frame flow from processor to processor through the pipeline for these frames.
    Log format: [EVENT TYPE]: [source processor] → [destination processor] at [timestamp]s
    """

    async def on_push_frame(self, data: FramePushed):
        time_sec = data.timestamp / 1_000_000_000
        arrow = "→" if data.direction == FrameDirection.DOWNSTREAM else "←"

        if isinstance(data.frame, StartInterruptionFrame):
            logger.info(f"⚡ INTERRUPTION START: {data.source} {arrow} {data.destination} at {time_sec:.2f}s")
        elif isinstance(data.frame, BotStartedSpeakingFrame):
            logger.info(f"🤖 BOT START SPEAKING: {data.source} {arrow} {data.destination} at {time_sec:.2f}s")
        elif isinstance(data.frame, BotStoppedSpeakingFrame):
            logger.info(f"🤖 BOT STOP SPEAKING: {data.source} {arrow} {data.destination} at {time_sec:.2f}s")
```

## Common Use Cases

Observers are particularly useful for:

* Debugging frame flow
* Logging specific events
* Monitoring pipeline behavior
* Collecting metrics
* Converting internal frames to external messages

---

# Source: https://docs.pipecat.ai/server/services/llm/ollama.md

> ## Documentation Index
> Fetch the complete documentation index at: https://docs.pipecat.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Ollama

> LLM service implementation using Ollama with OpenAI-compatible interface

## Overview

`OLLamaLLMService` provides access to locally-run Ollama models through an OpenAI-compatible interface. It inherits from `BaseOpenAILLMService` and allows you to run various open-source models locally while maintaining compatibility with OpenAI's API format for privacy and cost control.

<CardGroup cols={2}>
  <Card title="Ollama LLM API Reference" icon="code" href="https://reference-server.pipecat.ai/en/latest/api/pipecat.services.ollama.llm.html">
    Pipecat's API methods for Ollama integration
  </Card>

  <Card title="Example Implementation" icon="play" href="https://github.com/pipecat-ai/pipecat/tree/main/examples">
    Browse examples using Ollama models
  </Card>

  <Card title="Ollama Documentation" icon="book" href="https://ollama.com/library">
    Official Ollama documentation and model library
  </Card>

  <Card title="Download Ollama" icon="microphone" href="https://ollama.com/download">
    Download and setup instructions for Ollama
  </Card>
</CardGroup>

## Installation

To use Ollama services, you need to install both Ollama and the Pipecat dependency:

1. **Install Ollama** on your system from [ollama.com/download](https://ollama.com/download)
2. **Install Pipecat dependency**:

```bash  theme={null}
pip install "pipecat-ai[ollama]"
```

3. **Pull a model** (first time only):

```bash  theme={null}
ollama pull llama2
```

## Prerequisites

### Ollama Local Setup

Before using Ollama LLM services, you need:

1. **Ollama Installation**: Download and install Ollama from [ollama.com](https://ollama.com/download)
2. **Model Selection**: Pull your desired models (llama2, mistral, codellama, etc.)
3. **Local Service**: Ensure Ollama service is running (default port 11434)
4. **Hardware**: Sufficient RAM and storage for your chosen models

### Configuration

* **No API Keys Required**: Ollama runs entirely locally
* **Model Management**: Use `ollama pull <model>` to download models
* **Service URL**: Default is `http://localhost:11434` (configurable)

<Tip>
  Ollama runs as a local service on port 11434. No API key required for complete
  privacy!
</Tip>

---

# Source: https://docs.pipecat.ai/guides/features/openai-audio-models-and-apis.md

> ## Documentation Index
> Fetch the complete documentation index at: https://docs.pipecat.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Building With OpenAI Audio Models and APIs

> Create voice agents with OpenAI audio models and Pipecat

This guide provides an overview of the audio capabilities OpenAI offers via their APIs. We'll also link to Pipecat sample code.

## Two Ways To Build Voice-to-voice

You can build voice-to-voice applications in two ways:

1. The cascaded models approach, using separate models for transcription, the LLM, and voice generation.

<Frame><img src="https://mintcdn.com/daily/2bYrACcmgvvzC075/images/openai-cascade.jpg?fit=max&auto=format&n=2bYrACcmgvvzC075&q=85&s=e2e389318021ff76c15da1734c28d49d" alt="OpenAI Cascaded Pipeline" data-og-width="2877" width="2877" data-og-height="868" height="868" data-path="images/openai-cascade.jpg" data-optimize="true" data-opv="3" srcset="https://mintcdn.com/daily/2bYrACcmgvvzC075/images/openai-cascade.jpg?w=280&fit=max&auto=format&n=2bYrACcmgvvzC075&q=85&s=2ca91677fd50766e17d22df889b03678 280w, https://mintcdn.com/daily/2bYrACcmgvvzC075/images/openai-cascade.jpg?w=560&fit=max&auto=format&n=2bYrACcmgvvzC075&q=85&s=1d4cbb071ef2ba79bcf7044a725d2ab8 560w, https://mintcdn.com/daily/2bYrACcmgvvzC075/images/openai-cascade.jpg?w=840&fit=max&auto=format&n=2bYrACcmgvvzC075&q=85&s=339ce699cfbdc88094b8ca90a5e699f8 840w, https://mintcdn.com/daily/2bYrACcmgvvzC075/images/openai-cascade.jpg?w=1100&fit=max&auto=format&n=2bYrACcmgvvzC075&q=85&s=bab07cf5169f9b0003260f4163cb348b 1100w, https://mintcdn.com/daily/2bYrACcmgvvzC075/images/openai-cascade.jpg?w=1650&fit=max&auto=format&n=2bYrACcmgvvzC075&q=85&s=1a03211a75161830777a34500702fe5a 1650w, https://mintcdn.com/daily/2bYrACcmgvvzC075/images/openai-cascade.jpg?w=2500&fit=max&auto=format&n=2bYrACcmgvvzC075&q=85&s=09f5c4be9ce33ddd98627188e805e836 2500w" /></Frame>

A cascaded pipeline looks like this, in Pipecat code. Here's a [single-file example that uses a cascaded pipeline](https://github.com/pipecat-ai/pipecat/blob/main/examples/foundational/07g-interruptible-openai.py). (See below for an overview of Pipecat core concepts.)

```python  theme={null}
pipeline = Pipeline(
    [
        transport.input(),
        speech_to_text,
        context_aggregator.user(),
        llm,
        text_to_speech,
        context_aggregator.assistant(),
        transport.output(),
    ]
)
```

2. Using a single, speech-to-speech model. This is conceptually much simpler. Though note that most applications also need to implement things like function calling, retrieval-augmented search, context management, and integration with existing systems. So the core pipeline is only part of an app's complexity.

<Frame><img src="https://mintcdn.com/daily/2bYrACcmgvvzC075/images/openai-s2s.jpg?fit=max&auto=format&n=2bYrACcmgvvzC075&q=85&s=a28ba89b79e0fafb91775bf572e3eac5" alt="OpenAI S2S Pipeline" data-og-width="2877" width="2877" data-og-height="868" height="868" data-path="images/openai-s2s.jpg" data-optimize="true" data-opv="3" srcset="https://mintcdn.com/daily/2bYrACcmgvvzC075/images/openai-s2s.jpg?w=280&fit=max&auto=format&n=2bYrACcmgvvzC075&q=85&s=faeabe5c34f66148011561de5e349e56 280w, https://mintcdn.com/daily/2bYrACcmgvvzC075/images/openai-s2s.jpg?w=560&fit=max&auto=format&n=2bYrACcmgvvzC075&q=85&s=353f953d2464947c780ab8b4983edaad 560w, https://mintcdn.com/daily/2bYrACcmgvvzC075/images/openai-s2s.jpg?w=840&fit=max&auto=format&n=2bYrACcmgvvzC075&q=85&s=8d482bcbe3227763c975063cbb45e3f8 840w, https://mintcdn.com/daily/2bYrACcmgvvzC075/images/openai-s2s.jpg?w=1100&fit=max&auto=format&n=2bYrACcmgvvzC075&q=85&s=5ef9f830148f10e9304ebc85701c0b77 1100w, https://mintcdn.com/daily/2bYrACcmgvvzC075/images/openai-s2s.jpg?w=1650&fit=max&auto=format&n=2bYrACcmgvvzC075&q=85&s=c0cd04809992dd8c2c5f391b5edefb09 1650w, https://mintcdn.com/daily/2bYrACcmgvvzC075/images/openai-s2s.jpg?w=2500&fit=max&auto=format&n=2bYrACcmgvvzC075&q=85&s=67d1da965ce7efa8ae60e6eb39835819 2500w" /></Frame>

Here's a speech-to-speech pipeline in Pipecat code. And here's a [single-file example that uses the OpenAI Realtime API](https://github.com/pipecat-ai/pipecat/blob/main/examples/foundational/19-openai-realtime-beta.py).

```python  theme={null}
pipeline = Pipeline(
    [
        transport.input(),
        context_aggregator.user(),
        speech_to_speech_llm,
        context_aggregator.assistant(),
        transport.output(),
    ]
)
```

Which approach should you choose?

* The cascaded models approach is preferable if you are implementing a complex workflow and need the best possible instruction following performance and function calling reliability. The `gpt-4o` model operating in text-to-text mode has the strongest instruction following and function calling performance.
* The speech-to-speech approach offers better audio understanding and human-like voice output. If your application is primarily free-form, open-ended conversation, these attributes might be more important than instruction following and function calling performance. Note also that `gpt-4o-audio-preview` and the OpenAI Realtime API are currently beta products.

## OpenAI Audio Models and APIs

### Transcription API

* Models: `gpt-4o-transcribe`, `gpt-4o-mini-transcribe`
* Pipecat service: `OpenAISTTService` ([reference docs](/server/services/stt/openai))
* OpenAI endpoint: `/v1/audio/transcriptions` ([docs](https://platform.openai.com/docs/api-reference/audio/createTranscription))

### Chat Completions API

* Models: `gpt-4o`, `gpt-4o-mini`, `gpt-4o-audio-preview`
* Pipecat service: `OpenAILLMService` ([reference docs](/server/services/llm/openai))
* OpenAI endpoint: `/v1/chat/completions` ([docs](https://platform.openai.com/docs/api-reference/chat))

### Realtime API

* Models: `gpt-4o-realtime-preview`, `gpt-4o-mini-realtime-preview`
* Pipecat service: `OpenAIRealtimeBetaLLMService` ([reference docs](/server/services/s2s/openai))
* OpenAI docs ([overview](https://platform.openai.com/docs/guides/realtime))

### Speech API

* Models: `gpt-4o-mini-tts`
* Pipecat service: `OpenAITTSService` ([reference docs](/server/services/tts/openai))
* OpenAI endpoint: `/v1/audio/speech` ([docs](https://platform.openai.com/docs/api-reference/audio/createSpeech))

## Sample code and starter kits

*If you have a code example or starter kit you would like this doc to link to, please let us know. We can add examples that help people get started with the OpenAI audio models and APIs.*

### Single-file examples

<CardGroup cols={2}>
  <Card title="OpenAI STT → LLM → TTS" icon="code" href="https://github.com/pipecat-ai/pipecat/blob/main/examples/foundational/07g-interruptible-openai.py">
    A complete implementation demonstrating the cascaded approach with OpenAI services
  </Card>

  <Card title="OpenAI Realtime API" icon="tower-broadcast" href="https://github.com/pipecat-ai/pipecat/blob/main/examples/foundational/19-openai-realtime-beta.py">
    A speech-to-speech implementation using OpenAI's Realtime API
  </Card>
</CardGroup>

### OpenAI + Twilio + Pipecat Cloud

[This starter kit](https://github.com/daily-co/pcc-openai-twilio/) is a complete telephone voice agent that can talk about the NCAA March Madness basketball tournaments and look up realtime game information using function calls.

<Frame>
    <img src="https://mintcdn.com/daily/2bYrACcmgvvzC075/images/openai-twilio.png?fit=max&auto=format&n=2bYrACcmgvvzC075&q=85&s=fcde44430a2b87c8467af8de0871fb24" alt="OpenAI Twilio" data-og-width="1598" width="1598" data-og-height="1284" height="1284" data-path="images/openai-twilio.png" data-optimize="true" data-opv="3" srcset="https://mintcdn.com/daily/2bYrACcmgvvzC075/images/openai-twilio.png?w=280&fit=max&auto=format&n=2bYrACcmgvvzC075&q=85&s=05f08079f15eb0c0b6e15dda86c7c65b 280w, https://mintcdn.com/daily/2bYrACcmgvvzC075/images/openai-twilio.png?w=560&fit=max&auto=format&n=2bYrACcmgvvzC075&q=85&s=94bcd70b14e4f5d361bff1842a99f578 560w, https://mintcdn.com/daily/2bYrACcmgvvzC075/images/openai-twilio.png?w=840&fit=max&auto=format&n=2bYrACcmgvvzC075&q=85&s=3fa9a58977aeb866e43001346832f4c5 840w, https://mintcdn.com/daily/2bYrACcmgvvzC075/images/openai-twilio.png?w=1100&fit=max&auto=format&n=2bYrACcmgvvzC075&q=85&s=af0fdc815aeca2417d96364d00a17092 1100w, https://mintcdn.com/daily/2bYrACcmgvvzC075/images/openai-twilio.png?w=1650&fit=max&auto=format&n=2bYrACcmgvvzC075&q=85&s=af4f08d50588f303d81d83bf001aeaa3 1650w, https://mintcdn.com/daily/2bYrACcmgvvzC075/images/openai-twilio.png?w=2500&fit=max&auto=format&n=2bYrACcmgvvzC075&q=85&s=57142b67b5e8eb1338a4e8ff4bceb52c 2500w" />
</Frame>

The starter kit includes two bot configurations: cascaded model and speech-to-speech. The code can be packaged for deployment to Pipecat Cloud, a commercial platform for Pipecat agent hosting.

---

# Source: https://docs.pipecat.ai/client/js/transports/openai-webrtc.md

# Source: https://docs.pipecat.ai/client/ios/transports/openai-webrtc.md

# Source: https://docs.pipecat.ai/client/android/transports/openai-webrtc.md

> ## Documentation Index
> Fetch the complete documentation index at: https://docs.pipecat.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# OpenAI Realtime WebRTC Transport

> WebRTC implementation for Android using OpenAI

The OpenAI Realtime WebRTC transport implementation enables real-time audio communication with the OpenAI Realtime service, using a direct WebRTC connection.

## Installation

Add the transport dependency to your `build.gradle`:

```gradle  theme={null}
implementation "ai.pipecat:openai-realtime-webrtc-transport:0.3.7"
```

## Usage

Create a client:

```kotlin  theme={null}
val transport = OpenAIRealtimeWebRTCTransport.Factory(context)

val options = RTVIClientOptions(
    params = RTVIClientParams(
        baseUrl = null,
        config = OpenAIRealtimeWebRTCTransport.buildConfig(
            apiKey = apiKey,
            initialMessages = listOf(
                LLMContextMessage(role = "user", content = "How tall is the Eiffel Tower?")
            ),
            initialConfig = OpenAIRealtimeSessionConfig(
                voice = "ballad",
                turnDetection = Value.Object("type" to Value.Str("semantic_vad")),
                inputAudioNoiseReduction = Value.Object("type" to Value.Str("near_field")),
                inputAudioTranscription = Value.Object("model" to Value.Str("gpt-4o-transcribe"))
            )
        )
    )
)

val client = RTVIClient(transport, callbacks, options)

client.start().withCallback {
    // ...
}
```

## Resources

<CardGroup cols={2}>
  <Card horizontal title="Demo" icon="play" href="https://github.com/pipecat-ai/pipecat-client-android-openai-realtime-webrtc-demo">
    Simple Chatbot Demo
  </Card>

  <Card horizontal title="Source" icon="github" href="https://github.com/pipecat-ai/pipecat-client-android-transports/">
    Client Transports
  </Card>
</CardGroup>

<Card title="Pipecat Android Client Reference" icon="book-open" href="https://docs-android.rtvi.ai/">
  Complete API documentation for the Pipecat Android client.
</Card>

---

# Source: https://docs.pipecat.ai/server/services/tts/openai.md

# Source: https://docs.pipecat.ai/server/services/stt/openai.md

# Source: https://docs.pipecat.ai/server/services/s2s/openai.md

# Source: https://docs.pipecat.ai/server/services/llm/openai.md

# Source: https://docs.pipecat.ai/server/services/image-generation/openai.md

> ## Documentation Index
> Fetch the complete documentation index at: https://docs.pipecat.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# OpenAI Image Generation

> Image generation service implementation using OpenAI's DALL-E models

## Overview

`OpenAIImageGenService` provides high-quality image generation capabilities using OpenAI's DALL-E models. It transforms text prompts into images with various size options and model configurations, offering both artistic and photorealistic image creation capabilities.

<CardGroup cols={3}>
  <Card title="OpenAI Image Gen API Reference" icon="code" href="https://reference-server.pipecat.ai/en/latest/api/pipecat.services.openai.image.html">
    Pipecat's API methods for OpenAI image generation integration
  </Card>

  <Card title="OpenAI Documentation" icon="book" href="https://platform.openai.com/docs/guides/images">
    Official OpenAI DALL-E API documentation and guides
  </Card>

  <Card title="OpenAI Platform" icon="image" href="https://platform.openai.com/">
    Access DALL-E models and manage API keys
  </Card>
</CardGroup>

## Installation

To use OpenAI image generation services, install the required dependencies:

```bash  theme={null}
pip install "pipecat-ai[openai]"
```

## Prerequisites

### OpenAI Account Setup

Before using OpenAI image generation services, you need:

1. **OpenAI Account**: Sign up at [OpenAI Platform](https://platform.openai.com/)
2. **API Key**: Generate an OpenAI API key from your account dashboard
3. **Model Access**: Ensure access to DALL-E models
4. **HTTP Session**: Configure aiohttp session for image downloading

### Required Environment Variables

* `OPENAI_API_KEY`: Your OpenAI API key for authentication

---

# Source: https://docs.pipecat.ai/server/services/llm/openpipe.md

> ## Documentation Index
> Fetch the complete documentation index at: https://docs.pipecat.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# OpenPipe

> LLM service implementation using OpenPipe for LLM request logging and fine-tuning

## Overview

`OpenPipeLLMService` extends the BaseOpenAILLMService to provide integration with OpenPipe, enabling request logging, model fine-tuning, and performance monitoring. It maintains compatibility with OpenAI's API while adding OpenPipe's logging and optimization capabilities.

<CardGroup cols={2}>
  <Card title="OpenPipe LLM API Reference" icon="code" href="https://reference-server.pipecat.ai/en/latest/api/pipecat.services.openpipe.llm.html">
    Pipecat's API methods for OpenPipe integration
  </Card>

  <Card title="Example Implementation" icon="play" href="https://github.com/pipecat-ai/pipecat/tree/main/examples">
    Browse examples using OpenPipe logging
  </Card>

  <Card title="OpenPipe Documentation" icon="book" href="https://docs.openpipe.ai/api-reference/post-chatcompletions">
    Official OpenPipe API documentation and features
  </Card>

  <Card title="OpenPipe Platform" icon="microphone" href="https://openpipe.ai/">
    Access logging and fine-tuning features
  </Card>
</CardGroup>

## Installation

To use OpenPipe services, install the required dependencies:

```bash  theme={null}
pip install "pipecat-ai[openpipe]"
```

## Prerequisites

### OpenPipe Account Setup

Before using OpenPipe LLM services, you need:

1. **OpenPipe Account**: Sign up at [OpenPipe](https://openpipe.ai/)
2. **API Keys**: Generate both OpenPipe and OpenAI API keys
3. **Project Setup**: Configure logging and fine-tuning projects

### Required Environment Variables

* `OPENPIPE_API_KEY`: Your OpenPipe API key for logging and fine-tuning
* `OPENAI_API_KEY`: Your OpenAI API key for underlying model access

---

# Source: https://docs.pipecat.ai/server/services/llm/openrouter.md

> ## Documentation Index
> Fetch the complete documentation index at: https://docs.pipecat.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# OpenRouter

> LLM service implementation using OpenRouter's API with OpenAI-compatible interface

## Overview

`OpenRouterLLMService` provides access to OpenRouter's language models through an OpenAI-compatible interface. It inherits from `OpenAILLMService` and supports streaming responses, function calling, and context management with access to multiple model providers through a single API.

<CardGroup cols={2}>
  <Card title="OpenRouter LLM API Reference" icon="code" href="https://reference-server.pipecat.ai/en/latest/api/pipecat.services.openrouter.llm.html">
    Pipecat's API methods for OpenRouter integration
  </Card>

  <Card title="Example Implementation" icon="play" href="https://github.com/pipecat-ai/pipecat/blob/main/examples/foundational/14m-function-calling-openrouter.py">
    Complete example with function calling
  </Card>

  <Card title="OpenRouter Documentation" icon="book" href="https://openrouter.ai/docs/api-reference/chat-completion">
    Official OpenRouter API documentation and features
  </Card>

  <Card title="OpenRouter Platform" icon="microphone" href="https://openrouter.ai/">
    Access multiple model providers and manage API keys
  </Card>
</CardGroup>

## Installation

To use OpenRouter services, install the required dependencies:

```bash  theme={null}
pip install "pipecat-ai[openrouter]"
```

## Prerequisites

### OpenRouter Account Setup

Before using OpenRouter LLM services, you need:

1. **OpenRouter Account**: Sign up at [OpenRouter](https://openrouter.ai/)
2. **API Key**: Generate an API key from your account dashboard
3. **Model Selection**: Choose from hundreds of available models from different providers
4. **Credits**: Add credits to your account for model usage

### Required Environment Variables

* `OPENROUTER_API_KEY`: Your OpenRouter API key for authentication

---

# Source: https://docs.pipecat.ai/server/utilities/opentelemetry.md

> ## Documentation Index
> Fetch the complete documentation index at: https://docs.pipecat.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# OpenTelemetry Tracing

> Monitor and analyze your Pipecat conversational pipelines using OpenTelemetry

## Overview

Pipecat includes built-in support for OpenTelemetry tracing, allowing you to gain deep visibility into your voice applications. Tracing helps you:

* Track latency and performance across your conversation pipeline
* Monitor service health and identify bottlenecks
* Visualize conversation turns and service dependencies
* Collect usage metrics and operational analytics

## Installation

To use OpenTelemetry tracing with Pipecat, install the tracing dependencies:

```bash  theme={null}
pip install "pipecat-ai[tracing]"
```

<Tip>
  For local development and testing, we recommend using Jaeger as a trace collector.
  You can run it with Docker:

  ```bash  theme={null}
  docker run -d --name jaeger \
    -p 16686:16686 \
    -p 4317:4317 \
    -p 4318:4318 \
    jaegertracing/all-in-one:latest
  ```

  Then access the UI at [http://localhost:16686](http://localhost:16686)
</Tip>

## Basic Setup

Enabling tracing in your Pipecat application requires two steps:

1. **Initialize the OpenTelemetry SDK** with your preferred exporter
2. **Enable tracing in your PipelineTask**

```python  theme={null}
import os
from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter
from pipecat.utils.tracing.setup import setup_tracing
from pipecat.pipeline.task import PipelineTask, PipelineParams

# Step 1: Initialize OpenTelemetry with your chosen exporter
exporter = OTLPSpanExporter(
    endpoint="http://localhost:4317",  # Jaeger or other collector endpoint
    insecure=True,
)

setup_tracing(
    service_name="my-voice-app",
    exporter=exporter,
    console_export=False,  # Set to True for debug output
)

# Step 2: Enable tracing in your PipelineTask
task = PipelineTask(
    pipeline,
    params=PipelineParams(
        enable_metrics=True,                              # Required for some service metrics
    ),
    enable_tracing=True,                                  # Enable tracing for this task
    enable_turn_tracking=True,                            # Enable turn tracking for this task
    conversation_id="customer-123",                       # Optional - will auto-generate if not provided
    additional_span_attributes={"session.id": "abc-123"} # Optional - additional attributes to attach to the otel span
)
```

<Info>
  For complete working examples, see our sample implementations:

  * [Jaeger Tracing Example](https://github.com/pipecat-ai/pipecat-examples/tree/main/open-telemetry/jaeger) - Uses gRPC exporter with Jaeger
  * [Langfuse Tracing Example](https://github.com/pipecat-ai/pipecat-examples/tree/main/open-telemetry/langfuse) - Uses HTTP exporter with Langfuse for LLM-focused observability
</Info>

## Trace Structure

Pipecat organizes traces hierarchically, following the natural structure of conversations:

```
Conversation (conversation)
├── turn
│   ├── stt
│   ├── llm
│   └── tts
└── turn
    ├── stt
    ├── llm
    └── tts
    turn...
```

For real-time multimodal services like Gemini Live and OpenAI Realtime, the structure adapts to their specific patterns:

```
Conversation (conversation)
├── turn
│   ├── llm_setup (session configuration)
│   ├── stt (user input)
│   ├── llm_response (complete response with usage)
│   └── llm_tool_call/llm_tool_result (for function calls)
└── turn
    ├── stt (user input)
    └── llm_response (complete response)
    turn...
```

This hierarchical structure makes it easy to:

* Track the full lifecycle of a conversation
* Measure latency for individual turns
* Identify which services are contributing to delays
* Compare performance across different conversations

## Exporter Options

Pipecat supports any OpenTelemetry-compatible exporter. Common options include:

### OTLP Exporter (for Jaeger, Grafana, etc.)

```python  theme={null}
from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter

exporter = OTLPSpanExporter(
    endpoint="http://localhost:4317",  # Your collector endpoint
    insecure=True,  # Use False for TLS connections
)
```

### HTTP OTLP Exporter (for Langfuse, etc.)

```python  theme={null}
from opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporter

exporter = OTLPSpanExporter(
    # Configure with environment variables:
    # OTEL_EXPORTER_OTLP_ENDPOINT
    # OTEL_EXPORTER_OTLP_HEADERS
)
```

See our [Langfuse example](https://github.com/pipecat-ai/pipecat-examples/tree/main/open-telemetry/langfuse) for details on configuring this exporter.

### Console Exporter (for debugging)

The console exporter can be enabled alongside any other exporter by setting `console_export=True`:

```python  theme={null}
setup_tracing(
    service_name="my-voice-app",
    exporter=otlp_exporter,
    console_export=True,  # Prints traces to stdout
)
```

### Cloud Provider Exporters

Many cloud providers offer OpenTelemetry-compatible observability services:

* **AWS X-Ray**
* **Google Cloud Trace**
* **Azure Monitor**
* **Datadog APM**

Check the OpenTelemetry documentation for specific exporter configurations: [OpenTelemetry Vendors](https://opentelemetry.io/ecosystem/vendors/)

### OpenInference

Arize-ai provides OpenInference instrumentation, compatible with OpenTelemetry.

* [openinference-instrumentation-pipecat](https://github.com/Arize-ai/openinference/tree/main/python/instrumentation/openinference-instrumentation-pipecat)

See [Observability community integrations](https://docs.pipecat.ai/server/services/community-integrations#observability) for more details.

## Span Attributes

Pipecat enriches spans with detailed attributes about service operations:

### TTS Service Spans

* `gen_ai.system`: Service provider (e.g., "cartesia")
* `gen_ai.request.model`: Model ID/name
* `voice_id`: Voice identifier
* `text`: The text being synthesized
* `metrics.character_count`: Number of characters in the text
* `metrics.ttfb`: Time to first byte in seconds
* `settings.*`: Service-specific configuration parameters

### STT Service Spans

* `gen_ai.system`: Service provider (e.g., "deepgram")
* `gen_ai.request.model`: Model ID/name
* `transcript`: The transcribed text
* `is_final`: Whether the transcription is final
* `language`: Detected or configured language
* `vad_enabled`: Whether voice activity detection is enabled
* `metrics.ttfb`: Time to first byte in seconds
* `settings.*`: Service-specific configuration parameters

### LLM Service Spans

* `gen_ai.system`: Service provider (e.g., "openai", "gcp.gemini")
* `gen_ai.request.model`: Model ID/name
* `gen_ai.operation.name`: Operation type (e.g., "chat")
* `stream`: Whether streaming is enabled
* `input`: JSON-serialized input messages
* `output`: Complete response text
* `tools`: JSON-serialized tools configuration
* `tools.count`: Number of tools available
* `tools.names`: Comma-separated tool names
* `system`: System message content
* `gen_ai.usage.input_tokens`: Number of prompt tokens
* `gen_ai.usage.output_tokens`: Number of completion tokens
* `metrics.ttfb`: Time to first byte in seconds
* `gen_ai.request.*`: Standard parameters (temperature, max\_tokens, etc.)

### Multimodal Service Spans (Gemini Live & OpenAI Realtime)

#### Setup Spans

* `gen_ai.system`: "gcp.gemini" or "openai"
* `gen_ai.request.model`: Model identifier
* `tools.count`: Number of available tools
* `tools.definitions`: JSON-serialized tool schemas
* `system_instruction`: System prompt (truncated)
* `session.*`: Session configuration parameters

#### Request Spans (OpenAI Realtime)

* `input`: JSON-serialized context messages being sent
* `gen_ai.operation.name`: "llm\_request"

#### Response Spans

* `output`: Complete assistant response text
* `output_modality`: "TEXT" or "AUDIO" (Gemini Live)
* `gen_ai.usage.input_tokens`: Prompt tokens used
* `gen_ai.usage.output_tokens`: Completion tokens generated
* `function_calls.count`: Number of function calls made
* `function_calls.names`: Comma-separated function names
* `metrics.ttfb`: Time to first response in seconds

#### Tool Call/Result Spans (Gemini Live)

* `tool.function_name`: Name of the function being called
* `tool.call_id`: Unique identifier for the call
* `tool.arguments`: Function arguments (truncated)
* `tool.result`: Function execution result (truncated)
* `tool.result_status`: "completed", "error", or "parse\_error"

### Turn Spans

* `turn.number`: Sequential turn number
* `turn.type`: Type of turn (e.g., "conversation")
* `turn.duration_seconds`: Duration of the turn
* `turn.was_interrupted`: Whether the turn was interrupted
* `conversation.id`: ID of the parent conversation

### Conversation Spans

* `conversation.id`: Unique identifier for the conversation
* `conversation.type`: Type of conversation (e.g., "voice")

## Usage Metrics

Pipecat's tracing implementation automatically captures usage metrics for LLM and TTS services:

### LLM Token Usage

Token usage is captured in LLM spans as:

* `gen_ai.usage.input_tokens`
* `gen_ai.usage.output_tokens`

### TTS Character Count

Character count is captured in TTS spans as:

* `metrics.character_count`

## Performance Metrics

Pipecat traces capture key performance metrics for each service:

### Time To First Byte (TTFB)

The time it takes for a service to produce its first response:

* `metrics.ttfb` (in seconds)

### Processing Duration

The total time spent processing in each service is captured in the span duration.

## Configuration Options

### PipelineTask Parameters

<ParamField path="enable_tracing" type="bool" default="True">
  Enable or disable tracing for the pipeline
</ParamField>

<ParamField path="enable_turn_tracking" type="bool" default="False">
  Whether to enable turn tracking.
</ParamField>

<ParamField path="conversation_id" type="Optional[str]" default="None">
  Custom ID for the conversation. If not provided, a UUID will be generated
</ParamField>

<ParamField path="additional_span_attributes" type="Optional[dict]" default="None">
  Any additional attributes to add to top-level OpenTelemetry conversation span.
</ParamField>

### setup\_tracing() Parameters

<ParamField path="service_name" type="str" default="pipecat">
  Name of the service for traces
</ParamField>

<ParamField path="exporter" type="Optional[SpanExporter]" default="None">
  A pre-configured OpenTelemetry span exporter instance
</ParamField>

<ParamField path="console_export" type="bool" default="False">
  Whether to also export traces to console (useful for debugging)
</ParamField>

## Example

Here's a complete example showing OpenTelemetry tracing setup with Jaeger:

```python  theme={null}
import os
from dotenv import load_dotenv
from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter

from pipecat.pipeline.pipeline import Pipeline
from pipecat.pipeline.task import PipelineParams, PipelineTask
from pipecat.services.deepgram.stt import DeepgramSTTService
from pipecat.services.openai.llm import OpenAILLMService
from pipecat.services.cartesia.tts import CartesiaTTSService
from pipecat.utils.tracing.setup import setup_tracing

load_dotenv()

# Initialize tracing if enabled
if os.getenv("ENABLE_TRACING"):
    # Create the exporter
    otlp_exporter = OTLPSpanExporter(
        endpoint=os.getenv("OTEL_EXPORTER_OTLP_ENDPOINT", "http://localhost:4317"),
        insecure=True,
    )

    # Set up tracing with the exporter
    setup_tracing(
        service_name="pipecat-demo",
        exporter=otlp_exporter,
        console_export=bool(os.getenv("OTEL_CONSOLE_EXPORT")),
    )

# Create your services
stt = DeepgramSTTService(api_key=os.getenv("DEEPGRAM_API_KEY"))
llm = OpenAILLMService(api_key=os.getenv("OPENAI_API_KEY"))
tts = CartesiaTTSService(
    api_key=os.getenv("CARTESIA_API_KEY"),
    voice_id="your-voice-id"
)

# Build pipeline
pipeline = Pipeline([
    transport.input(),
    stt,
    context_aggregator.user(),
    llm,
    tts,
    transport.output(),
    context_aggregator.assistant(),
])

# Create pipeline task with tracing enabled
task = PipelineTask(
    pipeline,
    params=PipelineParams(
        enable_metrics=True,
        enable_usage_metrics=True,
    ),
    enable_tracing=True,
    enable_turn_tracking=True,
    conversation_id="customer-123",                       # Optional - will auto-generate if not provided
    additional_span_attributes={"session.id": "abc-123"} # Optional - additional attributes to attach to the otel span
)

# Run the pipeline
runner = PipelineRunner()
await runner.run(task)
```

## Troubleshooting

If you're having issues with tracing:

* **No Traces Visible**: Ensure the OpenTelemetry packages are installed and that your collector endpoint is correct
* **Missing Service Data**: Verify that `enable_metrics=True` is set in PipelineParams
* **Debugging Tracing**: Enable console export with `console_export=True` to view traces in your logs
* **Connection Errors**: Check network connectivity to your trace collector
* **Collector Configuration**: Verify your collector is properly set up to receive traces

## References

* [OpenTelemetry Python Documentation](https://opentelemetry-python.readthedocs.io/)
* [OpenTelemetry Tracing Specification](https://opentelemetry.io/docs/reference/specification/trace/api/)
* [Jaeger Documentation](https://www.jaegertracing.io/docs/latest/)
* [Langfuse OpenTelemetry Documentation](https://langfuse.com/docs/opentelemetry/get-started)

---

# Source: https://docs.pipecat.ai/cli/cloud/organizations.md

> ## Documentation Index
> Fetch the complete documentation index at: https://docs.pipecat.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# organizations

> Organization and API key management commands

The `organizations` command group helps you manage your Pipecat Cloud organizations and API keys. You can list and select organizations, as well as create, list, and manage API keys for use with the platform.

<Note>
  Organization and user management is not available via the CLI. Please use the
  [Pipecat Cloud Dashboard](https://pipecat.daily.co) to manage organizations
  and users.
</Note>

## list

Lists all organizations that your account has access to, highlighting the currently active one used for CLI operations.

**Usage:**

```shell  theme={null}
pipecat cloud organizations list
```

## select

Changes your active organization for CLI operations.

This command either presents an interactive selection menu or directly sets a specified organization as your default. The selection is stored in your local configuration file (defaults to `~/.config/pipecatcloud/pipecatcloud.toml`) and used for all subsequent CLI commands.

**Usage:**

```shell  theme={null}
pipecat cloud organizations select [OPTIONS]
```

**Options:**

<ParamField path="--organization / -o" type="string">
  Bypass prompt by directly specifying a namespace / organization string.
</ParamField>

## keys

The `keys` sub-commands manage API keys for authenticating with Pipecat Cloud services.

### keys list

Lists all API keys for the current organization.

**Usage:**

```shell  theme={null}
pipecat cloud organizations keys list [OPTIONS]
```

**Options:**

<ParamField path="--organization / -o" type="string">
  Organization ID to list keys for. If not provided, the default organization
  will be used.
</ParamField>

### keys create

Create a new public API key for account / organization. Command will prompt the user to enter
a human-readable name for the key.

**Usage:**

```shell  theme={null}
pipecat cloud organizations keys create [OPTIONS]
```

**Options:**

<ParamField path="--organization / -o" type="string">
  Organization ID to create key for. If not provided, the default organization
  will be used.
</ParamField>

### keys delete

Delete an API key from your organization. Command will prompt the user to select which key they wish to delete.

**Usage:**

```shell  theme={null}
pipecat cloud organizations keys delete [OPTIONS]
```

**Options:**

<ParamField path="--organization / -o" type="string">
  Organization ID to delete key for. If not provided, the default organization
  will be used.
</ParamField>

### keys use

Sets a specific API key as your default for CLI operations.

The selected key is stored in the local configuration file (defaults to `~/.config/pipecatcloud/pipecatcloud.toml`) and will be used for all subsequent requests to the Pipecat Cloud API.

Please note that the key must be associated with the same user account or organization as you are making requests to.

If the public key is revoked or deleted via the dashboard, the user will need to re-run this command to select a new key.

**Usage:**

```shell  theme={null}
pipecat cloud organizations keys use [OPTIONS]
```

**Options:**

<ParamField path="--organization / -o" type="string">
  Organization ID to select default key from. If not provided, the default
  organization will be used.
</ParamField>

## properties

The `properties` sub-commands manage organization properties such as default region settings.

### properties list

Lists all current property values for your organization.

**Usage:**

```shell  theme={null}
pipecat cloud organizations properties list [OPTIONS]
```

**Options:**

<ParamField path="--organization / -o" type="string">
  Organization ID to list properties for. If not provided, the default
  organization will be used.
</ParamField>

### properties schema

Shows available properties with detailed metadata including type information, current values, default values, and available values.

**Usage:**

```shell  theme={null}
pipecat cloud organizations properties schema [OPTIONS]
```

**Options:**

<ParamField path="--organization / -o" type="string">
  Organization ID to show properties schema for. If not provided, the default
  organization will be used.
</ParamField>

### properties set

Updates a specific organization property.

**Usage:**

```shell  theme={null}
pipecat cloud organizations properties set PROPERTY_NAME VALUE [OPTIONS]
```

**Arguments:**

<ParamField path="PROPERTY_NAME" type="string" required>
  Name of the property to set (e.g., `defaultRegion`)
</ParamField>

<ParamField path="VALUE" type="string" required>
  Value to set for the property
</ParamField>

**Options:**

<ParamField path="--organization / -o" type="string">
  Organization ID to update property for. If not provided, the default
  organization will be used.
</ParamField>

**Example:**

```shell  theme={null}
pipecat cloud organizations properties set defaultRegion eu-central
```

## default-region

Convenience command to get or set the default region for your organization.

**Usage:**

```shell  theme={null}
pipecat cloud organizations default-region [REGION] [OPTIONS]
```

**Arguments:**

<ParamField path="REGION" type="string">
  Region to set as default. If omitted, displays the current default region and
  available regions.
</ParamField>

**Options:**

<ParamField path="--organization / -o" type="string">
  Organization ID to configure. If not provided, the default organization will
  be used.
</ParamField>

**Examples:**

View current default region:

```shell  theme={null}
pipecat cloud organizations default-region
```

Set default region:

```shell  theme={null}
pipecat cloud organizations default-region eu-central
```

***

<Card title="More about organizations" icon="people-group" href="/deployment/pipecat-cloud/fundamentals/accounts-and-organizations">
  Managing your account and collaborating on agents as part of a team
</Card>

---

# Source: https://docs.pipecat.ai/server/utilities/text/overview.md

# Source: https://docs.pipecat.ai/guides/telephony/overview.md

# Source: https://docs.pipecat.ai/guides/learn/overview.md

# Source: https://docs.pipecat.ai/deployment/pipecat-cloud/sdk-reference/overview.md

# Source: https://docs.pipecat.ai/deployment/pipecat-cloud/guides/container-registries/overview.md

# Source: https://docs.pipecat.ai/deployment/overview.md

# Source: https://docs.pipecat.ai/cli/overview.md

> ## Documentation Index
> Fetch the complete documentation index at: https://docs.pipecat.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# CLI Overview

> Command-line tool for scaffolding, deploying, and monitoring Pipecat bots

<CardGroup cols={3}>
  <Card title="Scaffold Projects" icon="wand-magic-sparkles" href="/cli/init">
    Create new phone or web/mobile bots with interactive setup
  </Card>

  <Card title="Deploy to Cloud" icon="rocket" href="/cli/cloud/auth">
    Push your bots to production with one command
  </Card>

  <Card title="Monitor Live Bots" icon="chart-line" href="/cli/tail">
    Watch real-time logs, conversations, and metrics
  </Card>
</CardGroup>

## Requirements

* Python 3.10 or later

## Installation

Install the CLI globally using [uv](https://docs.astral.sh/uv/) (recommended) or [pipx](https://pipx.pypa.io/):

```bash  theme={null}
uv tool install pipecat-ai-cli
# or
pipx install pipecat-ai-cli
```

Verify installation:

```bash  theme={null}
pipecat --version
```

<Tip>All commands can use either `pipecat` or the shorter `pc` alias.</Tip>

## Commands

**[`pipecat init`](/cli/init)** - Scaffold new projects with interactive setup

**[`pipecat tail`](/cli/tail)** - Monitor sessions in real-time with a terminal dashboard

**[`pipecat cloud`](/cli/cloud/auth)** - Deploy and manage bots on Pipecat Cloud

## Getting Help

View help for any command:

```bash  theme={null}
pipecat --help
pipecat init --help
pipecat tail --help
pipecat cloud --help
```

## Next Steps

<Card title="Create Your First Bot" icon="rocket" href="/cli/init">
  Scaffold a new project with pipecat init
</Card>

---

# Source: https://docs.pipecat.ai/server/pipeline/parallel-pipeline.md

> ## Documentation Index
> Fetch the complete documentation index at: https://docs.pipecat.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# ParallelPipeline

> Run multiple pipeline branches in parallel, with synchronized inputs and outputs for complex flows

## Overview

`ParallelPipeline` allows you to create multiple independent processing branches that run simultaneously, sharing input and coordinating output. It's particularly useful for multi-agent systems, parallel stream processing, and creating redundant service paths.

Each branch receives the same downstream frames, processes them independently, and the results are merged back into a single stream. System frames (like `StartFrame` and `EndFrame`) are synchronized across all branches.

## Constructor Parameters

<ParamField path="*args" type="List[List[FrameProcessor]]" required>
  Multiple lists of processors, where each list defines a parallel branch. All
  branches execute simultaneously when frames flow through the pipeline.
</ParamField>

## Usage Examples

### Multi-Agent Conversation

Create a conversation with two AI agents that can interact with the user independently:

```python  theme={null}
pipeline = Pipeline([
    transport.input(),
    ParallelPipeline(
        # Agent 1: Customer service representative
        [
            stt_1,
            context_aggregator.user_a(),
            llm_agent_1,
            tts_agent_1,
        ],
        # Agent 2: Technical specialist
        [   stt_2,
            context_aggregator.user_b(),
            llm_agent_2,
            tts_agent_2,
        ]
    ),
    transport.output(),
])
```

### Redundant Services with Failover

Set up redundant services with automatic failover:

```python  theme={null}
pipeline = Pipeline([
    transport.input(),
    stt,
    ParallelPipeline(
        # Primary LLM service
        [   gate_primary,
            primary_llm,
            error_detector,
        ],
        # Backup LLM service (used only if primary fails)
        [   gate_backup,
            backup_llm,
            fallback_processor,
        ]
    ),
    tts,
    transport.output(),
])
```

### Cross-Branch Communication

Using Producer/Consumer processors to share data between branches:

```python  theme={null}
# Create producer/consumer pair for cross-branch communication
frame_producer = ProducerProcessor(filter=is_important_frame)
frame_consumer = ConsumerProcessor(producer=frame_producer)

pipeline = Pipeline([
    transport.input(),
    ParallelPipeline(
        # Branch that generates important frames
        [
            stt,
            llm,
            tts,
            frame_producer,  # Share frames with other branch
        ],
        # Branch that consumes those frames
        [
            frame_consumer,  # Receive frames from other branch
            llm,             # Speech to Speech LLM (audio in)
        ]
    ),
    transport.output(),
])
```

## How It Works

1. `ParallelPipeline` adds special source and sink processors to each branch
2. System frames (like `StartFrame` and `EndFrame`) are sent to all branches
3. Other frames flow downstream to all branch sources
4. Results from each branch are collected at the sinks
5. The pipeline ensures `EndFrame`s are only passed through after all branches complete

---

# Source: https://docs.pipecat.ai/server/utilities/text/pattern-pair-aggregator.md

> ## Documentation Index
> Fetch the complete documentation index at: https://docs.pipecat.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# PatternPairAggregator

> Text aggregator that identifies and processes content between pattern pairs in streaming text

## Overview

`PatternPairAggregator` is a specialized text aggregator that buffers streaming text until it can identify complete pattern pairs (like XML tags, markdown formatting, or custom delimiters). It processes the content between these patterns using a set of pre-defined actions (remove, keep, or aggregate) and returns text outside those patterns at sentence boundaries. The aggregator supports registering callback functions that are invoked when specific pattern pairs are matched, allowing for custom processing when matches occur. Note: These callbacks do not support modifying the text being aggregated; they are intended for side effects like logging or updating state.

This aggregator is particularly useful for applications like voice switching, structured content processing, and extracting metadata from LLM outputs, ensuring that patterns spanning multiple text chunks are correctly identified or categorizing text based on embedded markers for downstream services and processing to treat different segments appropriately. For example: identifying URL patterns, code blocks, or special formatting in LLM responses that may need special speech handling in the TTS or client-side handling via RTVI.

<Tip>
  Want to see it in action? Check out the [voice switching
  demo](https://github.com/pipecat-ai/pipecat/blob/main/examples/foundational/35-pattern-pair-voice-switching.py) or the [bot output demo](https://github.com/pipecat-ai/pipecat-examples/blob/main/code-helper).
</Tip>

## Constructor

```python  theme={null}
pattern_aggregator = PatternPairAggregator()
```

No parameters are required for initialization. The aggregator starts with an empty buffer and no registered patterns.

## Methods

### add\_pattern

```python  theme={null}
pattern_aggregator.add_pattern(
  type,
  start_pattern,
  end_pattern,
  action
)
```

Registers a new pattern pair to detect in the text.

<ParamField path="type" type="str" required>
  Unique identifier for this pattern pair that should also represent what the text between the tags represents (e.g., "voice", "xml", "credit\_card", etc.). This value will be returned as part of both PatternMatch provided to callbacks and the Aggregation object returned from `aggregate()`.

  <Warning>
    This type may not be set to "sentence" or "word" as those are reserved for standard aggregations.
  </Warning>

  <Tip>
    Choose descriptive and unique type names to avoid confusion when handling multiple patterns.
    This type will also be referenced in the TTS for optionally skipping these types or providing custom text transformations before speaking these types. It's also optionally referenced in RTVI for similar purposes of not sending certain types or transforming them before sending to the client.
  </Tip>
</ParamField>

<ParamField path="start_pattern" type="str" required>
  Pattern that marks the beginning of content
</ParamField>

<ParamField path="end_pattern" type="str" required>
  Pattern that marks the end of content
</ParamField>

<ParamField path="action" type="MatchAction" default="MatchAction.REMOVE">
  What to do with the matched pattern and its content:

  * `MatchAction.REMOVE`: The text along with its delimiters will be removed from the streaming text. Sentence aggregation will continue on as if this text did not exist.
  * `MatchAction.KEEP`: The delimiters will be removed, but the content between them will be kept. Sentence aggregation will continue on with the internal text included. This is helpful if you want to keep the content but be notified when it occurs via a callback.
  * `MatchAction.AGGREGATE`: Aggregate the matched pattern and its content as a separate aggregation. The matched content will be returned in an `Aggregation` object with the specified type when the pattern is completed. When the start of this pattern is detected, any buffered text up to that point will be returned as a standard "sentence" aggregation.
</ParamField>

<ResponseField name="Returns">Self for method chaining</ResponseField>

### add\_pattern\_pair

<Warning>
  This method is deprecated. Use `add_pattern` instead.
</Warning>

```python  theme={null}
pattern_aggregator.add_pattern_pair(
  pattern_id,
  start_pattern,
  end_pattern,
  remove_match=True
)
```

### on\_pattern\_match

```python  theme={null}
pattern_aggregator.on_pattern_match(type, handler)
```

Registers a handler function to be called when a specific pattern pair is matched.

<ParamField path="type" type="str" required>
  The pattern pair type to listen for (as defined in `add_pattern`)
</ParamField>

<ParamField path="handler" type="Callable[[PatternMatch], Awaitable[None]]" required>
  Function to call when the pattern is matched. The function should accept a
  PatternMatch object.
</ParamField>

<ResponseField name="Returns">Self for method chaining</ResponseField>

## Pattern Match Object

When a pattern is matched, the handler function receives a `PatternMatch` object which is a subclass of the `Aggregation` object. It contains the following fields:

<ResponseField name="type" type="str">
  The identifier and descriptor of the matched pattern pair. This field is part of the `Aggregation` base class.
</ResponseField>

<ResponseField name="text" type="str">
  The text content between the start and end patterns. This field is part of the `Aggregation` base class.
</ResponseField>

<ResponseField name="full_match" type="str">
  The complete text including start and end patterns.
</ResponseField>

## Usage Examples

### Voice Switching in TTS

This example demonstrates finding custom `<voice>` tags in streaming text to switch voices dynamically in a TTS service like Cartesia. It removes the tags and the content between them, such that the content is treated as if it does not exist. It will not be spoken by the TTS, it will not be added to the context, and it will not be sent to clients via RTVI. Instead, it simply triggers a voice switch side effect.

```python  theme={null}
# Define voice IDs
VOICE_IDS = {
    "narrator": "c45bc5ec-dc68-4feb-8829-6e6b2748095d",
    "female": "71a7ad14-091c-4e8e-a314-022ece01c121",
    "male": "7cf0e2b1-8daf-4fe4-89ad-f6039398f359",
}

# Initialize TTS service, starting with default narrator voice
tts = CartesiaTTSService(
    api_key=os.getenv("CARTESIA_API_KEY"),
    voice_id=VOICE_IDS["narrator"],
)

# Create pattern aggregator
pattern_aggregator = PatternPairAggregator()

# Add pattern for voice tags
pattern_aggregator.add_pattern(
    type="voice",
    start_pattern="<voice>",
    end_pattern="</voice>",
    action=MatchAction.REMOVE
)

# Register handler for voice switching
def on_voice_tag(match: PatternMatch):
    voice_name = match.content.strip().lower()
    if voice_name in VOICE_IDS:
        voice_id = VOICE_IDS[voice_name]
        tts.set_voice(voice_id)
        logger.info(f"Switched to {voice_name} voice")

pattern_aggregator.on_pattern_match("voice", on_voice_tag)

# Set the aggregator on an LLMTextProcessor
llm_text_processor = LLMTextProcessor(text_aggregator=pattern_aggregator)

# add the llm_text_processor to your pipeline after the llm and before the tts
# llm -> llm_text_processor -> tts
```

### Extracting Structured Data from LLM Outputs

This example shows how to extract JSON data blocks from LLM outputs, aggregating them separately to be removed from the spoken text, but not from the context or client display.

````python  theme={null}
# Create pattern aggregator
pattern_aggregator = PatternPairAggregator()

# Add pattern for JSON data
pattern_aggregator.add_pattern(
    type="json",
    start_pattern="```json",
    end_pattern="```",
    action=MatchAction.AGGREGATE
)

# Set the aggregator on an LLMTextProcessor
llm_text_processor = LLMTextProcessor(text_aggregator=pattern_aggregator)

# Initialize TTS service, and don't speak JSON data
tts = CartesiaTTSService(
    api_key=os.getenv("CARTESIA_API_KEY"),
    skip_aggregator_types=["json"],
)

# add the llm_text_processor to your pipeline after the llm and before the tts
# llm -> llm_text_processor -> tts
````

### Handling Special Values in LLM Output

This example demonstrates how to identify and process custom tags in LLM output that denote special content, such as credit cards that should be handled differently by downstream services.
In this case, the TTS should spell it out, while RTVI should obfuscate the number.

```python  theme={null}
from pipecat.utils.text.pattern_pair_aggregator import MatchAction, PatternPairAggregator

# Create pattern aggregator
pattern_aggregator = PatternPairAggregator()

# Add patterns for different parts of an explanation
pattern_aggregator.add_pattern(
    type="credit_card",
    start_pattern="<card>",
    end_pattern="</card>",
    action=MatchAction.AGGREGATE
)

# Set the aggregator on an LLMTextProcessor
llm_text_processor = LLMTextProcessor(text_aggregator=pattern_aggregator)

# Text-to-Speech service
tts = CartesiaTTSService(
    api_key=os.getenv("CARTESIA_API_KEY"),
)

# Text transformers for TTS
# This will insert Cartesia's spell tags around the provided text.
async def spell_out_text(text: str, type: str) -> str:
    return CartesiaTTSService.SPELL(text)

# Setup the text transformers in TTS to spell out credit card numbers.
# The string below matches the type defined in the PatternPairAggregator
# above so that whenever those segments are encountered, this transform
# is applied
tts.add_text_transformer(spell_out_text, "credit_card")

# RTVI is automatically enabled. Use rtvi_observer_params to add
# a text transformer that obfuscates credit card numbers in client output.
from pipecat.processors.frameworks.rtvi import RTVIObserverParams

def obfuscate_credit_card(text: str, type: str) -> str:
    return "XXXX-XXXX-XXXX-" + text[-4:]

task = PipelineTask(
    pipeline,  # llm -> llm_text_processor -> tts
    rtvi_observer_params=RTVIObserverParams(
        bot_output_transforms=[("credit_card", obfuscate_credit_card)],
    ),
)
```

## How It Works

```mermaid  theme={null}
flowchart TD
  A[New Text] --> B[Add to Buffer]
  B --> C{Start & End Patterns Found?}
    C --> |Yes| C1[Call Handlers]
    C1 --> D{Pattern Action?}
      D --> E{{AGGREGATE}}
        E --> E1[Reset buffer with trailing text]
        E1 --> E2[RETURN contents as Aggregation with specified type]
      D --> F{{REMOVE}}
        F --> F1[Remove pattern from buffer]
        F1 --> S{End of Sentence?}
      D --> G{{KEEP}}
        G --> S
    C --> |No| H{Start Pattern Found?}
      H --> |No| S
      H --> |Yes| I{Pattern Action == AGGREGATE?}
        I --> |Yes| I1[Reset buffer to pattern start]
        I1 --> I2[RETURN prior buffer as Aggregation of type 'sentence']
    S --> |Yes| S1[return Aggregation of type 'sentence']
    S --> | No| S2[return None]
```

## Notes

* Patterns are processed in the order they appear in the text
* Handlers are called when complete patterns are found
* Patterns can span multiple sentences of text, but be aware that encoding many "reasoning" tokens may slow down the LLM response

---

# Source: https://docs.pipecat.ai/deployment/pattern.md

> ## Documentation Index
> Fetch the complete documentation index at: https://docs.pipecat.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Deployment pattern

> Basic deployment pattern for Pipecat bots

## Project structure

A Pipecat project will often consist of the following:

<Card title="1. Bot file">
  E.g. `bot.py`. Your Pipecat bot / agent, containing all the pipelines that you
  want to run in order to communicate with an end-user. A bot file may take some
  command line arguments, such as a transport URL and configuration.
</Card>

<Card title="2. Bot runner">
  E.g. `bot_runner.py`. Typically a basic HTTP service that listens for incoming
  user requests and spawns the relevant bot file in response.
</Card>

<Note>
  You can call these files whatever you like! We use `bot.py` and
  `bot_runner.py` for simplicity.
</Note>

### Typical user / bot flow

There are many ways to approach connecting users to bots. Pipecat is unopinionated about *how* exactly you should do this, but it's helpful to put an idea forward.

At a very basic level, it may look something like this:

<Steps>
  <Step title="User requests to join session via client / app">
    Client initiates a HTTP request to a hosted bot runner service.
  </Step>

  <Step title="Bot runner handles the request">
    Authenticates, configures and instantiates everything necessary for the
    session to commence (e.g. a new WebSocket channel, or WebRTC room, etc.)
  </Step>

  <Step title="Bot runner spawns bot / agent">
    A new bot process / VM is created for the user to connect with (passing
    across any necessary configuration.) Your project may load just one bot
    file, contextually swap between multiple, or launch many at once.
  </Step>

  <Step title="Bot instantiates and joins session via specified transport credentials">
    Bot initializes, connects to the session (e.g. locally or via WebSockets,
    WebRTC etc) and runs your bot code.
  </Step>

  <Step title="Bot runner returns status to client">
    Once the bot is ready, the runner resolves the HTTP request with details for
    the client to connect.
  </Step>
</Steps>

<Frame><img src="https://mintcdn.com/daily/2bYrACcmgvvzC075/deployment/images/deployment-1.png?fit=max&auto=format&n=2bYrACcmgvvzC075&q=85&s=314cfcd2d3ea7f3473c86c21418cedb7" alt="Basic pipeline image" data-og-width="1148" width="1148" data-og-height="573" height="573" data-path="deployment/images/deployment-1.png" data-optimize="true" data-opv="3" srcset="https://mintcdn.com/daily/2bYrACcmgvvzC075/deployment/images/deployment-1.png?w=280&fit=max&auto=format&n=2bYrACcmgvvzC075&q=85&s=5afb211472eb55bb37571d2d609b325b 280w, https://mintcdn.com/daily/2bYrACcmgvvzC075/deployment/images/deployment-1.png?w=560&fit=max&auto=format&n=2bYrACcmgvvzC075&q=85&s=161899a4e9f7c446503456e47c106eba 560w, https://mintcdn.com/daily/2bYrACcmgvvzC075/deployment/images/deployment-1.png?w=840&fit=max&auto=format&n=2bYrACcmgvvzC075&q=85&s=05e1e875ae44bd1bbff3aa81d9a9b3cd 840w, https://mintcdn.com/daily/2bYrACcmgvvzC075/deployment/images/deployment-1.png?w=1100&fit=max&auto=format&n=2bYrACcmgvvzC075&q=85&s=60818d6a2e68772a3885965b4d68d9e6 1100w, https://mintcdn.com/daily/2bYrACcmgvvzC075/deployment/images/deployment-1.png?w=1650&fit=max&auto=format&n=2bYrACcmgvvzC075&q=85&s=3dab29b58f9e120628bcf1599dfc56eb 1650w, https://mintcdn.com/daily/2bYrACcmgvvzC075/deployment/images/deployment-1.png?w=2500&fit=max&auto=format&n=2bYrACcmgvvzC075&q=85&s=f798c9d545b3abac6a578c5e9aebfd75 2500w" /></Frame>

***

## Bot runner

The majority of use-cases require a way to trigger and manage a bot session over the internet. We call these bot runners; a HTTP service that provides a gateway for spawning bots on-demand.

The anatomy of a bot runner service is entirery arbitrary, but at very least will have a method that spawns a new bot process, for example:

```python  theme={null}
import uvicorn

from fastapi import FastAPI, Request, HTTPException
from fastapi.responses import JSONResponse

app = FastAPI()

@app.post("/start_bot")
async def start_bot(request: Request) -> JSONResponse:
    # ... handle / authenticate the request
    # ... setup the transport session

    # Spawn a new bot process
    try:
       #... create a new bot instance
    except Exception as e:
        raise HTTPException(
            status_code=500, detail=f"Failed to start bot: {e}")

    # Return a URL for the user to join
    return JSONResponse({...})

if __name__ == "__main__":
    uvicorn.run(
        "bot_runner:app",
        host="0.0.0.0",
        port=7860
    )

```

This pseudo code defines a `/start_bot/` endpoint which listens for incoming user `POST` requests or webhooks, then configures the session (such as creating rooms on your transport provider) and instantiates a new bot process.

A client will typically require some information regarding the newly spawned bot, such as a web address, so we also return some JSON with the necessary details.

### Data transport

Your [transport layer](/server/services/transport/daily) is responsible for sending and receiving audio and video data over the internet.

You will have implemented a transport layer as part of your `bot.py` pipeline. This may be a service that you want to host and include in your deployment, or it may be a third-party service waiting for peers to connect (such as [Daily](https://www.daily.co), or a websocket.)

For this example, we will make use of Daily’s WebRTC transport. This will mean that our `bot_runner.py` will need to do some configuration when it spawns a new bot:

1. Create and configure a new Daily room for the session to take place in.
2. Issue both the bot and the user an authentication token to join the session.

Whatever you use for your transport layer, you’ll likely need to setup some environmental variables and run some custom code before spawning the agent.

### Best practice for bot files

A good pattern to work to is the assumption that your `bot.py` is an encapsulated entity and does not have any knowledge of the `bot_runner.py`. You should provide the bot everything it needs to operate during instantiation.

Sticking to this approach helps keep things simple and makes it easier to step through debugging (if the bot launched and something goes wrong, you know to look for errors in your bot file.)

***

## Example

Let's assume we have a fully service-driven `bot.py` that connects to a WebRTC session, passes audio transcription to GPT4 and returns audio text-to-speech with ElevenLabs.

We'll also use Silero voice activity detection, to better know when the user has stopped talking.

```python bot.py theme={null}
import argparse
import asyncio
import os
import sys

import aiohttp
from dotenv import load_dotenv
from loguru import logger

from pipecat.audio.vad.silero import SileroVADAnalyzer
from pipecat.frames.frames import EndFrame, LLMContextFrame
from pipecat.pipeline.pipeline import Pipeline
from pipecat.pipeline.runner import PipelineRunner
from pipecat.pipeline.task import PipelineParams, PipelineTask
from pipecat.processors.aggregators.llm_context import LLMContext
from pipecat.processors.aggregators.llm_response import (
    LLMAssistantResponseAggregator,
    LLMUserResponseAggregator,
)
from pipecat.services.elevenlabs.tts import ElevenLabsTTSService
from pipecat.services.openai.llm import OpenAILLMService
from pipecat.transports.daily.transport import DailyParams, DailyTransport

load_dotenv(override=True)

logger.remove(0)
logger.add(sys.stderr, level="DEBUG")

daily_api_key = os.getenv("DAILY_API_KEY", "")
daily_api_url = os.getenv("DAILY_API_URL", "https://api.daily.co/v1")


async def main(room_url: str, token: str):
    async with aiohttp.ClientSession() as session:
        transport = DailyTransport(
            room_url,
            token,
            "Chatbot",
            DailyParams(
                api_url=daily_api_url,
                api_key=daily_api_key,
                audio_in_enabled=True,
                audio_out_enabled=True,
                video_out_enabled=False,
                transcription_enabled=True,
            ),
        )

        tts = ElevenLabsTTSService(
            aiohttp_session=session,
            api_key=os.getenv("ELEVENLABS_API_KEY", ""),
            voice_id=os.getenv("ELEVENLABS_VOICE_ID", ""),
        )

        llm = OpenAILLMService(api_key=os.getenv("OPENAI_API_KEY"), model="gpt-4o")

        messages = [
            {
                "role": "system",
                "content": "You are Chatbot, a friendly, helpful robot. Your output will be converted to audio so don't include special characters other than '!' or '?' in your answers. Respond to what the user said in a creative and helpful way, but keep your responses brief. Start by saying hello.",
            },
        ]

        context = LLMContext(messages)
        user_aggregator, assistant_aggregator = LLMContextAggregatorPair(
            context,
            user_params=LLMUserAggregatorParams(
                vad_analyzer=SileroVADAnalyzer(),
            ),
        )

        pipeline = Pipeline(
            [
                transport.input(),
                user_aggregator,
                llm,
                tts,
                transport.output(),
                assistant_aggregator,
            ]
        )

        task = PipelineTask(pipeline)

        @transport.event_handler("on_first_participant_joined")
        async def on_first_participant_joined(transport, participant):
            transport.capture_participant_transcription(participant["id"])
            await task.queue_frames([LLMContextFrame(LLMContext(messages))])

        @transport.event_handler("on_participant_left")
        async def on_participant_left(transport, participant, reason):
            await task.queue_frame(EndFrame())

        @transport.event_handler("on_call_state_updated")
        async def on_call_state_updated(transport, state):
            if state == "left":
                await task.queue_frame(EndFrame())

        runner = PipelineRunner()

        await runner.run(task)


if __name__ == "__main__":
    parser = argparse.ArgumentParser(description="Pipecat Bot")
    parser.add_argument("-u", type=str, help="Room URL")
    parser.add_argument("-t", type=str, help="Token")
    config = parser.parse_args()

    asyncio.run(main(config.u, config.t))
```

### HTTP API

To launch this bot, let's create a `bot_runner.py` that:

* Creates an API for users to send requests to.
* Launches a bot as a subprocess.

```python bot_runner.py theme={null}
import os
import argparse
import subprocess

from pipecat.transports.services.helpers.daily_rest import DailyRESTHelper, DailyRoomObject, DailyRoomProperties, DailyRoomParams

from fastapi import FastAPI, Request, HTTPException
from fastapi.middleware.cors import CORSMiddleware
from fastapi.responses import JSONResponse

# Load API keys from env
from dotenv import load_dotenv
load_dotenv(override=True)


# ------------ Configuration ------------ #

MAX_SESSION_TIME = 5 * 60  # 5 minutes
# List of require env vars our bot requires
REQUIRED_ENV_VARS = [
    'DAILY_API_KEY',
    'OPENAI_API_KEY',
    'ELEVENLABS_API_KEY',
    'ELEVENLABS_VOICE_ID']

daily_rest_helper = DailyRESTHelper(
    os.getenv("DAILY_API_KEY", ""),
    os.getenv("DAILY_API_URL", 'https://api.daily.co/v1'))


# ----------------- API ----------------- #

app = FastAPI()

app.add_middleware(
    CORSMiddleware,
    allow_origins=["*"],
    allow_credentials=True,
    allow_methods=["*"],
    allow_headers=["*"]
)

# ----------------- Main ----------------- #


@app.post("/start_bot")
async def start_bot(request: Request) -> JSONResponse:
    try:
        # Grab any data included in the post request
        data = await request.json()
    except Exception as e:
        pass

    # Create a new Daily WebRTC room for the session to take place in
    try:
        params = DailyRoomParams(
            properties=DailyRoomProperties()
        )
        room: DailyRoomObject = daily_rest_helper.create_room(params=params)
    except Exception as e:
        raise HTTPException(
            status_code=500,
            detail=f"Unable to provision room {e}")

    # Give the agent a token to join the session
    token = daily_rest_helper.get_token(room.url, MAX_SESSION_TIME)

    # Return an error if we were unable to create a room or a token
    if not room or not token:
        raise HTTPException(
            status_code=500, detail=f"Failed to get token for room: {room_url}")

    try:
        # Start a new subprocess, passing the room and token to the bot file
        subprocess.Popen(
            [f"python3 -m bot -u {room.url} -t {token}"],
            shell=True,
            bufsize=1,
            cwd=os.path.dirname(os.path.abspath(__file__)))
    except Exception as e:
        raise HTTPException(
            status_code=500, detail=f"Failed to start subprocess: {e}")

    # Grab a token for the user to join with
    user_token = daily_rest_helper.get_token(room.url, MAX_SESSION_TIME)

    # Return the room url and user token back to the user
    return JSONResponse({
        "room_url": room.url,
        "token": user_token,
    })


if __name__ == "__main__":
    # Check for required environment variables
    for env_var in REQUIRED_ENV_VARS:
        if env_var not in os.environ:
            raise Exception(f"Missing environment variable: {env_var}.")

    parser = argparse.ArgumentParser(description="Pipecat Bot Runner")
    parser.add_argument("--host", type=str,
                        default=os.getenv("HOST", "0.0.0.0"), help="Host address")
    parser.add_argument("--port", type=int,
                        default=os.getenv("PORT", 7860), help="Port number")
    parser.add_argument("--reload", action="store_true",
                        default=False, help="Reload code on change")

    config = parser.parse_args()

    try:
        import uvicorn

        uvicorn.run(
            "bot_runner:app",
            host=config.host,
            port=config.port,
            reload=config.reload
        )

    except KeyboardInterrupt:
        print("Pipecat runner shutting down...")

```

### Dockerfile

Since our bot is just using Python, our Dockerfile can be quite simple:

<CodeGroup>
  ```shell Dockerfile theme={null}
  FROM python:3.11-bullseye

  # Open port 7860 for http service

  ENV FAST_API_PORT=7860
  EXPOSE 7860

  # Install Python dependencies

  COPY \*.py .
  COPY ./requirements.txt requirements.txt
  RUN pip3 install --no-cache-dir --upgrade -r requirements.txt

  # Install models

  RUN python3 install_deps.py

  # Start the FastAPI server

  CMD python3 bot_runner.py --port ${FAST_API_PORT}

  ```

  ```python install_deps.py theme={null}
  import torch

  # Download (cache) the Silero VAD model
  torch.hub.load(repo_or_dir='snakers4/silero-vad', model='silero_vad', force_reload=True)
  ```
</CodeGroup>

The bot runner and bot `requirements.txt`:

```shell requirements.txt theme={null}
pipecat-ai[daily,openai,silero]
fastapi
uvicorn
python-dotenv
```

And finally, let's create a `.env` file with our service keys

```shell .env theme={null}
DAILY_API_KEY=...
OPENAI_API_KEY=...
ELEVENLABS_API_KEY=...
ELEVENLABS_VOICE_ID=...
```

***

### How it works

Right now, this runner is spawning `bot.py` as a subprocess. When spawning the process, we pass through the transport room and token as system arguments to our bot, so it knows where to connect.

Subprocesses serve as a great way to test out your bot in the cloud without too much hassle, but depending on the size of the host machine, it will likely not hold up well under load.

Whilst some bots are just simple operators between the transport and third-party AI services (such as OpenAI), others have somewhat CPU-intensive operations, such as running and loading VAD models, so you may find you’re only able to scale this to support up to 5-10 concurrent bots.

Scaling your setup would require virtualizing your bot with it's own set of system resources, the process of which depends on your cloud provider.

### Best practices

In an ideal world, we'd recommend containerizing your bot and bot runner independently so you can deploy each without any unnecessary dependencies or models.

Most cloud providers will offer a way to deploy various images programmatically, which we explore in the various provider examples in these docs.

For the sake of simplicity defining this pattern, we're just using one container for everything.

### Build and run

We should now have a project that contains the following files:

* `bot.py`
* `bot_runner.py`
* `requirements.txt`
* `.env`
* `Dockerfile`

You can now `docker build ...` and deploy your container. Of course, you can still work with your bot in local development too:

```shell  theme={null}
# Install and activate a virtual env
python -m venv venv
source venv/bin/activate # or OS equivalent

pip install -r requirements.txt

python bot_runner.py --host localhost --reload
```

---

# Source: https://docs.pipecat.ai/server/services/llm/perplexity.md

> ## Documentation Index
> Fetch the complete documentation index at: https://docs.pipecat.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Perplexity

> LLM service implementation using Perplexity's API with OpenAI-compatible interface

## Overview

`PerplexityLLMService` provides access to Perplexity's language models through an OpenAI-compatible interface. It inherits from `OpenAILLMService` and supports streaming responses and context management, with special handling for Perplexity's incremental token reporting and built-in internet search capabilities.

<CardGroup cols={2}>
  <Card title="Perplexity LLM API Reference" icon="code" href="https://reference-server.pipecat.ai/en/latest/api/pipecat.services.perplexity.llm.html">
    Pipecat's API methods for Perplexity integration
  </Card>

  <Card title="Example Implementation" icon="play" href="https://github.com/pipecat-ai/pipecat/blob/main/examples/foundational/14n-function-calling-perplexity.py">
    Complete example with search capabilities
  </Card>

  <Card title="Perplexity Documentation" icon="book" href="https://docs.perplexity.ai/api-reference/chat-completions-post">
    Official Perplexity API documentation and features
  </Card>

  <Card title="Perplexity Platform" icon="microphone" href="https://www.perplexity.ai/">
    Access search-enhanced models and API keys
  </Card>
</CardGroup>

## Installation

To use Perplexity services, install the required dependencies:

```bash  theme={null}
pip install "pipecat-ai[perplexity]"
```

## Prerequisites

### Perplexity Account Setup

Before using Perplexity LLM services, you need:

1. **Perplexity Account**: Sign up at [Perplexity](https://www.perplexity.ai/)
2. **API Key**: Generate an API key from your account dashboard
3. **Model Selection**: Choose from available models with built-in search capabilities

### Required Environment Variables

* `PERPLEXITY_API_KEY`: Your Perplexity API key for authentication

<Note>
  Unlike other LLM services, Perplexity does not support function calling.
  Instead, they offer native internet search built in without requiring special
  function calls.
</Note>

---

# Source: https://docs.pipecat.ai/deployment/platforms/pipecat-cloud.md

> ## Documentation Index
> Fetch the complete documentation index at: https://docs.pipecat.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Pipecat Cloud

> Deploy Pipecat agents with managed infrastructure

[Pipecat Cloud](https://pipecat.daily.co) is a managed platform for hosting and scaling Pipecat agents in production.

<Card title="Get Started with Pipecat Cloud" icon="cloud" href="../pipecat-cloud/introduction">
  Visit the complete Pipecat Cloud documentation to deploy your first agent
</Card>

---

# Source: https://docs.pipecat.ai/server/frameworks/flows/pipecat-flows.md

# Source: https://docs.pipecat.ai/guides/features/pipecat-flows.md

> ## Documentation Index
> Fetch the complete documentation index at: https://docs.pipecat.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Pipecat Flows

> Learn how to create structured conversations using Pipecat's flow system

## What is Pipecat Flows?

Pipecat Flows is an add-on framework for Pipecat that allows you to build structured conversations in your AI applications. It enables you to define conversation paths while handling the complexities of state management and LLM interactions.

Want to dive right in? Check out these examples:

<CardGroup cols={2}>
  <Card title="Hello World" icon="rocket" href="https://github.com/pipecat-ai/pipecat-flows/tree/main/examples/quickstart">
    A great first Flow to show you the ropes
  </Card>

  <Card title="Restaurant Reservation" icon="utensils" href="https://github.com/pipecat-ai/pipecat-flows/blob/main/examples/restaurant_reservation.py">
    A simple, practical Flow example
  </Card>
</CardGroup>

## How do Pipecat and Pipecat Flows work together?

**Pipecat** defines the core capabilities of your bot. This includes the pipeline and processors which, at a minimum, enable your bot to:

* Receive audio from a user
* Transcribe the user's input
* Run an LLM completion
* Convert the LLM response to audio
* Send audio back to the user

**Pipecat Flows** complements Pipecat's core functionality by providing structure to a conversation, managing context and tools as the conversation progresses from one state to another. This is separate from the core pipeline, allowing you to separate conversation logic from core pipeline mechanics.

## When to Use Pipecat Flows?

Pipecat Flows is best suited for use cases where:

* **You need precise control** over how a conversation progresses through specific steps
* **Your bot handles complex tasks** that can be broken down into smaller, manageable pieces
* **You want to improve LLM accuracy** by focusing the model on one specific task at a time instead of managing multiple responsibilities simultaneously

This approach addresses a common problem: traditional methods often use large, monolithic prompts with many tools available at once, leading to hallucinations and lower accuracy.

Pipecat Flows solves this by:

* **Breaking complex tasks into focused steps** - Each node has a clear, single purpose
* **Providing relevant tools only when needed** - Functions are available only in the appropriate context
* **Giving clear, specific instructions** - Task messages focus the LLM on exactly what to do next

## Selecting a Flows Pattern

Pipecat Flows provides two general patterns for how to build your application: Dynamic Flows and Static Flows.

* **Dynamic Flows (recommended)**: When conversation paths need to be determined at runtime based on user input, external data, or business logic.
* **Static Flows**: When your conversation structure is known upfront and follows predefined paths.

<Note>
  Dynamic Flows can handle both simple and complex use cases. Selecting it
  provides you with an option that can grow with the complexity of your
  application. For these reasons, we strongly recommend it as the API pattern to
  follow.
</Note>

## Technical Overview

Pipecat Flows represents a conversation as a graph, where each step of the conversation is represented by a node. Nodes are of type `NodeConfig`, and may contain the following properties:

* `name`: The name of the node; used as a reference to transition to the node.
* `role_messages`: A list of message `dicts` defining the bot's role/personality. Typically set once in the initial node.
* `task_messages`: A list of message `dicts` defining the current node's objectives.
* `functions`: A list of function call definitions and their corresponding handlers.
* `pre_actions`: Actions to execute before LLM inference. Actions run once upon transitioning to a node.
* `post_actions`: Actions to execute after LLM inference. Actions run once after the node's initial LLM inference.
* `context_strategy`: Strategy for updating context during transitions. The default behavior is to append messages to the context.
* `respond_immediately`: Whether to run LLM inference as soon as the node is set. The default is True.

<Info>
  The only required field is `task_messages`, as your bot always needs a prompt
  to advance the conversation.
</Info>

Now that we've defined the structure of a node, let's look at the components that make up a node.

### Messages

Messages define what your bot should do and how it should behave at each node in your conversation flow.

#### Message Types

There are two types of messages you can configure:

**Role Messages** (Optional)
Define your bot's personality, tone, and overall behavior. These are typically set once at the beginning of your flow and establish the consistent persona your bot maintains throughout the conversation.

**Task Messages** (Required)
Define the specific objective your bot should accomplish in the current node. These messages focus the LLM on the immediate task at hand, such as asking a specific question or processing particular information.

#### Message Format

Messages are specified in OpenAI format as a list of `dicts`:

```python  theme={null}
"role_messages": [
    {
        "role": "system",
        "content": "You are an inquisitive child.",
    }
],
"task_messages": [
    {
        "role": "system",
        "content": "Say 'Hello world' and ask what is the user's favorite color.",
    }
],
```

#### Cross-Provider Compatibility

Pipecat's default message format uses the OpenAI message spec. With this message format, messages are automatically translated by Pipecat to work with your chosen LLM provider, making it easy to switch between OpenAI, Anthropic, Google, and other providers without changing your code.

<Warning>
  Some LLMs, like Anthropic and Gemini, can only set the system instruction at
  initialization time. This means you will only be able to set the
  `role_messages` at initialization for those LLMs.
</Warning>

### Functions

Functions in Pipecat Flows serve two key purposes:

1. **Process data** by interfacing with external systems and APIs to read or write information
2. **Progress the conversation** by transitioning between nodes in your flow

#### How Functions Work

When designing your nodes, clearly define the task in the `task_messages` and reference the available functions. The LLM will use these functions to complete the task and signal when it's ready to move forward.

For example, if your node's job is to collect a user's favorite color:

1. The LLM asks the question
2. The user provides their answer
3. The LLM calls the function with the answer
4. The function processes the data and determines the next node

#### Function Definition

Flows provides a universal `FlowsFunctionSchema` that works across all LLM providers:

```python  theme={null}
from pipecat_flows import FlowsFunctionSchema

record_favorite_color_func = FlowsFunctionSchema(
    name="record_favorite_color_func",
    description="Record the color the user said is their favorite.",
    required=["color"],
    handler=record_favorite_color_and_set_next_node,
    properties={"color": {"type": "string"}},
)
```

#### Function Handlers

Each function has a corresponding `handler` where you implement your application logic and specify the next node:

```python  theme={null}
async def record_favorite_color_and_set_next_node(
    args: FlowArgs, flow_manager: FlowManager
) -> tuple[str, NodeConfig]:
    """Function handler that records the color then sets the next node.

    Here "record" means print to the console, but any logic could go here:
    Write to a database, make an API call, etc.
    """
    print(f"Your favorite color is: {args['color']}")
    return args["color"], create_end_node()
```

#### Handler Return Values

Function handlers must return a tuple containing:

* **Result**: Data provided to the LLM for context in subsequent completions
* **Next Node**: The `NodeConfig` for Flows to transition to next

Some handlers may not want to transition conversational state, in which case you can return `None` for the next node. Other handlers may *only* want to transition conversational state without doing other work, in which case you can return `None` for the result.

#### Direct Functions

For more concise code, you can optionally use Direct Functions where the function definition and handler are combined in a single function. The function signature and docstring are automatically used to generate the function schema:

```python  theme={null}
async def record_favorite_color(
    flow_manager: FlowManager,
    color: str
) -> tuple[FlowResult, NodeConfig]:
    """
    Record the color the user said is their favorite.

    Args:
        color (str): The user's favorite color
    """
    print(f"Your favorite color is: {args["color"]}")
    return args["color"], create_end_node()

# Use directly in NodeConfig
node_config = {
    "functions": [record_favorite_color]
}
```

This approach eliminates the need for separate `FlowsFunctionSchema` definitions while maintaining the same functionality.

### Actions

Actions allow you to execute custom functionality at specific points in your conversation flow, giving you precise control over timing and sequencing.

#### Action Types

* `pre_actions` execute immediately when transitioning to a new node, *before* the LLM inference begins.
* `post_actions` execute after the LLM inference completes and any TTS has finished speaking.

#### Built-in Actions

Pipecat Flows includes several ready-to-use actions for common scenarios:

* **`tts_say`**: Speak a phrase immediately (useful for "please wait" messages)

```python  theme={null}
"pre_actions": [
    {
        "type": "tts_say",
        "text": "Please hold while I process your request..."
    }
]
```

* **`end_conversation`**: Gracefully terminate the conversation

```python  theme={null}
"post_actions": [
    {
        "type": "end_conversation",
        "text": "Thank you for your time!"
    }
]
```

* **`function`**: Execute a custom function at the specified timing

```python  theme={null}
"post_actions": [
    {
        "type": "function",
        "handler": end_conversation_handler
    }
]
```

#### Custom Actions

You can define your own actions to handle specific business logic or integrations. In most cases, consider using a **function action** first, as it executes at the expected time in the pipeline.

Custom actions give you complete flexibility to execute any functionality your application needs, but require careful timing considerations.

#### Action Timing

The execution order ensures predictable behavior:

1. **Pre-actions** run first upon node entry (in the order they are defined)
2. **LLM inference** processes the node's messages and functions
3. **TTS** speaks the LLM's response
4. **Post-actions** run after TTS completes (in the order they are defined)

This timing guarantees that actions execute in the correct sequence, such as ensuring the bot finishes speaking before ending the conversation. Note that custom actions may not follow this predictable timing, which is another reason to prefer function actions when possible.

### Context Strategy

Flows provides three built-in ways to manage conversation context as you move between nodes:

#### Strategy Types

1. **APPEND** (Default): New node messages are added to the existing context, preserving the full conversation history. The context grows as the conversation progresses.

2. **RESET**: The context is cleared and replaced with only the new node's messages. Useful when previous conversation history is no longer relevant or to reduce context window size.

3. **RESET\_WITH\_SUMMARY**: The context is cleared but includes an AI-generated summary of the previous conversation along with the new node's messages. Helps reduce context size while preserving key information.

#### When to Use Each Strategy

* Use **APPEND** when full conversation history is important for context
* Use **RESET** when starting a new topic or when previous context might confuse the current task
* Use **RESET\_WITH\_SUMMARY** for long conversations where you need to preserve key points but reduce context size

#### Configuration Examples

Context strategies can be defined globally in the FlowManager constructor:

```python  theme={null}
from pipecat_flows import ContextStrategy, ContextStrategyConfig

# Global strategy configuration
flow_manager = FlowManager(
    task=task,
    llm=llm,
    context_aggregator=context_aggregator,
    context_strategy=ContextStrategyConfig(
        strategy=ContextStrategy.APPEND,
    )
)
```

Or on a per-node basis:

```python  theme={null}
# Per-node strategy configuration
node_config = {
    "task_messages": [...],
    "functions": [...],
    "context_strategy": ContextStrategyConfig(
        strategy=ContextStrategy.RESET_WITH_SUMMARY,
        summary_prompt="Provide a concise summary of the customer's order details and preferences."
    )
}
```

### Respond Immediately

For each node in the conversation, you can decide whether the LLM should respond immediately upon entering the node (the default behavior) or whether the LLM should wait for the user to speak first before responding. You do this using the `respond_immediately` field.

<Tip>
  `respond_immediately=False` may be particularly useful in the very first node,
  especially in outbound-calling cases where the user has to first answer the
  phone to trigger the conversation.
</Tip>

```python  theme={null}
NodeConfig(
    task_messages=[
        {
            "role": "system",
            "content": "Warmly greet the customer and ask how many people are in their party. This is your only job for now; if the customer asks for something else, politely remind them you can't do it.",
        }
    ],
    respond_immediately=False,
    # ... other fields
)
```

<Warning>
  Keep in mind that if you specify `respond_immediately=False`, the user may not
  be aware of the conversational task at hand when entering the node (the bot
  hasn't told them yet). While it's always important to have guardrails in your
  node messages to keep the conversation on topic, letting the user speak first
  makes it even more so.
</Warning>

## Initialization

Initialize your flow by creating a `FlowManager` instance and calling `initialize()` to start the conversation.

### Dynamic Flow

First, create the FlowManager:

```python  theme={null}
flow_manager = FlowManager(
    task=task,                              # PipelineTask
    llm=llm,                                # LLMService
    context_aggregator=context_aggregator,  # Context aggregator
    transport=transport,                    # Transport
)
```

Then, initialize by passing the first `NodeConfig` into the `initialize()` method:

```python  theme={null}
@transport.event_handler("on_client_connected")
async def on_client_connected(transport, client):
    logger.info(f"Client connected")
    # Kick off the conversation.
    await flow_manager.initialize(create_initial_node())
```

### Static Flow

First, create the FlowManager:

```python  theme={null}
flow_manager = FlowManager(
    task=task,                              # PipelineTask
    llm=llm,                                # LLMService
    context_aggregator=context_aggregator,  # Context aggregator
    flow_config=flow_config,                # FlowConfig
)
```

Then initialize:

```python  theme={null}
@transport.event_handler("on_first_participant_joined")
async def on_first_participant_joined(transport, participant):
    await transport.capture_participant_transcription(participant["id"])
    logger.debug("Initializing flow")
    await flow_manager.initialize()
```

<Note>For Static Flows, `initialize()` does not take any args.</Note>

## Cross-Node Logic

### State

Pipecat Flows supports cross-node state through the `flow_manager.state` dictionary. This persistent storage lets you share data across nodes throughout the entire conversation:

```python  theme={null}
async def record_favorite_color_and_set_next_node(
    args: FlowArgs, flow_manager: FlowManager
) -> tuple[str, NodeConfig]:
    """Function handler that records the color then sets the next node.

    Here "record" means print to the console, but any logic could go here;
    Write to a database, make an API call, etc.
    """
    flow_manager.state["color"] = args["color"] # Cross-node state setting
    print(f"Your favorite color is: {args['color']}")
    return args["color"], create_end_node()
```

### Functions

Pipecat Flows supports defining functions that are available across all nodes in your flow. They're defined in the same way as node-specific functions, but are passed into the FlowManager at initialization:

```python  theme={null}
flow_manager = FlowManager(
    task=task,
    llm=llm,
    context_aggregator=context_aggregator,
    transport=transport,
    global_functions=[global_function_1, global_function_2], # Cross-node functions
)
```

## Usage Examples:

### Dynamic Flow (Recommended)

Here's an example that ties together all the concepts we've covered:

```python  theme={null}
def create_initial_node() -> NodeConfig:
    """Create the initial node of the flow.

    Define the bot's role and task for the node as well as the function for it to call.
    The function call includes a handler which provides the function call result to
    Pipecat and then transitions to the next node.
    """
    record_favorite_color_func = FlowsFunctionSchema(
        name="record_favorite_color_func",
        description="Record the color the user said is their favorite.",
        required=["color"],
        handler=record_favorite_color_and_set_next_node,
        properties={"color": {"type": "string"}},
    )

    return {
        "name": "initial",
        "role_messages": [
            {
                "role": "system",
                "content": "You are an inquisitive child. Use very simple language. Ask simple questions. You must ALWAYS use one of the available functions to progress the conversation. Your responses will be converted to audio. Avoid outputting special characters and emojis.",
            }
        ],
        "task_messages": [
            {
                "role": "system",
                "content": "Say 'Hello world' and ask what is the user's favorite color.",
            }
        ],
        "functions": [record_favorite_color_func],
    }
```

### Static Flow

For static flows, you define your entire flow structure upfront using a `FlowConfig`. This follows the same rules as the NodeConfig, but is assembled as a single JSON object that defines the entire state of the program.

## Next Steps

Now that you understand the basics of Pipecat Flows, explore the reference docs and more examples:

<CardGroup cols={2}>
  <Card title="Pipecat Flows Reference" icon="book" href="/server/frameworks/flows/pipecat-flows">
    Complete API reference and technical details
  </Card>

  <Card title="Pipecat Flows Examples" icon="rocket" href="https://github.com/pipecat-ai/pipecat-flows/tree/main/examples">
    Explore more complex examples and use cases
  </Card>
</CardGroup>

---

# Source: https://docs.pipecat.ai/server/pipeline/pipeline-idle-detection.md

> ## Documentation Index
> Fetch the complete documentation index at: https://docs.pipecat.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Pipeline Idle Detection

> Automatically detect and handle idle pipelines with no bot activity

## Overview

Pipeline idle detection is a feature that monitors activity in your pipeline and can automatically cancel tasks when no meaningful bot interactions are occurring. This helps prevent pipelines from running indefinitely when a conversation has naturally ended but wasn't properly terminated.

## How It Works

The system monitors specific "activity frames" that indicate the bot is actively engaged in the conversation. By default, these are:

* `BotSpeakingFrame` - When the bot is speaking
* `LLMFullResponseEndFrame` - When the LLM has completed a response

If no activity frames are detected within the configured timeout period (5 minutes by default), the system considers the pipeline idle and can automatically terminate it.

<Note>
  Idle detection only starts after the pipeline has begun processing frames. The
  idle timer resets whenever an activity frame (as specified in
  `idle_timeout_frames`) is received.
</Note>

## Configuration

You can configure idle detection behavior when creating a `PipelineTask`:

```python  theme={null}
from pipecat.pipeline.task import PipelineParams, PipelineTask

# Default configuration - cancel after 5 minutes of inactivity
task = PipelineTask(pipeline)

# Custom configuration
task = PipelineTask(
    pipeline,
    idle_timeout_secs=600,  # 10 minute timeout
    idle_timeout_frames=(BotSpeakingFrame,),  # Only monitor bot speaking
    cancel_on_idle_timeout=False,  # Don't auto-cancel, just notify
)
```

## Configuration Parameters

<ParamField path="idle_timeout_secs" type="Optional[float]" default="300">
  Timeout in seconds before considering the pipeline idle. Set to `None` to
  disable idle detection.
</ParamField>

<ParamField path="idle_timeout_frames" type="Tuple[Type[Frame], ...]" default="(BotSpeakingFrame, LLMFullResponseEndFrame)">
  Frame types that should prevent the pipeline from being considered idle.
</ParamField>

<ParamField path="cancel_on_idle_timeout" type="bool" default="True">
  Whether to automatically cancel the pipeline task when idle timeout is
  reached.
</ParamField>

## Handling Idle Timeouts

You can respond to idle timeout events by adding an event handler:

```python  theme={null}
@task.event_handler("on_idle_timeout")
async def on_idle_timeout(task):
    logger.info("Pipeline has been idle for too long")
    # Perform any custom cleanup or logging
    # Note: If cancel_on_idle_timeout=True, the pipeline will be cancelled after this handler runs
```

## Example Implementation

Here's a complete example showing how to configure idle detection with custom handling:

```python  theme={null}
from pipecat.frames.frames import BotSpeakingFrame, LLMFullResponseEndFrame, TTSSpeakFrame
from pipecat.pipeline.runner import PipelineRunner
from pipecat.pipeline.task import PipelineParams, PipelineTask

# Create pipeline
pipeline = Pipeline([...])

# Configure task with custom idle settings
task = PipelineTask(
    pipeline,
    idle_timeout_secs=180,  # 3 minutes
    cancel_on_idle_timeout=False  # Don't auto-cancel
)

# Add event handler for idle timeout
@task.event_handler("on_idle_timeout")
async def on_idle_timeout(task):
    logger.info("Conversation has been idle for 3 minutes")

    # Add a farewell message
    await task.queue_frame(TTSSpeakFrame("I haven't heard from you in a while. Goodbye!"))

    # Then end the conversation gracefully
    await task.stop_when_done()

runner = PipelineRunner()

await runner.run(task)
```

---

# Source: https://docs.pipecat.ai/server/pipeline/pipeline-params.md

> ## Documentation Index
> Fetch the complete documentation index at: https://docs.pipecat.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# PipelineParams

> Configure pipeline execution with PipelineParams

## Overview

The `PipelineParams` class provides a structured way to configure various aspects of pipeline execution. These parameters control behaviors like audio settings, metrics collection, heartbeat monitoring, and interruption handling.

## Basic Usage

```python  theme={null}
from pipecat.pipeline.task import PipelineParams, PipelineTask

# Create with default parameters
params = PipelineParams()

# Or customize specific parameters
params = PipelineParams(
    audio_in_sample_rate=16000,
    enable_metrics=True
)

# Pass to PipelineTask
pipeline = Pipeline([...])
task = PipelineTask(pipeline, params=params)
```

## Available Parameters

<ParamField path="allow_interruptions" type="bool" default="False">
  <Warning>
    DEPRECATED: This parameter is deprecated. Configure interruption behavior
    via [User Turn
    Strategies](/server/utilities/turn-management/user-turn-strategies) instead.
    See the `enable_interruptions` parameter on start strategies.
  </Warning>

  Whether to allow pipeline interruptions. When enabled, a user's speech will immediately
  interrupt the bot's response.
</ParamField>

<ParamField path="audio_in_sample_rate" type="int" default="16000">
  Input audio sample rate in Hz.

  <Info>
    Setting the `audio_in_sample_rate` as a `PipelineParam` sets the input sample
    rate for all corresponding services in the pipeline.
  </Info>
</ParamField>

<ParamField path="audio_out_sample_rate" type="int" default="24000">
  Output audio sample rate in Hz.

  <Info>
    Setting the `audio_out_sample_rate` as a `PipelineParam` sets the output
    sample rate for all corresponding services in the pipeline.
  </Info>
</ParamField>

<ParamField path="enable_heartbeats" type="bool" default="False">
  Whether to enable heartbeat monitoring to detect pipeline stalls. See
  [Heartbeats](/server/pipeline/heartbeats) for details.
</ParamField>

<ParamField path="heartbeats_period_secs" type="float" default="1.0">
  Period between heartbeats in seconds (when heartbeats are enabled).
</ParamField>

<ParamField path="enable_metrics" type="bool" default="False">
  Whether to enable metrics collection for pipeline performance.
</ParamField>

<ParamField path="enable_usage_metrics" type="bool" default="False">
  Whether to enable usage metrics tracking.
</ParamField>

<ParamField path="report_only_initial_ttfb" type="bool" default="False">
  Whether to report only initial time to first byte metric.
</ParamField>

<ParamField path="send_initial_empty_metrics" type="bool" default="True">
  Whether to send initial empty metrics frame at pipeline start.
</ParamField>

<ParamField path="start_metadata" type="Dict[str, Any]" default="{}">
  Additional metadata to include in the StartFrame.
</ParamField>

## Common Configurations

### Audio Processing Configuration

You can set the audio input and output sample rates in the `PipelineParams` to set the sample rate for all input and output services in the pipeline. This acts as a convenience to avoid setting the sample rate for each service individually. Note, if services are set individually, they will supersede the values set in `PipelineParams`.

```python  theme={null}
params = PipelineParams(
    audio_in_sample_rate=8000,   # Lower quality input audio
    audio_out_sample_rate=8000  # High quality output audio
)
```

### Performance Monitoring Configuration

Pipeline heartbeats provide a way to monitor the health of your pipeline by sending periodic heartbeat frames through the system. When enabled, the pipeline will send heartbeat frames every second and monitor their progress through the pipeline.

```python  theme={null}
params = PipelineParams(
    enable_heartbeats=True,
    heartbeats_period_secs=2.0,  # Send heartbeats every 2 seconds
    enable_metrics=True
)
```

## How Parameters Are Used

The parameters you set in `PipelineParams` are passed to various components of the pipeline:

1. **StartFrame**: Many parameters are included in the StartFrame that initializes the pipeline
2. **Metrics Collection**: Metrics settings configure what performance data is gathered
3. **Heartbeat Monitoring**: Controls the pipeline's health monitoring system
4. **Audio Processing**: Sample rates affect how audio is processed throughout the pipeline

## Complete Example

```python  theme={null}
from pipecat.frames.frames import TTSSpeakFrame
from pipecat.observers.file_observer import FileObserver
from pipecat.pipeline.task import PipelineParams, PipelineTask
from pipecat.pipeline.runner import PipelineRunner

# Create comprehensive parameters
params = PipelineParams(
    audio_in_sample_rate=8000,
    audio_out_sample_rate=8000,
    enable_heartbeats=True,
    enable_metrics=True,
    enable_usage_metrics=True,
    heartbeats_period_secs=1.0,
    report_only_initial_ttfb=False,
    start_metadata={
        "conversation_id": "conv-123",
        "session_data": {
            "user_id": "user-456",
            "start_time": "2023-10-25T14:30:00Z"
        }
    }
)

# Create pipeline and task
pipeline = Pipeline([...])
task = PipelineTask(
    pipeline,
    params=params,
    observers=[FileObserver("pipeline_logs.jsonl")]
)

# Run the pipeline
runner = PipelineRunner()
await runner.run(task)
```

## Additional Information

* Parameters are immutable once the pipeline starts
* The `start_metadata` dictionary can contain any serializable data
* For metrics collection to work properly, `enable_metrics` must be set to `True`

---

# Source: https://docs.pipecat.ai/server/pipeline/pipeline-task.md

> ## Documentation Index
> Fetch the complete documentation index at: https://docs.pipecat.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# PipelineTask

> Manage pipeline execution and lifecycle with PipelineTask

## Overview

`PipelineTask` is the central class for managing pipeline execution. It handles the lifecycle of the pipeline, processes frames in both directions, manages task cancellation, and provides event handlers for monitoring pipeline activity.

## Basic Usage

```python  theme={null}
from pipecat.pipeline.pipeline import Pipeline
from pipecat.pipeline.runner import PipelineRunner
from pipecat.pipeline.task import PipelineParams, PipelineTask

# Create a pipeline
pipeline = Pipeline([...])

# Create a task with the pipeline
task = PipelineTask(pipeline)

# Queue frames for processing
await task.queue_frame(TTSSpeakFrame("Hello, how can I help you today?"))

# Run the pipeline
runner = PipelineRunner()
await runner.run(task)
```

## Constructor Parameters

<ParamField path="pipeline" type="BasePipeline" required>
  The pipeline to execute.
</ParamField>

<ParamField path="params" type="PipelineParams" default="PipelineParams()">
  Configuration parameters for the pipeline. See
  [PipelineParams](/server/pipeline/pipeline-params) for details.
</ParamField>

<ParamField path="observers" type="List[BaseObserver]" default="[]">
  List of observers for monitoring pipeline execution. See
  [Observers](/server/utilities/observers/observer-pattern) for details.
</ParamField>

<ParamField path="clock" type="BaseClock" default="SystemClock()">
  Clock implementation for timing operations.
</ParamField>

<ParamField path="task_manager" type="Optional[BaseTaskManager]" default="None">
  Custom task manager for handling asyncio tasks. If None, a default TaskManager
  is used.
</ParamField>

<ParamField path="check_dangling_tasks" type="bool" default="True">
  Whether to check for processors' tasks finishing properly.
</ParamField>

<ParamField path="idle_timeout_secs" type="Optional[float]" default="300">
  Timeout in seconds before considering the pipeline idle. Set to None to
  disable idle detection. See [Pipeline Idle
  Detection](/server/pipeline/pipeline-idle-detection) for details.
</ParamField>

<ParamField path="idle_timeout_frames" type="Tuple[Type[Frame], ...]" default="(BotSpeakingFrame, LLMFullResponseEndFrame)">
  Frame types that should prevent the pipeline from being considered idle. See
  [Pipeline Idle Detection](/server/pipeline/pipeline-idle-detection) for
  details.
</ParamField>

<ParamField path="cancel_on_idle_timeout" type="bool" default="True">
  Whether to automatically cancel the pipeline task when idle timeout is
  reached. See [Pipeline Idle
  Detection](/server/pipeline/pipeline-idle-detection) for details.
</ParamField>

<ParamField path="enable_tracing" type="bool" default="False">
  Whether to enable OpenTelemetry tracing. See [The OpenTelemetry
  guide](/server/utilities/opentelemetry) for details.
</ParamField>

<ParamField path="enable_turn_tracking" type="bool" default="False">
  Whether to enable turn tracking. See [The OpenTelemetry
  guide](/server/utilities/opentelemetry) for details.
</ParamField>

<ParamField path="conversation_id" type="Optional[str]" default="None">
  Custom ID for the conversation. If not provided, a UUID will be generated. See
  [The OpenTelemetry guide](/server/utilities/opentelemetry) for details.
</ParamField>

<ParamField path="additional_span_attributes" type="Optional[dict]" default="None">
  Any additional attributes to add to top-level OpenTelemetry conversation span.
  See [The OpenTelemetry guide](/server/utilities/opentelemetry) for details.
</ParamField>

## Methods

### Task Lifecycle Management

<ResponseField name="run()" type="async">
  Starts and manages the pipeline execution until completion or cancellation.

  ```python  theme={null}
  await task.run()
  ```
</ResponseField>

<ResponseField name="stop_when_done()" type="async">
  Sends an EndFrame to the pipeline to gracefully stop the task after all queued
  frames have been processed.

  ```python  theme={null}
  await task.stop_when_done()
  ```
</ResponseField>

<ResponseField name="cancel()" type="async">
  Stops the running pipeline immediately by sending a CancelFrame.

  ```python  theme={null}
    await task.cancel()
  ```
</ResponseField>

<ResponseField name="has_finished()" type="bool">
  Returns whether the task has finished (all processors have stopped).

  ```python  theme={null}
  if task.has_finished(): print("Task is complete")
  ```
</ResponseField>

### Frame Management

<ResponseField name="queue_frame()" type="async">
  Queues a single frame to be pushed down the pipeline.

  ```python  theme={null}
  await task.queue_frame(TTSSpeakFrame("Hello!"))
  ```
</ResponseField>

<ResponseField name="queue_frames()" type="async">
  Queues multiple frames to be pushed down the pipeline.

  ```python  theme={null}
  frames = [TTSSpeakFrame("Hello!"), TTSSpeakFrame("How are you?")]

  await task.queue_frames(frames)

  ```
</ResponseField>

## Event Handlers

PipelineTask provides an event handler that can be registered using the `event_handler` decorator:

### on\_idle\_timeout

Triggered when no activity frames (as specified by `idle_timeout_frames`) have been received within the idle timeout period.

```python  theme={null}
@task.event_handler("on_idle_timeout")
async def on_idle_timeout(task):
    print("Pipeline has been idle too long")
    await task.queue_frame(TTSSpeakFrame("Are you still there?"))
```

---

# Source: https://docs.pipecat.ai/guides/learn/pipeline-termination.md

> ## Documentation Index
> Fetch the complete documentation index at: https://docs.pipecat.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Pipeline Termination

> Learn how to properly terminate Pipecat pipelines for clean shutdown and resource management

**Pipeline termination** ensures your voice AI applications shut down cleanly without resource leaks or hanging processes. Understanding the different termination methods helps you handle various scenarios from natural conversation endings to unexpected disconnections.

## Pipeline Integration

Pipeline termination works through the same frame-based system as other pipeline operations:

```python  theme={null}
pipeline = Pipeline([
    transport.input(),
    stt,
    context_aggregator.user(),
    llm,
    tts,
    transport.output(),
    context_aggregator.assistant(),
])
# EndFrame or CancelFrame flows through entire pipeline for shutdown
```

**Termination frames:**

* **`EndFrame`**: A queued ControlFrame that triggers graceful shutdown after processing pending frames
* **`CancelFrame`**: A SystemFrame that triggers immediate shutdown, discarding pending frames

Both frames flow downstream through the pipeline, allowing each processor to clean up resources appropriately.

## Termination Methods

Pipecat provides two primary approaches for pipeline termination, each designed for different scenarios:

### 1. Graceful Termination

Graceful termination allows the bot to complete its current processing before shutting down. This is ideal when you want the bot to properly end a conversation. For example, after completing a specific task or reaching a natural conclusion.

**When to use:**

* Natural conversation endings
* Task completion scenarios
* When the bot should say goodbye

**Implementation options:**

Push an `EndFrame` from outside your pipeline:

```python  theme={null}
# From outside the pipeline
from pipecat.frames.frames import EndFrame, TTSSpeakFrame

await task.queue_frame(EndFrame())
```

Push an `EndTaskFrame` upstream from inside your pipeline:

```python  theme={null}
# From inside a function call
from pipecat.frames.frames import EndTaskFrame, TTSSpeakFrame
from pipecat.processors.frame_processor import FrameDirection

async def end_conversation(params: FunctionCallParams):
    await params.llm.push_frame(TTSSpeakFrame("Have a nice day!"))

    # Signal that the task should end after processing this frame
    await params.llm.push_frame(EndTaskFrame(), FrameDirection.UPSTREAM)
```

**How graceful termination works:**

1. `EndFrame` is queued and processes after any pending frames (like goodbye messages)
2. All processors shutdown when they receive the `EndFrame`
3. Once the `EndFrame` reaches the end of the pipeline, shutdown is complete
4. Resources are cleaned up and the process terminates

<Tip>
  Graceful termination allows your bot to say goodbye and complete any final
  actions before terminating.
</Tip>

### 2. Immediate Termination

Immediate termination cancels the pipeline without waiting for pending frames to complete. This is appropriate when the user is no longer active in the conversation.

**When to use:**

* User disconnections (browser closed, call ended)
* Error conditions requiring immediate shutdown
* When completing the conversation is no longer necessary

**Implementation:**

Use event handlers to detect disconnections and trigger cancellation:

```python  theme={null}
@transport.event_handler("on_client_disconnected")
async def on_client_disconnected(transport, client):
    logger.info("Client disconnected - terminating pipeline")
    await task.cancel()
```

**How immediate termination works:**

1. An event triggers the cancellation (like client disconnection)
2. `task.cancel()` pushes a `CancelFrame` downstream from the PipelineTask
3. `CancelFrame`s are `SystemFrame`s and bypass queues for immediate processing
4. Processors handle the `CancelFrame` and shut down immediately
5. Any pending frames are discarded during shutdown

<Warning>
  Immediate termination will discard any pending frames in the pipeline. Use
  this approach when completing the conversation is no longer necessary.
</Warning>

## Automatic Termination

### Pipeline Idle Detection

Pipecat includes automatic idle detection to prevent hanging pipelines. This feature monitors activity and can automatically cancel tasks when no meaningful bot interactions occur for an extended period.

**How it works:**

* Monitors pipeline activity for meaningful bot interactions
* Automatically triggers termination after configured idle timeout
* Serves as a safety net for anomalous behavior or forgotten sessions

**Configuration:**

```python  theme={null}
task = PipelineTask(
    pipeline,
    # Configure idle detection timeout
    cancel_on_idle_timeout=True, # Default is True
    idle_timeout_seconds=600,  # Default is 300 seconds
    idle_timeout_frames=(BotSpeakingFrame,), # Default is (BotSpeakingFrame, LLMFullResponseEndFrame)
)
```

You can further configure the idle detection behavior. To learn more, refer to the Pipeline Idle Detection documentation:

<Card title="Pipeline Idle Detection" icon="clock" href="/server/pipeline/pipeline-idle-detection">
  Learn how to configure and customize idle detection for your use case
</Card>

<Note>
  Pipeline Idle Detection is enabled by default and helps prevent resources from
  being wasted on inactive conversations.
</Note>

## Implementation Patterns

### Event-Driven Termination

Connect termination to transport events for automatic cleanup:

```python  theme={null}
@transport.event_handler("on_client_connected")
async def on_client_connected(transport, client):
    logger.info("Client connected - starting conversation")
    await task.queue_frames([LLMRunFrame()])

@transport.event_handler("on_client_disconnected")
async def on_client_disconnected(transport, client):
    logger.info("Client disconnected - immediate termination")
    await task.cancel()

# Run the pipeline
runner = PipelineRunner(handle_sigint=False)
await runner.run(task)
```

### Conditional Termination

Use function calling or other logic to determine when conversations should end:

```python  theme={null}
async def check_conversation_complete(params: FunctionCallParams):
    # Your logic to determine if conversation should end
    conversation_complete = await evaluate_completion_criteria()

    if conversation_complete:
        await params.llm.push_frame(TTSSpeakFrame("Thank you for using our service!"))
        await params.llm.push_frame(EndTaskFrame(), FrameDirection.UPSTREAM)

    await params.result_callback({"status": "complete" if conversation_complete else "continuing"})
```

### Error Handling

Ensure pipelines can terminate properly even when exceptions occur:

```python  theme={null}
try:
    runner = PipelineRunner(handle_sigint=False)
    await runner.run(task)
except Exception as e:
    logger.error(f"Pipeline error: {e}")
    # Ensure cleanup happens even on errors
    await task.cancel()
```

## Troubleshooting

If your pipeline isn't shutting down properly, check these common issues:

### Custom Processors Not Propagating Frames

**Problem:** Custom processors that don't call `push_frame()` can block termination frames from reaching the end of the pipeline.

**Solution:** Ensure your custom processors propagate all frames downstream, including `EndFrame` and `CancelFrame`:

```python  theme={null}
async def process_frame(self, frame: Frame, direction: FrameDirection):
    await super().process_frame(frame, direction)

    # Your custom processing logic here

    # Always push frames downstream (including termination frames)
    await self.push_frame(frame, direction)
```

### Incorrect Termination Frame Direction

**Problem:** Pushing `EndFrame` or `CancelFrame` from the middle of the pipeline may not reach the pipeline source properly.

**Solution:** Use the appropriate frame type and direction:

```python  theme={null}
# From inside the pipeline, push upstream to reach the source
await self.push_frame(EndTaskFrame(), FrameDirection.UPSTREAM)
await self.push_frame(CancelTaskFrame(), FrameDirection.UPSTREAM)

# The pipeline source will then convert these to proper termination frames
# and push them downstream through the entire pipeline
```

<Note>
  The pipeline source automatically converts `EndTaskFrame` to `EndFrame` and
  `CancelTaskFrame` to `CancelFrame` when pushing downstream, ensuring proper
  termination handling throughout the pipeline.
</Note>

## Key Takeaways

* **Frame-based termination** - shutdown uses the same frame system as processing
* **Choose the right method** - graceful for natural endings, immediate for disconnections
* **Event handlers enable automatic termination** - respond to user disconnections cleanly
* **Idle detection provides safety net** - prevents hanging processes and resource waste
* **SystemFrames bypass queues** - CancelFrames process immediately for fast shutdown
* **Resource cleanup is automatic** - proper termination ensures clean resource disposal

## What's Next

You now understand how to build, run, and properly terminate voice AI pipelines! Next, let's explore advanced development tools that can accelerate your Pipecat development process.

<Card title="Next Steps" icon="arrow-right" href="/guides/learn/whats-next">
  You've completed the learning guide! Continue to see recommended next steps.
</Card>

---

# Source: https://docs.pipecat.ai/guides/learn/pipeline.md

> ## Documentation Index
> Fetch the complete documentation index at: https://docs.pipecat.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Pipeline & Frame Processing

> Learn how Pipecat's pipeline architecture orchestrates frame processing for voice AI applications

The **Pipeline** is the core orchestration component in Pipecat that connects frame processors together, creating a structured path for data to flow through your voice AI application.

## Basic Pipeline Structure

A Pipeline takes a list of frame processors and connects them in sequence. Here's a simple voice AI pipeline that matches the voice AI agent architecture we discussed earlier:

```python  theme={null}
pipeline = Pipeline([
    transport.input(),              # Receives user audio
    stt,                            # Speech-to-text conversion
    context_aggregator.user(),      # Collect user responses
    llm,                            # Language model processing
    tts,                            # Text-to-speech conversion
    transport.output(),             # Sends audio to user
    context_aggregator.assistant(), # Collect assistant responses
])
```

## Understanding Frames and Frame Processing

Before diving deeper into pipelines, it's important to understand how data moves through them using **frames** and **frame processors**.

### What Are Frames?

**Frames** are data containers that carry information through your pipeline. Think of them as packages on a conveyor belt. Each frame contains a specific type of data that processors can examine and act upon.

Frames automatically receive unique identifiers and names (like `TranscriptionFrame#1`) that help with debugging and tracking data flow through your pipeline.

Common frame types include:

* **Audio frames**: Raw audio data from users or generated by TTS
* **Text frames**: Transcriptions, LLM responses, or other text content
* **Image frames**: Visual data for multimodal applications
* **System frames**: Control signals for pipeline management
* **Context frames**: Conversation history and state information

Frames flow through your pipeline from processor to processor, carrying the data your voice AI application needs to operate.

### What Are Frame Processors?

**Frame Processors** are the workers in your pipeline. Each processor has a specific job - like converting speech to text, generating responses, or playing audio. They:

* **Receive frames** from the previous processor in the pipeline
* **Process the data** (transcribe audio, generate text, etc.)
* **Create new frames** with their output
* **Pass frames along** to the next processor

Frame processors are modular and reusable. You can swap out different STT services or LLM providers without changing the rest of your pipeline.

### Frame Types

Frames in Pipecat have different base classes that determine how they're processed:

```python  theme={null}
@dataclass
class SystemFrame(Frame):
    """System frames are processed immediately, bypassing queues."""
    pass

@dataclass
class DataFrame(Frame):
    """Data frames are queued and processed in order."""
    pass

@dataclass
class ControlFrame(Frame):
    """Control frames are queued and processed in order."""
    pass
```

**Key differences:**

* **SystemFrames**: Processed immediately (interruptions, pipeline control, user input)
* **DataFrames & ControlFrames**: Queued and processed in order (audio output, text, images)

**Examples by type:**

```python  theme={null}
# SystemFrames (processed immediately)
InputAudioRawFrame         # User audio input
UserStartedSpeakingFrame   # Speech detection events
StartInterruptionFrame     # Interruption control
ErrorFrame                 # Error notifications

# DataFrames (queued and ordered)
OutputAudioRawFrame        # Audio for playback
TextFrame                  # Text content
TranscriptionFrame         # Speech-to-text results
LLMTextFrame               # LLM responses
AggregatedTextFrame        # Text aggregated into a describable unit
TTSTextFrame               # Text-to-speech text output

# ControlFrames (queued and ordered)
EndFrame                   # Pipeline shutdown
TTSStartedFrame            # TTS response boundaries
LLMFullResponseStartFrame  # LLM response boundaries
```

### Frame Processing Order

**Frames are processed in guaranteed order**, even across ParallelPipelines. This enables reliable sequencing. For example, you can push two frames in order and the order will be respected. Additionally, the corresponding processing will finish before allowing the next frame to be processed. Let's look at an example where we push two frames—`TTSSpeakFrame` and `EndFrame`—in order to say goodbye then end the pipeline:

```python  theme={null}
from pipecat.frames.frames import EndFrame, TTSSpeakFrame

# These will execute in order: speak first, then end pipeline
await task.queue_frames([
    TTSSpeakFrame("Goodbye!"),
    EndFrame()
])
```

### How Frame Processors Work

Every frame processor follows the same pattern with two key methods:

```python  theme={null}
class TranscriptionLogger(FrameProcessor):
    async def process_frame(self, frame: Frame, direction: FrameDirection):
        # Always call parent first
        await super().process_frame(frame, direction)

        # Handle specific frame types
        if isinstance(frame, TranscriptionFrame):
            print(f"Transcription: {frame.text}")

        # Push frame to next processor
        await self.push_frame(frame, direction)
```

**Key methods:**

* **`process_frame()`**: Inspect and handle incoming frames
* **`push_frame()`**: Send frames upstream or downstream

<Card title="Custom Frame Processors" icon="code" href="/guides/fundamentals/custom-frame-processor">
  Learn how to build your own frame processors
</Card>

## How Data Flows Through Pipelines

Understanding data flow is crucial for building effective pipelines:

### Frame Processing Order

**Order matters**: Processors must be arranged so that each receives the frame types it needs:

1. `transport.input()` creates `InputAudioRawFrame`s from user audio
2. `stt` receives audio frames and outputs `TranscriptionFrame`s
3. `llm` processes text and generates `LLMTextFrame`s
4. `tts` converts text frames to `TTSAudioRawFrame`s, `AggregatedTextFrame`s, and `TTSTextFrame`s
5. `transport.output()` creates `OutputAudioRawFrame`s and sends audio back to user

* Note: An `LLMTextProcessor` can sit between the `llm` and `tts` to pre-aggregate `LLMTextFrame`s into `AggregatedTextFrame`s. This simply moves the aggregation step
  out of the TTS.

### Frame Propagation

**Processors always push frames**: Processors don't consume frames, they pass them along:

```python  theme={null}
# This pipeline allows multiple processors to use the same audio
pipeline = Pipeline([
    transport.input(),          # Creates InputAudioRawFrame
    stt,                        # Uses audio → creates TranscriptionFrame
    # ...                       # Other processors can use the same audio
    tts,                        # Uses various text frames → creates TTSAudioRawFrame
    transport.output(),         # Uses TTSAudioRawFrame → sends audio to user
    audio_buffer_processor,     # Also uses the same user and assistant audio for recording
])
```

This design allows multiple processors to operate on the same data stream without interfering with each other.

### Parallel Processing Patterns

Use `ParallelPipeline` to create branches where each branch receives all upstream frames. Frames are collected and pushed individually from each branch:

```python  theme={null}
pipeline = Pipeline([
    transport.input(),
    stt,
    context_aggregator.user(),
    llm,
    ParallelPipeline([
        # English branch
        [FunctionFilter(english_filter), english_tts],
        # Spanish branch
        [FunctionFilter(spanish_filter), spanish_tts],
    ]),
    transport.output(),
    context_aggregator.assistant(),
])
```

In this example:

* Both TTS branches receive all LLM output
* Each branch can filter and process frames independently
* Results from both branches flow to `transport.output()`

ParallelPipelines are traditionally paired with filters or gates to control which frames go where, allowing for complex conditional logic.

### Frame Queuing and Processing

Frame processors have internal queues that ensure ordered processing:

* **SystemFrames bypass queues** for immediate processing (interruptions, errors)
* **DataFrames and ControlFrames are queued** and processed in order
* **Queuing is managed automatically** by the pipeline infrastructure
* **Order is guaranteed** even across complex pipeline structures

<Note>
  Learn more about frame flow patterns in the [Custom Frame Processor
  Guide](/guides/fundamentals/custom-frame-processor).
</Note>

## Pipeline Execution

### Creating a Pipeline Task

Wrap your pipeline in a `PipelineTask` to configure execution parameters:

```python  theme={null}
task = PipelineTask(
    pipeline,
    params=PipelineParams(
        enable_metrics=True,
        enable_usage_metrics=True,
        audio_in_sample_rate=8000,
        audio_out_sample_rate=8000,
    ),
    observers=[RTVIObserver(rtvi)],
)
```

### Running the Pipeline

Use `PipelineRunner` to execute your pipeline task:

```python  theme={null}
# Create the runner
runner = PipelineRunner(handle_sigint=False)

# Run the pipeline
await runner.run(task)
```

**PipelineRunner features:**

* **Lifecycle Management**: Handles startup, execution, and cleanup
* **Signal Handling**: Responds to SIGINT/SIGTERM for graceful shutdown (when `handle_sigint=True`)
* **Resource Cleanup**: Ensures proper disposal of resources when tasks complete

**Common PipelineRunner options:**

```python  theme={null}
# Handle system signals automatically
# Useful when running a file directly
runner = PipelineRunner(handle_sigint=True)

# Custom signal handling
# Useful when the PipelineRunner is managed by
# another system like the development runner
runner = PipelineRunner(handle_sigint=False)
```

### Pipeline Lifecycle

The pipeline runs continuously, processing frames as they arrive, until one of these conditions occurs:

1. **Normal Completion**: Bot code returns (session ends)
2. **Signal Termination**: SIGINT or SIGTERM received (if `handle_sigint=True`)
3. **Task Cancellation**: Explicit call to `task.cancel()`
4. **Error Condition**: Unhandled exception in pipeline

**Common lifecycle patterns:**

```python  theme={null}
# Graceful shutdown on client disconnect
@transport.event_handler("on_client_disconnected")
async def on_client_disconnected(transport, client):
    logger.info("Client disconnected - stopping pipeline")
    await task.cancel()

# Await the pipeline task
await runner.run(task)
```

**Pipeline states during execution:**

* **Starting**: Initializing processors and connections
* **Running**: Actively processing frames and handling events
* **Stopping**: Gracefully shutting down processors
* **Stopped**: All resources cleaned up

## Configuring Pipeline Behavior

Beyond the basic pipeline structure, you can fine-tune behavior through PipelineParams and add monitoring capabilities:

### Pipeline Parameters

`PipelineParams` lets you configure how your entire pipeline operates:

```python  theme={null}
params = PipelineParams(
    # Audio configuration (affects all audio processors)
    audio_in_sample_rate=16000,
    audio_out_sample_rate=24000,

    # Performance monitoring
    enable_metrics=True,
    enable_usage_metrics=True,

    # Debugging and development
    send_initial_empty_metrics=False,
    report_only_initial_ttfb=True,
)
```

### Monitoring and Observability

Add observers to monitor pipeline events and track custom application metrics:

```python  theme={null}
task = PipelineTask(
    pipeline,
    params=params,
    observers=[
        RTVIObserver(rtvi),           # RTVI protocol events
        CustomMetricsObserver(),      # Your custom metrics
    ],
)
```

These configuration options enable you to optimize performance, debug issues, and monitor your pipeline in production:

<CardGroup cols={3}>
  <Card title="Pipeline Parameters" icon="screwdriver-wrench" href="/server/pipeline/pipeline-params">
    Configure execution parameters, audio settings, and debugging options
  </Card>

  <Card title="Metrics Guide" icon="chart-line" href="/guides/fundamentals/metrics">
    Monitor pipeline performance, latency, and API usage in real-time
  </Card>

  <Card title="Observer Pattern" icon="eye" href="/server/utilities/observers/observer-pattern">
    Monitor pipeline events, frame processing, and custom application events
  </Card>
</CardGroup>

## Key Takeaways

* **Order matters** - arrange processors so each gets the frames it needs
* **Processors push frames** - processors pass frames downstream, not consume them
* **Frame types determine processing** - SystemFrames bypass queues, others are ordered
* **Queuing ensures reliability** - frames are processed in guaranteed order
* **Parallel processing** enables conditional logic and multi-modal handling
* **PipelineTask** configures execution parameters and monitoring
* **PipelineRunner** manages the complete pipeline lifecycle
* **PipelineParams** control pipeline-wide behavior like audio settings and metrics

## What's Next

Now that you understand how pipelines orchestrate processing, let's explore the different transport options that connect your pipeline to users.

<Card title="Transports" icon="arrow-right" href="/guides/learn/transports">
  Learn about the different ways users can connect to your voice AI pipeline
</Card>

---

# Source: https://docs.pipecat.ai/server/services/tts/piper.md

> ## Documentation Index
> Fetch the complete documentation index at: https://docs.pipecat.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Piper

> Text-to-speech service implementation using the Piper TTS server

## Overview

`PiperTTSService` provides high-quality neural text-to-speech synthesis through a self-hosted HTTP server. The service offers complete privacy and control with no external API dependencies, making it ideal for on-premise deployments and applications requiring data sovereignty.

<CardGroup cols={2}>
  <Card title="Piper TTS API Reference" icon="code" href="https://reference-server.pipecat.ai/en/latest/api/pipecat.services.piper.tts.html">
    Pipecat's API methods for Piper TTS integration
  </Card>

  <Card title="Example Implementation" icon="play" href="https://github.com/pipecat-ai/pipecat/tree/main/examples">
    Browse examples using Piper TTS
  </Card>

  <Card title="Piper Documentation" icon="book" href="https://github.com/OHF-Voice/piper1-gpl">
    Official Piper TTS documentation and setup
  </Card>

  <Card title="HTTP Server Setup" icon="server" href="https://github.com/OHF-Voice/piper1-gpl/blob/main/docs/API_HTTP.md">
    Configure Piper HTTP server for Pipecat
  </Card>
</CardGroup>

## Installation

To use Piper services, no additional Pipecat dependencies are required:

```bash  theme={null}
pip install "pipecat-ai"  # Base installation is sufficient
```

## Prerequisites

### Piper Server Setup

Before using PiperTTSService, you need:

1. **Piper TTS Server**: Set up a running Piper TTS server following the [HTTP server documentation](https://github.com/OHF-Voice/piper1-gpl/blob/main/docs/API_HTTP.md)
2. **Voice Models**: Download and configure voice models for your target languages
3. **Server Configuration**: Configure server endpoint and voice selection

### Required Configuration

* **Server URL**: Configure the Piper server endpoint in your service initialization
* **Voice Models**: Ensure required voice models are available on the server

<Tip>
  Piper runs entirely locally, providing complete privacy and eliminating API
  key requirements.
</Tip>

---

# Source: https://docs.pipecat.ai/server/services/tts/playht.md

> ## Documentation Index
> Fetch the complete documentation index at: https://docs.pipecat.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# PlayHT

> Text-to-speech services using PlayHT's WebSocket and HTTP APIs

## Overview

PlayHT provides high-quality text-to-speech synthesis with two service implementations: `PlayHTTTSService` (WebSocket-based) for real-time streaming with ultra-low latency, and `PlayHTHttpTTSService` (HTTP-based) for simpler synthesis. `PlayHTTTSService` is recommended for interactive applications requiring low latency and advanced interruption handling.

<CardGroup cols={2}>
  <Card title="PlayHT TTS API Reference" icon="code" href="https://reference-server.pipecat.ai/en/latest/api/pipecat.services.playht.tts.html">
    Pipecat's API methods for PlayHT TTS integration
  </Card>

  <Card title="Example Implementation" icon="play" href="https://github.com/pipecat-ai/pipecat/blob/main/examples/foundational/07e-interruptible-playht.py">
    Complete example with voice cloning
  </Card>

  <Card title="PlayHT Documentation" icon="book" href="https://docs.play.ht/reference/playht-tts-websocket-api">
    Official PlayHT WebSocket API documentation
  </Card>

  <Card title="Voice Cloning" icon="microphone" href="https://play.ht/app">
    Create and manage custom voices
  </Card>
</CardGroup>

## Installation

To use PlayHT services, install the required dependencies:

```bash  theme={null}
pip install "pipecat-ai[playht]"
```

## Prerequisites

### PlayHT Account Setup

Before using PlayHT TTS services, you need:

1. **PlayHT Account**: Sign up at [PlayHT Dashboard](https://play.ht/app)
2. **API Credentials**: Get your User ID and API Key from the dashboard
3. **Voice Selection**: Choose from standard voices or create custom cloned voices

### Required Environment Variables

* `PLAY_HT_USER_ID`: Your PlayHT user ID
* `PLAY_HT_API_KEY`: Your PlayHT API key for authentication

---

# Source: https://docs.pipecat.ai/deployment/pipecat-cloud/guides/telephony/plivo-websocket.md

> ## Documentation Index
> Fetch the complete documentation index at: https://docs.pipecat.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Plivo Websocket Transport

> Using Plivo's WebSocket Transport for your Pipecat Cloud agents

<Note>
  This guide covers Pipecat Cloud-specific configuration for Plivo WebSocket
  integration. For a complete guide including dial-in, dial-out, and advanced
  features, see the [Plivo WebSocket Integration
  guide](/guides/telephony/plivo-websockets).
</Note>

Native support for Plivo's WebSocket Transport with Pipecat Cloud allows you to connect your AI agents with Plivo's voice infrastructure. This integration enables your Pipecat bots to handle real phone calls using Plivo's WebSocket streaming.

## How It Works

Pipecat Cloud implements Plivo's bidirectional [Media Streaming protocol](https://www.plivo.com/docs/voice/guides/stream-audio/). While audio streams flow through WebSockets, the call session is controlled by XML responses that tell Plivo how to handle each call.

When Pipecat Cloud receives an incoming WebSocket connection from Plivo, it processes the `start` message to initialize a new bot instance. All WebSocket messages are forwarded to your bot, including call information such as the caller's number and call UUID. This allows your bot to leverage Plivo's Voice API for advanced call control - such as recording conversations, transferring to human agents, or implementing complex call flows.

## Prerequisites

Before setting up this integration, ensure you have:

* A Plivo account with voice capabilities
* A Pipecat Cloud account with a Plivo WebSocket-compatible bot
* A web server to host XML responses (we'll show you how to set this up)

A ready-to-build example of a Plivo websockets bot with complete source code is available in the [pipecat-examples repo](https://github.com/pipecat-ai/pipecat-examples/tree/main/plivo-chatbot).

## Pipecat Cloud Configuration

### 1. Get Your Organization Name

Retrieve your Pipecat Cloud organization name using the Pipecat CLI:

```bash  theme={null}
$ pipecat cloud organizations list
```

This command will output a list of organizations associated with your account. For example:

```bash  theme={null}
Organization        Name
──────────────────────────────────────
Default Workspace   three-random-words-randomnumber (active)
```

### 2. Set Up an XML Server

Unlike Twilio's TwiML Bins, Plivo requires a web server to host your XML responses. Create a file called `server.py`:

```python  theme={null}
from fastapi import FastAPI, Query, HTTPException
from starlette.responses import Response
import uvicorn

app = FastAPI(title="Plivo XML Server")

@app.get("/plivo-xml")
async def plivo_xml(
    agent: str = Query(..., description="Agent name"),
    org: str = Query(..., description="Organization name"),
):
    """
    Returns XML for Plivo to start WebSocket streaming to Pipecat Cloud
    Example: /plivo-xml?agent=my-bot&org=my-org-123
    """
    if not agent or not org:
        raise HTTPException(status_code=400, detail="Both 'agent' and 'org' parameters are required")

    xml = f"""<?xml version="1.0" encoding="UTF-8"?>
<Response>
  <Stream bidirectional="true" keepCallAlive="true" contentType="audio/x-mulaw;rate=8000">
    wss://api.pipecat.daily.co/ws/plivo?serviceHost={agent}.{org}
  </Stream>
</Response>"""

    return Response(content=xml, media_type="application/xml")

if __name__ == "__main__":
    uvicorn.run(app, host="0.0.0.0", port=7860)
```

<Note>
  **Using Regional Endpoints**

  If you deployed your agent to a specific region, use the regional WebSocket endpoint:

  `wss://{region}.api.pipecat.daily.co/ws/plivo?serviceHost=AGENT_NAME.ORGANIZATION_NAME`

  For example, for Europe: `wss://eu-central.api.pipecat.daily.co/ws/plivo?serviceHost=my-agent.my-org`

  Learn more about [regional endpoints](/deployment/pipecat-cloud/guides/regions#regional-websocket-endpoints).
</Note>

### 3. Deploy Your XML Server

Install dependencies and run your server:

```bash  theme={null}
pip install fastapi uvicorn
python server.py
```

For local testing, use ngrok to make your server publicly accessible:

```bash  theme={null}
ngrok http 7860
```

This will give you a public URL like `https://abc123.ngrok.io`.

<Tip>
  Use the `--subdomain` flag with ngrok to keep your URL consistent across
  restarts.
</Tip>

For production, deploy your XML server to a reliable hosting platform.

### 4. Configure Plivo

[Purchase a phone number from Plivo](https://www.plivo.com/docs/numbers/quickstart/) if you haven't already, then:

1. Navigate to Voice → XML in your Plivo dashboard
2. Select Add New Application
3. Enter a name for your application (e.g., "Pipecat WebSocket")
4. Set the **Answer URL** to your server URL with agent and organization parameters:
   ```
   https://your-server.com/plivo-xml?agent=AGENT_NAME&org=ORGANIZATION_NAME
   ```
   * Replace `AGENT_NAME` with your deployed bot's name (e.g., `my-first-agent`)
   * Replace `ORGANIZATION_NAME` with your organization name from step 1 (e.g., `three-random-words-randomnumber`)
5. Set **HTTP Method** to `GET`
6. Click Create Application

### 5. Assign to Your Phone Number

1. Navigate to Phone Numbers and select your phone number
2. Select XML Application as the Application Type
3. Select your XML Application from the Plivo Application dropdown
4. Click Update to apply your changes

## Testing Your Integration

To test your integration, simply dial your Plivo phone number from any phone. The call will connect to your Pipecat Cloud bot, which will respond according to your bot's configuration.

## Next Steps

For complete implementation details including dial-out and advanced call control features, see the [Plivo WebSocket Integration guide](/guides/telephony/plivo-websockets).

---

# Source: https://docs.pipecat.ai/guides/telephony/plivo-websockets.md

> ## Documentation Index
> Fetch the complete documentation index at: https://docs.pipecat.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Plivo WebSocket Integration

> Complete guide to using Plivo Media Streaming with Pipecat for dial-in and dial-out functionality

## Things you'll need

* An active [Plivo](https://www.plivo.com) account with API credentials
* One or more Plivo provisioned phone numbers
* A public-facing server or tunneling service like [ngrok](https://ngrok.com/)
* API keys for speech-to-text, text-to-speech, and LLM services

<CardGroup cols={2}>
  <Card title="Plivo Dial-in Example" icon="phone-arrow-down-left" href="https://github.com/pipecat-ai/pipecat-examples/tree/main/plivo-chatbot/inbound">
    Complete dial-in implementation using Plivo Media Streaming over WebSocket
  </Card>

  <Card title="Plivo Dial-out Example" icon="phone-arrow-up-right" href="https://github.com/pipecat-ai/pipecat-examples/tree/main/plivo-chatbot/outbound">
    Outbound calling using Plivo Media Streaming with WebSocket transport
  </Card>
</CardGroup>

## Phone Number Setup

You'll need Plivo phone numbers for both dial-in and dial-out functionality:

* Visit [plivo.com](https://www.plivo.com) and purchase phone numbers
* Ensure your numbers support Voice capabilities
* Configure XML applications for dial-in numbers (covered below)

## Environment Setup

Configure your environment variables for Plivo and AI services:

```shell .env theme={null}
PLIVO_AUTH_ID=...
PLIVO_AUTH_TOKEN=...
OPENAI_API_KEY=...
DEEPGRAM_API_KEY=...
CARTESIA_API_KEY=...
```

## Dial-in

Dial-in allows users to call your Plivo number and connect to your Pipecat bot via WebSocket Media Streaming.

### How It Works

Here's the sequence of events when someone calls your Plivo number:

1. **Plivo receives an incoming call** to your phone number
2. **Plivo calls your XML server** with call details
3. **Your server responds with XML** that establishes a WebSocket connection
4. **Plivo opens a WebSocket** to your server with real-time audio
5. **Your bot processes the audio** using the Pipecat pipeline
6. **The bot responds with audio** sent back to Plivo over WebSocket
7. **Plivo plays the audio** to the caller in real-time

### Set up your XML server

Your server handles Plivo webhooks and WebSocket connections for real-time audio streaming. When Plivo receives a call, your server:

1. **Receives the webhook** with call details (CallUUID, caller information)
2. **Returns XML** that establishes a WebSocket connection to your server
3. **Accepts the WebSocket** connection from Plivo
4. **Parses call data** from the WebSocket messages
5. **Runs the Pipecat bot** with the parsed call information

The server orchestrates the real-time audio streaming between Plivo's telephony infrastructure and your Pipecat bot pipeline.

### Custom Data with Query Parameters

You can pass custom data to your bot by adding query parameters to the WebSocket URL in your XML response. This works for both dial-in and dial-out scenarios:

```python server.py theme={null}
# Example: Adding custom parameters to the WebSocket URL
xml = f"""<?xml version="1.0" encoding="UTF-8"?>
<Response>
  <Stream bidirectional="true" keepCallAlive="true" contentType="audio/x-mulaw;rate=8000">
    wss://your-server.com/ws?user_id={user_id}&amp;session_type=support&amp;campaign_id=summer_sale
  </Stream>
</Response>"""
```

These parameters are accessible in your server code and can be used to customize your bot's behavior based on the specific call context.

<Card title="Complete Server Implementation" icon="code" href="https://github.com/pipecat-ai/pipecat-examples/blob/main/plivo-chatbot/inbound/server.py">
  See the full FastAPI server code with XML generation and WebSocket handling
</Card>

### Configure your Pipecat bot for dial-in

Your bot receives the WebSocket connection and parses the call data to extract call information. The key components are:

**WebSocket Parsing**: Pipecat's built-in parser extracts call data from Plivo's WebSocket messages:

```python bot.py theme={null}
from pipecat.runner.utils import parse_telephony_websocket

async def run_bot(websocket: WebSocket):
    # Parse Plivo WebSocket data
    transport_type, call_data = await parse_telephony_websocket(websocket)

    # Extract call information
    stream_id = call_data["stream_id"]
    call_id = call_data["call_id"]  # Plivo's CallUUID
```

**Transport Creation**: Configure the WebSocket transport with Plivo serialization:

```python bot.py theme={null}
# Create Plivo serializer with call details
serializer = PlivoFrameSerializer(
    stream_id=stream_id,
    call_id=call_id,
    auth_id=os.getenv("PLIVO_AUTH_ID"),
    auth_token=os.getenv("PLIVO_AUTH_TOKEN"),
)

# Configure WebSocket transport
transport = FastAPIWebsocketTransport(
    websocket=websocket,
    params=FastAPIWebsocketParams(
        audio_in_enabled=True,
        audio_out_enabled=True,
        add_wav_header=False,
        serializer=serializer,
    ),
)

# Your bot pipeline setup here...
```

<Card title="Complete Bot Implementation" icon="code" href="https://github.com/pipecat-ai/pipecat-examples/blob/main/plivo-chatbot/inbound/bot.py">
  See the full bot.py with WebSocket parsing, transport setup, and pipeline
  configuration
</Card>

### Set up Plivo XML application

Configure your Plivo phone number to use your XML application:

1. Go to the [Plivo Console](https://console.plivo.com)
2. Navigate to Voice → XML Applications
3. Create a new XML Application with your server's webhook URL
4. Assign the XML Application to your phone number

### Run the Example

For local development, use [ngrok](https://ngrok.com/) to expose your local server:

```shell  theme={null}
# Start your server
python server.py

# In another terminal, start ngrok
ngrok http 8000

# Use the ngrok URL in your Plivo XML Application
```

<Card title="Complete Setup Instructions" icon="book-open" href="https://github.com/pipecat-ai/pipecat-examples/tree/main/plivo-chatbot/inbound">
  See the full README with step-by-step setup, XML configuration, and testing
</Card>

## Dial-out

Dial-out allows your bot to initiate calls to phone numbers using Plivo's outbound calling capabilities with WebSocket Media Streaming.

### How It Works

Here's the sequence of events for dial-out calls:

1. **Your application triggers a dial-out** (via API call or user action)
2. **Server initiates a Plivo call** to the target phone number
3. **Plivo establishes the call** and opens a WebSocket connection
4. **Your bot joins the WebSocket** and sets up the pipeline
5. **The recipient answers** and is connected to your bot
6. **The bot handles the conversation** with real-time audio streaming

### Set up your server for dial-out

The dial-out server creates outbound calls and manages WebSocket connections. When you trigger a dial-out call, your server:

1. **Receives the dial-out request** with target phone number and parameters
2. **Creates a Plivo call** using the Voice API with XML that establishes WebSocket
3. **Accepts the WebSocket** connection from Plivo
4. **Parses call data** from the WebSocket messages
5. **Runs the Pipecat bot** with the call information

<Card title="Complete Server Implementation" icon="code" href="https://github.com/pipecat-ai/pipecat-examples/blob/main/plivo-chatbot/outbound/server.py">
  See the full FastAPI server code with outbound call creation and WebSocket
  handling
</Card>

### Passing Custom Data to Your Bot

For dial-out calls, you often need to pass custom data to personalize the conversation—such as user information, campaign details, or context from your application. The dial-out example supports passing a `body` object that becomes available to your bot via `runner_args.body`.

**Triggering a call with custom data:**

```bash  theme={null}
curl -X POST https://your-server.com/start \
  -H "Content-Type: application/json" \
  -d '{
    "phone_number": "+1234567890",
    "body": {
      "user": {
        "id": "user123",
        "firstName": "John",
        "lastName": "Doe",
        "accountType": "premium"
      },
      "campaign": "renewal_reminder",
      "context": "subscription expiring in 3 days"
    }
  }'
```

**How it works:**

1. The `/start` endpoint receives the request with `phone_number` and optional `body`
2. The `body` is base64-encoded and passed through the TeXML URL chain
3. When the WebSocket connects, the server decodes the body and passes it to your bot via `runner_args.body`
4. Your bot accesses the custom data to personalize the conversation

**Accessing custom data in your bot:**

```python bot.py theme={null}
async def bot(runner_args: RunnerArguments):
    body_data = runner_args.body or {}
    first_name = body_data.get("user", {}).get("firstName", "there")
    # Use first_name to personalize greetings, prompts, etc.
```

This approach works consistently in both local development and Pipecat Cloud production deployments.

### Configure your Pipecat bot for dial-out

The dial-out bot configuration is similar to dial-in, with Plivo automatically providing call information.

**Call Information**: Extract call details from Plivo's WebSocket messages:

```python bot.py theme={null}
# Parse WebSocket data (same as dial-in)
transport_type, call_data = await parse_telephony_websocket(websocket)

# Extract call information
stream_id = call_data["stream_id"]
call_id = call_data["call_id"]  # Plivo's CallUUID

# Access custom data from runner args
body_data = runner_args.body or {}
user_name = body_data.get("user", {}).get("firstName", "there")

# Customize bot behavior for outbound calls
greeting = f"Hi {user_name}! This is an automated call. How can I help you today?"
```

**Transport Configuration**: Same transport setup as dial-in:

```python bot.py theme={null}
# Create Plivo serializer
serializer = PlivoFrameSerializer(
    stream_id=stream_id,
    call_id=call_id,
    auth_id=os.getenv("PLIVO_AUTH_ID"),
    auth_token=os.getenv("PLIVO_AUTH_TOKEN"),
)

# Configure transport
transport = FastAPIWebsocketTransport(
    websocket=websocket,
    params=FastAPIWebsocketParams(
        audio_in_enabled=True,
        audio_out_enabled=True,
        add_wav_header=False,
        serializer=serializer,
    ),
)
```

<Card title="Complete Bot Implementation" icon="code" href="https://github.com/pipecat-ai/pipecat-examples/blob/main/plivo-chatbot/outbound/bot.py">
  See the full bot.py with outbound call handling and call information usage
</Card>

### Run the Example

To test dial-out functionality:

1. **Start your server**: Run your FastAPI server
2. **Trigger a call**: Make an API request to start an outbound call
3. **Answer your phone**: The bot will call the specified number
4. **Talk to your bot**: Have a conversation with your AI agent

<Card title="Complete Setup Instructions" icon="book-open" href="https://github.com/pipecat-ai/pipecat-examples/tree/main/plivo-chatbot/outbound">
  See the full README with step-by-step setup, API usage, and outbound call
  configuration
</Card>

## Key Features

### Audio Format and Sample Rate

Plivo Media Streaming uses 8kHz mono audio with μ-law encoding. Configure your pipeline accordingly:

```python  theme={null}
task = PipelineTask(
    pipeline,
    params=PipelineParams(
        audio_in_sample_rate=8000,
        audio_out_sample_rate=8000,
    ),
)
```

### Automatic Call Termination

When you provide Plivo credentials to the `PlivoFrameSerializer`, it automatically ends calls when your pipeline ends:

```python  theme={null}
serializer = PlivoFrameSerializer(
    stream_id=stream_id,
    call_id=call_id,
    auth_id=os.getenv("PLIVO_AUTH_ID"),
    auth_token=os.getenv("PLIVO_AUTH_TOKEN"),  # Enables auto-termination
)
```

## Deployment

### Local Development

Use [ngrok](https://ngrok.com/) to expose your local server for testing:

```shell  theme={null}
python server.py
ngrok http 8000
# Use the ngrok URL in your Plivo XML Application
```

### Pipecat Cloud Deployment

For production deployment without managing your own infrastructure, use Pipecat Cloud:

<Card title="Plivo WebSocket on Pipecat Cloud" icon="cloud" href="/deployment/pipecat-cloud/guides/telephony/plivo-websocket">
  Deploy Plivo WebSocket bots with automatic scaling and managed infrastructure
</Card>

### Self-Hosted Production Deployment

For fully self-hosted production deployment, ensure your servers are:

* Publicly accessible with HTTPS
* Able to handle concurrent WebSocket connections
* Properly configured with Plivo API credentials
* Implementing proper error handling and logging

## Next Steps

* Explore the [complete examples](https://github.com/pipecat-ai/pipecat-examples/tree/main/plivo-chatbot) for full implementations
* Learn about [Daily + Twilio SIP integration](./twilio-daily-sip) for advanced telephony scenarios
* Check out [Daily PSTN integration](./daily-pstn) for direct phone number provisioning

---

# Source: https://docs.pipecat.ai/server/services/serializers/plivo.md

> ## Documentation Index
> Fetch the complete documentation index at: https://docs.pipecat.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# PlivoFrameSerializer

> Serializer for Plivo Audio Streaming WebSocket protocol

## Overview

`PlivoFrameSerializer` enables integration with Plivo's Audio Streaming WebSocket protocol, allowing your Pipecat application to handle phone calls via Plivo's voice services with bidirectional audio conversion, DTMF event handling, and automatic call termination.

<CardGroup cols={2}>
  <Card title="Plivo Serializer API Reference" icon="code" href="https://reference-server.pipecat.ai/en/latest/api/pipecat.serializers.plivo.html">
    Pipecat's API methods for Plivo Audio Streaming integration
  </Card>

  <Card title="Example Implementation" icon="play" href="https://github.com/pipecat-ai/pipecat-examples/tree/main/plivo-chatbot">
    Complete telephony examples with Plivo
  </Card>

  <Card title="Plivo Documentation" icon="book" href="https://www.plivo.com/docs/voice/api/stream/">
    Official Plivo Audio Streaming documentation
  </Card>

  <Card title="Plivo Console" icon="microphone" href="https://console.plivo.com/">
    Manage phone numbers and streaming configuration
  </Card>
</CardGroup>

## Installation

The `PlivoFrameSerializer` does not require any additional dependencies beyond the core Pipecat library:

```bash  theme={null}
pip install "pipecat-ai"
```

## Prerequisites

### Plivo Account Setup

Before using PlivoFrameSerializer, you need:

1. **Plivo Account**: Sign up at [Plivo Console](https://console.plivo.com/)
2. **Phone Number**: Purchase a Plivo phone number with voice capabilities
3. **Audio Streaming**: Configure your phone number for WebSocket streaming
4. **Webhook Configuration**: Set up webhook endpoints for call handling

### Required Environment Variables

* `PLIVO_AUTH_ID`: Your Plivo Auth ID for authentication
* `PLIVO_AUTH_TOKEN`: Your Plivo Auth Token for API operations

### Required Configuration

* **Stream ID**: Provided by Plivo during Audio Streaming connection
* **Call ID**: Required for automatic call termination (optional)

### Key Features

* **Bidirectional Audio**: Convert between Pipecat and Plivo audio formats
* **DTMF Handling**: Process touch-tone events from callers
* **Auto Hang-up**: Terminate calls via Plivo's REST API
* **μ-law Encoding**: Handle Plivo's standard audio encoding format

---

# Source: https://docs.pipecat.ai/server/utilities/frame/producer-consumer.md

> ## Documentation Index
> Fetch the complete documentation index at: https://docs.pipecat.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Producer & Consumer Processors

> Route frames between different parts of a pipeline, allowing selective frame sharing across parallel branches or within complex pipelines

## Overview

The Producer and Consumer processors work as a pair to route frames between different parts of a pipeline, particularly useful when working with [`ParallelPipeline`](/server/pipeline/parallel-pipeline). They allow you to selectively capture frames from one pipeline branch and inject them into another.

## ProducerProcessor

`ProducerProcessor` examines frames flowing through the pipeline, applies a filter to decide which frames to share, and optionally transforms these frames before sending them to connected consumers.

### Constructor Parameters

<ParamField path="filter" type="Callable[[Frame], Awaitable[bool]]" required>
  An async function that determines which frames should be sent to consumers.
  Should return `True` for frames to be shared.
</ParamField>

<ParamField path="transformer" type="Callable[[Frame], Awaitable[Frame]]" default="identity_transformer">
  Optional async function that transforms frames before sending to consumers. By
  default, passes frames unchanged.
</ParamField>

<ParamField path="passthrough" type="bool" default="True">
  When `True`, passes all frames through the normal pipeline flow. When `False`,
  only passes through frames that don't match the filter.
</ParamField>

## ConsumerProcessor

`ConsumerProcessor` receives frames from a `ProducerProcessor` and injects them into its pipeline branch.

### Constructor Parameters

<ParamField path="producer" type="ProducerProcessor" required>
  The producer processor that will send frames to this consumer.
</ParamField>

<ParamField path="transformer" type="Callable[[Frame], Awaitable[Frame]]" default="identity_transformer">
  Optional async function that transforms frames before injecting them into the
  pipeline.
</ParamField>

<ParamField path="direction" type="FrameDirection" default="FrameDirection.DOWNSTREAM">
  The direction in which to push received frames. Usually `DOWNSTREAM` to send
  frames forward in the pipeline.
</ParamField>

## Usage Examples

### Basic Usage: Moving TTS Audio Between Branches

```python  theme={null}
# Create a producer that captures TTS audio frames
async def is_tts_audio(frame: Frame) -> bool:
    return isinstance(frame, TTSAudioRawFrame)

# Define an async transformer function
async def tts_to_input_audio_transformer(frame: Frame) -> Frame:
    if isinstance(frame, TTSAudioRawFrame):
        # Convert TTS audio to input audio format
        return InputAudioRawFrame(
            audio=frame.audio,
            sample_rate=frame.sample_rate,
            num_channels=frame.num_channels
        )
    return frame

producer = ProducerProcessor(
    filter=is_tts_audio,
    transformer=tts_to_input_audio_transformer
    passthrough=True  # Keep these frames in original pipeline
)

# Create a consumer to receive the frames
consumer = ConsumerProcessor(
    producer=producer,
    direction=FrameDirection.DOWNSTREAM
)

# Use in a ParallelPipeline
pipeline = Pipeline([
    transport.input(),
    ParallelPipeline(
        # Branch 1: LLM for bot responses
        [
            llm,
            tts,
            producer,  # Capture TTS audio here
        ],
        # Branch 2: Audio processing branch
        [
            consumer,  # Receive TTS audio here
            llm, # Speech-to-Speech LLM (audio in)
        ]
    ),
    transport.output(),
])
```

---

# Source: https://docs.pipecat.ai/deployment/pipecat-cloud/rest-reference/endpoint/properties-get.md

> ## Documentation Index
> Fetch the complete documentation index at: https://docs.pipecat.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Get Organization Properties

> Retrieve current values of configurable properties for your organization.



## OpenAPI

````yaml GET /properties
openapi: 3.0.0
info:
  title: Pipecat Cloud - Get Organization Properties
  version: 1.0.0
  description: Get current organization properties via Pipecat Cloud Private API
servers:
  - url: https://api.pipecat.daily.co/v1
    description: API server
security:
  - PrivateKeyAuth: []
paths:
  /properties:
    get:
      summary: Get organization properties
      description: >-
        Retrieve the current values of all configurable properties for your
        organization. Returns only public (user-configurable) properties without
        schema metadata. For schema information including available values and
        constraints, use the `/properties/schema` endpoint.
      operationId: getProperties
      responses:
        '200':
          description: Properties retrieved successfully
          content:
            application/json:
              schema:
                type: object
                properties:
                  properties:
                    $ref: '#/components/schemas/OrganizationProperties'
              example:
                properties:
                  defaultRegion: us-west
        '401':
          description: Unauthorized - Invalid or missing API key
          content:
            application/json:
              schema:
                $ref: '#/components/schemas/ErrorResponse'
              example:
                error: Unauthorized
                code: UNAUTHORIZED
        '500':
          description: Internal server error
          content:
            application/json:
              schema:
                $ref: '#/components/schemas/ErrorResponse'
              example:
                error: Internal server error
                code: INTERNAL_SERVER_ERROR
      security:
        - PrivateKeyAuth: []
components:
  schemas:
    OrganizationProperties:
      type: object
      description: Organization configuration properties
      properties:
        defaultRegion:
          type: string
          description: Default region for new service deployments
          example: us-west
    ErrorResponse:
      type: object
      properties:
        error:
          type: string
          description: Error message
        code:
          type: string
          description: Error code
  securitySchemes:
    PrivateKeyAuth:
      type: http
      scheme: bearer
      description: >-
        Authentication requires a Pipecat Cloud Private API token.


        Generate a Private API key from your Dashboard (Settings > API Keys >
        Private > Create key) and include it as a Bearer token in the
        Authorization header.

````

---

# Source: https://docs.pipecat.ai/deployment/pipecat-cloud/rest-reference/endpoint/properties-update.md

> ## Documentation Index
> Fetch the complete documentation index at: https://docs.pipecat.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Update Organization Properties

> Update configurable properties for your organization, such as the default deployment region.



## OpenAPI

````yaml PATCH /properties
openapi: 3.0.0
info:
  title: Pipecat Cloud - Organization Properties
  version: 1.0.0
  description: Manage organization properties via Pipecat Cloud Private API
servers:
  - url: https://api.pipecat.daily.co/v1
    description: API server
security:
  - PrivateKeyAuth: []
paths:
  /properties:
    patch:
      summary: Update organization properties
      description: >-
        Update one or more configurable properties for your organization. Only
        public (user-configurable) properties can be modified via this endpoint.
        Properties are validated against their schema and constraints before
        being applied. Returns the complete property schema including metadata
        for all properties.
      operationId: updateProperties
      requestBody:
        required: true
        content:
          application/json:
            schema:
              type: object
              properties:
                defaultRegion:
                  type: string
                  description: >-
                    Default region for new service deployments. Must be one of
                    the [available
                    regions](/deployment/pipecat-cloud/rest-reference/endpoint/regions-list)
                    for your organization.
                  example: us-west
              additionalProperties: false
            examples:
              updateDefaultRegion:
                summary: Update default region
                value:
                  defaultRegion: eu-central
      responses:
        '200':
          description: Properties successfully updated
          content:
            application/json:
              schema:
                type: object
                properties:
                  properties:
                    type: object
                    additionalProperties:
                      $ref: '#/components/schemas/PropertySchema'
              example:
                properties:
                  defaultRegion:
                    description: Default region for new service deployments
                    type: string
                    readOnly: false
                    currentValue: eu-central
                    default: us-west
                    availableValues:
                      - us-west
                      - us-east
                      - eu-central
                      - ap-south
        '400':
          description: Bad request - Invalid property name, value, or validation error
          content:
            application/json:
              schema:
                $ref: '#/components/schemas/ErrorResponse'
              examples:
                invalidProperty:
                  summary: Invalid property name
                  value:
                    error: Property 'invalidProp' is not a valid property
                    code: GENERIC_BAD_REQUEST
                invalidRegion:
                  summary: Invalid region value
                  value:
                    error: >-
                      Region 'invalid-region' is not available. Available
                      regions: us-west, us-east, eu-central
                    code: GENERIC_BAD_REQUEST
                emptyUpdate:
                  summary: No properties provided
                  value:
                    error: No properties provided
                    code: GENERIC_BAD_REQUEST
        '401':
          description: Unauthorized - Invalid or missing API key
          content:
            application/json:
              schema:
                $ref: '#/components/schemas/ErrorResponse'
              example:
                error: Unauthorized
                code: UNAUTHORIZED
        '500':
          description: Internal server error
          content:
            application/json:
              schema:
                $ref: '#/components/schemas/ErrorResponse'
              example:
                error: Internal server error
                code: INTERNAL_SERVER_ERROR
      security:
        - PrivateKeyAuth: []
components:
  schemas:
    PropertySchema:
      type: object
      description: Schema information for a single property
      properties:
        type:
          type: string
          description: Property data type (string, number, boolean, array, etc.)
          example: string
        description:
          type: string
          description: Human-readable description of the property
          example: Default region for new service deployments
        default:
          description: Default value when property is not set
        currentValue:
          description: Current value of the property for this organization
        readOnly:
          type: boolean
          description: Whether this property is read-only (cannot be modified via API)
          example: false
        availableValues:
          type: array
          description: List of valid values for this property (if applicable)
          items: {}
      required:
        - type
        - description
        - default
        - currentValue
        - readOnly
    ErrorResponse:
      type: object
      properties:
        error:
          type: string
          description: Error message
        code:
          type: string
          description: Error code
  securitySchemes:
    PrivateKeyAuth:
      type: http
      scheme: bearer
      description: >-
        Authentication requires a Pipecat Cloud Private API token.


        Generate a Private API key from your Dashboard (Settings > API Keys >
        Private > Create key) and include it as a Bearer token in the
        Authorization header.

````

---

# Source: https://docs.pipecat.ai/getting-started/quickstart.md

> ## Documentation Index
> Fetch the complete documentation index at: https://docs.pipecat.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Quickstart

> Run your first Pipecat bot in under 5 minutes

This quickstart guide will help you build and deploy your first Pipecat voice AI bot. You'll create a simple conversational agent that you can talk to in real-time, then deploy it to production on Pipecat Cloud.

**Two steps**: Local Development (5 min) → Production Deployment (5 min)

## Step 1: Local Development

### Prerequisites

**Environment**

* Python 3.10 or later
* [uv](https://docs.astral.sh/uv/getting-started/installation/) package manager installed

**AI Service API Keys**

This quickstart uses three AI services working together in a pipeline. You'll need API keys from each service:

<CardGroup cols={3}>
  <Card title="Deepgram (STT)" icon="microphone" href="https://console.deepgram.com/signup">
    Create an account and generate your API key for real-time
    speech recognition.
  </Card>

  <Card title="OpenAI (LLM)" icon="brain" href="https://auth.openai.com/create-account">
    Create an account and generate an API key for intelligent conversation
    responses.
  </Card>

  <Card title="Cartesia (TTS)" icon="volume-high" href="https://play.cartesia.ai/sign-up">
    Sign up and generate your API key for natural voice
    synthesis.
  </Card>
</CardGroup>

Have these API keys ready. You'll add them to your environment file in the next section.

### Setup

1. Clone the quickstart repository

```bash  theme={null}
git clone https://github.com/pipecat-ai/pipecat-quickstart.git
cd pipecat-quickstart
```

2. Configure your API keys

Create your environment file:

```bash  theme={null}
cp env.example .env
```

Open the `.env` file in your text editor and add your API keys:

```ini  theme={null}
DEEPGRAM_API_KEY=your_deepgram_api_key
OPENAI_API_KEY=your_openai_api_key
CARTESIA_API_KEY=your_cartesia_api_key
```

3. Set up virtual environment and install dependencies

```bash  theme={null}
uv sync
```

### Run your bot locally

Now you're ready to run your bot! Start it with

```bash  theme={null}
uv run bot.py
```

You should see output similar to this:

```
🚀 WebRTC server starting at http://localhost:7860/client
   Open this URL in your browser to connect!
```

Open [http://localhost:7860/client](http://localhost:7860/client) in your browser and click **Connect** to start talking to your bot.

<Note>
  **First run note**: The initial startup may take \~20 seconds as Pipecat
  downloads required models and imports. Subsequent runs will be much faster.
</Note>

🎉 **Success!** Your bot is running locally. Now let's deploy it to production so others can use it.

***

## Step 2: Deploy to Production

Transform your local bot into a production-ready service. Pipecat Cloud handles scaling, monitoring, and global deployment.

### Prerequisites

1. Sign up for Pipecat Cloud

[Create your Pipecat Cloud account](https://pipecat.daily.co/sign-up) to deploy and manage your bots.

2. Set up Docker

* **Install [Docker](https://www.docker.com/)** on your system
* **Create a [Docker Hub](https://hub.docker.com/) account**
* **Login to Docker Hub:**

```bash  theme={null}
docker login
```

3. Pipecat CLI

The Pipecat CLI is already installed with your quickstart project.

Log in to Pipecat Cloud using the CLI to get started:

```bash  theme={null}
uv run pipecat cloud auth login
```

You'll be presented with a link that you can click to authenticate your client.

### Configure your deployment

The `pcc-deploy.toml` file tells Pipecat Cloud how to run your bot. **Update the image field** with your Docker Hub username:

```toml  theme={null}
agent_name = "quickstart"
image = "YOUR_DOCKERHUB_USERNAME/quickstart:0.1"  # 👈 Update this line
secret_set = "quickstart-secrets"

[scaling]
	min_agents = 1
```

**Understanding the configuration:**

* `agent_name`: Your bot's name in Pipecat Cloud
* `image`: The Docker image to deploy (format: `username/image:version`)
* `secret_set`: Where your API keys are stored securely
* `min_agents`: Number of bot instances to keep ready (1 = instant start)

<Tip>
  [Set up
  `image_credentials`](/deployment/pipecat-cloud/fundamentals/secrets#image-pull-secrets)
  in your TOML file for authenticated image pulls.
</Tip>

### Configure secrets

Upload your API keys to Pipecat Cloud's secure storage:

```bash  theme={null}
uv run pipecat cloud secrets set quickstart-secrets --file .env
```

This creates a secret set called `quickstart-secrets` (matching your TOML file) and uploads all your API keys from `.env`.

### Build and deploy

1. Build and push your Docker image

```bash  theme={null}
uv run pipecat cloud docker build-push
```

This builds your Docker image and pushes it to Docker Hub based on the `image` information in your `pcc-deploy.toml` file.

2. Deploy to Pipecat Cloud

```bash  theme={null}
uv run pipecat cloud deploy
```

### Connect to your agent

1. Open your [Pipecat Cloud dashboard](https://pipecat.daily.co/)
2. Select your `quickstart` agent → **Sandbox**
3. Allow microphone access and click **Connect**

🎉 **Your bot is now live in production!**

<Card title="Ready to scale?" icon="rocket" href="/deployment/pipecat-cloud/introduction">
  Explore advanced Pipecat Cloud features like scaling, monitoring, secrets
  management, and production best practices.
</Card>

***

## Understanding the Quickstart Bot

When you speak to your bot, here's the real-time pipeline that processes your conversation:

1. **Audio Capture**: Your browser captures microphone audio and sends it via WebRTC
2. **Voice Activity Detection**: Silero VAD detects when you start and stop speaking
3. **Speech Recognition**: Deepgram converts your speech to text in real-time
4. **Language Processing**: OpenAI's GPT model generates an intelligent response
5. **Speech Synthesis**: Cartesia converts the response text back to natural speech
6. **Audio Playback**: The generated audio streams back to your browser

Each step happens with minimal latency, typically completing the full round-trip in under one second.

### AI Services

Your bot uses three AI services, each configured with API keys from your `.env` file:

```python  theme={null}
# Create AI Services
stt = DeepgramSTTService(api_key=os.getenv("DEEPGRAM_API_KEY"))
tts = CartesiaTTSService(
    api_key=os.getenv("CARTESIA_API_KEY"),
    voice_id="71a7ad14-091c-4e8e-a314-022ece01c121",
)
llm = OpenAILLMService(api_key=os.getenv("OPENAI_API_KEY"))
```

Pipecat supports many different AI services. You can swap out Deepgram for Azure Speech, OpenAI for Anthropic, or Cartesia for ElevenLabs without changing the rest of your code.

See the [supported services documentation](/server/services/supported-services) for all available options.

### Context and Messages

Your bot maintains conversation history using a context object, enabling multi-turn interactions where the bot remembers what was said earlier.

The context is initialized with a system message that defines the bot's personality:

```python  theme={null}
# All messages use the OpenAI message format
messages = [
    {
        "role": "system",
        "content": "You are a friendly AI assistant. Respond naturally and keep your answers conversational.",
    },
]

context = LLMContext(messages)
user_aggregator, assistant_aggregator = LLMContextAggregatorPair(
    context,
    user_params=LLMUserAggregatorParams(
        vad_analyzer=SileroVADAnalyzer(params=VADParams(stop_secs=0.2)),
    ),
)
```

The context aggregator automatically collects user messages (after speech-to-text) and assistant responses (after text-to-speech), maintaining the conversation flow without manual intervention.

### RTVI Protocol

When building web or mobile clients, you can use [Pipecat's client SDKs](/client/introduction) that communicate with your bot via the [RTVI (Real-Time Voice Interaction) protocol](/client/rtvi-standard). Pipecat comes with RTVI enabled by default, allowing your the Pipecat server and client to exchange messages and events in real-time.

### Pipeline Configuration

The core of your bot is a Pipeline that processes data through a series of processors:

```python  theme={null}
# Create the pipeline with the processors
pipeline = Pipeline([
    transport.input(),     # Receive audio from browser
    stt,                   # Speech-to-text (Deepgram)
    user_aggregator,       # Add user message to context
    llm,                   # Language model (OpenAI)
    tts,                   # Text-to-speech (Cartesia)
    transport.output(),    # Send audio back to browser
    assistant_aggregator,  # Add bot response to context
])
```

Data flows through the pipeline as "frames", objects containing audio, text, or other data types. The ordering is crucial: audio must be transcribed before it can be processed by the LLM, and text must be synthesized before it can be played back.

The pipeline is managed by a PipelineTask:

```python  theme={null}
# Create a PipelineTask to manage the pipeline execution
task = PipelineTask(
    pipeline,
    params=PipelineParams(
        enable_metrics=True,
        enable_usage_metrics=True,
    ),
)
```

The task handles pipeline execution, collects metrics, and manages RTVI events through [observers](/server/utilities/observers/observer-pattern).

### Event Handlers

Event handlers manage the bot's lifecycle and user interactions:

```python  theme={null}
# Event handler for when a client connects
@transport.event_handler("on_client_connected")
async def on_client_connected(transport, client):
    logger.info(f"Client connected")
    # Add a greeting message to the context
    messages.append({"role": "system", "content": "Say hello and briefly introduce yourself."})
    # Prompt the bot to start talking when the client connects
    await task.queue_frames([LLMRunFrame()])

# Event handler for when a client disconnects
@transport.event_handler("on_client_disconnected")
async def on_client_disconnected(transport, client):
    logger.info(f"Client disconnected")
    # Cancel the task when the client disconnects
    # This stops the pipeline and all processors, cleaning up resources
    await task.cancel()
```

When a client connects, the bot adds a greeting instruction and queues a context frame to initiate the conversation. When disconnecting, it properly cancels the task to clean up resources.

### Running the Pipeline

Finally, the pipeline is executed by a PipelineRunner:

```python  theme={null}
# Create a PipelineRunner to run the task
runner = PipelineRunner(handle_sigint=False)

# Finally, run the task using the runner
# This will start the pipeline and begin processing frames
await runner.run(task)
```

The runner manages the pipeline's execution lifecycle. Note that `handle_sigint=False` because the main runner handles system signals.

### Bot Entry Point

The quickstart uses Pipecat's runner system:

```python  theme={null}
async def bot(runner_args: RunnerArguments):
    """Main bot entry point."""

    # Configure transport parameters for different environments
    transport_params = {
        "daily": lambda: DailyParams(
            audio_in_enabled=True,
            audio_out_enabled=True,
        ),
        "webrtc": lambda: TransportParams(
            audio_in_enabled=True,
            audio_out_enabled=True,
        ),
    }

    transport = await create_transport(runner_args, transport_params)
    await run_bot(transport, runner_args)

if __name__ == "__main__":
    from pipecat.runner.run import main
    main()
```

This runner automatically handles WebRTC connection setup and management, making it easy to get started with minimal configuration. The same code works for both local development and production deployment.

<Tip>
  **Production ready**: This bot pattern is fully compatible with Pipecat Cloud,
  meaning you can deploy your bot without any code changes.
</Tip>

## Troubleshooting

* **Browser permissions**: Make sure to allow microphone access when prompted by your browser.
* **Connection issues**: If the WebRTC connection fails, first try a different browser. If that fails, make sure you don't have a VPN or firewall rules blocking traffic. WebRTC uses UDP to communicate.
* **Audio issues**: Check that your microphone and speakers are working and not muted.

## Next Steps

Congratulations! You've built and deployed your first Pipecat bot. Here's what to do next based on your goals:

### 🚀 Ready to Build Your Own Application?

The quickstart gave you a working example, but the Pipecat CLI helps you scaffold production-ready projects with your choice of platform (phone vs web/mobile), transport providers, and AI services—all tailored to your specific use case.

<Card title="Build Your Next Bot" icon="rocket" href="/getting-started/build-your-next-bot">
  Use the CLI to scaffold phone or web/mobile projects customized for your needs
</Card>

### 🧠 Want to Understand How It Works?

Dive deeper into Pipecat's architecture and learn how to build custom solutions.

<CardGroup cols={2}>
  <Card title="Learn Core Concepts" icon="book-open" href="/guides/learn/overview">
    Master pipelines, processors, transports, and context management
  </Card>

  <Card title="Browse Examples" icon="code" href="https://github.com/pipecat-ai/pipecat-examples">
    30+ production-ready examples for inspiration
  </Card>
</CardGroup>

---

# Source: https://docs.pipecat.ai/server/services/llm/qwen.md

> ## Documentation Index
> Fetch the complete documentation index at: https://docs.pipecat.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Qwen

> LLM service implementation using Alibaba Cloud's Qwen models through an OpenAI-compatible interface

## Overview

`QwenLLMService` provides access to Alibaba Cloud's Qwen language models through an OpenAI-compatible interface. It inherits from `OpenAILLMService` and supports streaming responses, function calling, and context management, with particularly strong capabilities for Chinese language processing.

<CardGroup cols={2}>
  <Card title="Qwen LLM API Reference" icon="code" href="https://reference-server.pipecat.ai/en/latest/api/pipecat.services.qwen.llm.html">
    Pipecat's API methods for Qwen integration
  </Card>

  <Card title="Example Implementation" icon="play" href="https://github.com/pipecat-ai/pipecat/blob/main/examples/foundational/14q-function-calling-qwen.py">
    Complete example with function calling
  </Card>

  <Card title="Qwen Documentation" icon="book" href="https://www.alibabacloud.com/help/en/model-studio/use-qwen-by-calling-api">
    Official Qwen API documentation and features
  </Card>

  <Card title="Alibaba Cloud Console" icon="microphone" href="https://www.alibabacloud.com/">
    Access Qwen models and manage API keys
  </Card>
</CardGroup>

## Installation

To use Qwen services, install the required dependencies:

```bash  theme={null}
pip install "pipecat-ai[qwen]"
```

## Prerequisites

### Qwen Account Setup

Before using Qwen LLM services, you need:

1. **Alibaba Cloud Account**: Sign up at [Alibaba Cloud](https://www.alibabacloud.com/)
2. **API Key**: Generate an API key from your Model Studio dashboard
3. **Model Selection**: Choose from available Qwen models with multilingual capabilities

### Required Environment Variables

* `QWEN_API_KEY`: Your Qwen API key for authentication

---

# Source: https://docs.pipecat.ai/recipes.md

> ## Documentation Index
> Fetch the complete documentation index at: https://docs.pipecat.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Recipes

> Code snippets and techniques to enhance your Pipecat applications

<Update label={<><Icon icon="code" size={16} color="currentColor" /><span style={{marginLeft: '8px'}}>Function Calling</span></>} tags={["LLM"]}>
  Add function calling to your Pipecat bot. Examples exist for each LLM provider supported in Pipecat.

  [View Recipe →](https://github.com/pipecat-ai/pipecat/blob/main/examples/foundational/14-function-calling.py)
</Update>

<Update label={<><Icon icon="microphone" size={16} color="currentColor" /><span style={{marginLeft: '8px'}}>Record Audio</span></>} tags={["Recording & Logging"]}>
  Collect audio frames from the user and bot for later processing or storage.

  [View Recipe →](https://github.com/pipecat-ai/pipecat/blob/main/examples/foundational/34-audio-recording.py)
</Update>

<Update label={<><Icon icon="file-lines" size={16} color="currentColor" /><span style={{marginLeft: '8px'}}>Capture Transcripts</span></>} tags={["Recording & Logging"]}>
  Capture user and bot transcripts for later processing or storage.

  [View Recipe →](https://github.com/pipecat-ai/pipecat/blob/main/examples/foundational/28-transcription-processor.py)
</Update>

<Update label={<><Icon icon="volume-high" size={16} color="currentColor" /><span style={{marginLeft: '8px'}}>Play Background Sound</span></>} tags={["Audio"]}>
  Play a background sound in your Pipecat bot. The audio is mixed with the transport audio to create a single integrated audio stream.

  [View Recipe →](https://github.com/pipecat-ai/pipecat/blob/main/examples/foundational/23-bot-background-sound.py)
</Update>

<Update label={<><Icon icon="microphone-slash" size={16} color="currentColor" /><span style={{marginLeft: '8px'}}>Mute User Input</span></>} tags={["User Interaction"]}>
  Specify a strategy for mute to mute user input, allowing the bot to continue without interruption.

  [View Recipe →](https://github.com/pipecat-ai/pipecat/blob/main/examples/foundational/24-stt-mute-filter.py)
</Update>

<Update label={<><Icon icon="bell" size={16} color="currentColor" /><span style={{marginLeft: '8px'}}>Wake Phrase</span></>} tags={["User Interaction"]}>
  Use a wake phrase to wake up your Pipecat bot.

  [View Recipe →](https://github.com/pipecat-ai/pipecat/blob/main/examples/foundational/10-wake-phrase.py)
</Update>

<Update label={<><Icon icon="music" size={16} color="currentColor" /><span style={{marginLeft: '8px'}}>Play Sound Effects</span></>} tags={["Audio"]}>
  Play sound effects in your Pipecat bot.

  [View Recipe →](https://github.com/pipecat-ai/pipecat/blob/main/examples/foundational/11-sound-effects.py)
</Update>

<Update label={<><Icon icon="language" size={16} color="currentColor" /><span style={{marginLeft: '8px'}}>Switch Languages</span></>} tags={["Multilingual"]}>
  A ParallelPipeline example showing how to dynamically switch languages.

  [View Recipe →](https://github.com/pipecat-ai/pipecat/blob/main/examples/foundational/15a-switch-languages.py)
</Update>

<Update label={<><Icon icon="hourglass" size={16} color="currentColor" /><span style={{marginLeft: '8px'}}>Detect an Idle User</span></>} tags={["User Interaction"]}>
  Detect when a user is idle and automatically respond.

  [View Recipe →](https://github.com/pipecat-ai/pipecat/blob/main/examples/foundational/17-detect-user-idle.py)
</Update>

<Update label={<><Icon icon="bug" size={16} color="currentColor" /><span style={{marginLeft: '8px'}}>Debug with an Observer</span></>} tags={["Debugging"]}>
  Learn how to debug your Pipecat bot with an Observer by observing frames flowing through the pipeline.

  [View Recipe →](https://github.com/pipecat-ai/pipecat/blob/main/examples/foundational/30-observer.py)
</Update>

<Update label={<><Icon icon="monitor-waveform" size={16} color="currentColor" /><span style={{marginLeft: '8px'}}>Live Debugger (Whisker)</span></>} tags={["Debugging"]}>
  A live graphical debugger for the Pipecat voice and multimodal conversational AI framework. It lets you visualize pipelines and debug frames in real time — so you can see exactly what your bot is thinking and doing.

  [View Recipe →](https://github.com/pipecat-ai/whisker)
</Update>

<Update label={<><Icon icon="envelope" size={16} color="currentColor" /><span style={{marginLeft: '8px'}}>Collect Emails</span></>} tags={["Recording & Logging"]}>
  Parse user email from the LLM response.

  [View Recipe →](https://github.com/pipecat-ai/pipecat/blob/main/examples/foundational/36-user-email-gathering.py)
</Update>

<Update label={<><Icon icon="brain" size={16} color="currentColor" /><span style={{marginLeft: '8px'}}>Smart Turn Detection</span></>} tags={["User Interaction"]}>
  Detect when a user has finished speaking and automatically respond. Learn more about [smart-turn model](https://github.com/pipecat-ai/smart-turn).

  [View Recipe →](https://github.com/pipecat-ai/pipecat/blob/main/examples/foundational/38b-smart-turn-local.py)
</Update>

<Update label={<><Icon icon="screwdriver-wrench" size={16} color="currentColor" /><span style={{marginLeft: '8px'}}>MCP Tools</span></>} tags={["Integration"]}>
  Use MCP tools to interact with external services.

  [View Recipe →](https://github.com/pipecat-ai/pipecat/blob/main/examples/foundational/39-mcp-stdio.py)
</Update>

<Update label={<><Icon icon="hand" size={16} color="currentColor" /><span style={{marginLeft: '8px'}}>Interruption Strategies</span></>} tags={["User Interaction"]}>
  Learn how to configure interruption strategies for your Pipecat bot.

  [View Recipe →](https://github.com/pipecat-ai/pipecat/blob/main/examples/foundational/42-interruption-config.py)
</Update>

<Update label={<><Icon icon="video" size={16} color="currentColor" /><span style={{marginLeft: '8px'}}>Describe Video</span></>} tags={["Vision"]}>
  Pass a video frame from a live video stream to a model and get a description.

  [View Recipe →](https://github.com/pipecat-ai/pipecat/blob/main/examples/foundational/12b-describe-video-gpt-4o.py)
</Update>

<Update label={<><Icon icon="users" size={16} color="currentColor" /><span style={{marginLeft: '8px'}}>User and Bot Turn Events</span></>} tags={["Events"]}>
  Handle user and bot end of turn events to add custom logic after a turn.

  [View Recipe →](https://github.com/pipecat-ai/pipecat/blob/main/examples/foundational/29-turn-tracking-observer.py)
</Update>

***

## 💡 Need Help Getting Started?

<CardGroup cols={3}>
  <Card title="More Recipes" icon="github" href="https://github.com/pipecat-ai/pipecat/tree/main/examples/foundational">
    Browse all recipes on GitHub
  </Card>

  <Card title="Examples" icon="folder" href="/examples">
    Browse complete applications
  </Card>

  <Card title="Discord Community" icon="discord" href="https://discord.gg/pipecat">
    Get help from the community
  </Card>
</CardGroup>

---

# Source: https://docs.pipecat.ai/guides/fundamentals/recording-audio.md

> ## Documentation Index
> Fetch the complete documentation index at: https://docs.pipecat.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Recording Conversation Audio

> Learn how to record and save audio from conversations between users and your bot

## Overview

Recording audio from conversations provides valuable data for analysis, debugging, and quality control. You have two options for how to record with Pipecat:

### Option 1: Record using your transport service provider

Record without writing custom code by using your [transport](/server/services/supported-services#transports) provider's recording capabilities. In addition to saving you development time, some providers offer unique recording capabilities.

<Info>Refer to your service provider's documentation to learn more.</Info>

### Option 2: Create your own recording pipeline

Pipecat's `AudioBufferProcessor` makes it easy to capture high-quality audio recordings of both the user and bot during interactions. Opt for this approach if you want more control over your recording.

This guide focuses on how to recording using the `AudioBufferProcessor`, including high-level guidance for how to set up post-processing jobs for longer recordings.

## How the AudioBufferProcessor Works

The `AudioBufferProcessor` captures audio by:

1. Collecting audio frames from both the user (input) and bot (output)
2. Emitting events with recorded audio data
3. Providing options for composite or separate track recordings

<Note>
  Add the processor to your pipeline after the `transport.output()` to capture
  both the user audio and the bot audio as it's spoken.
</Note>

## Audio Recording Options

The `AudioBufferProcessor` offers several configuration options:

* **Composite recording**: Combined audio from both user and bot
  * `on_audio_data` event handler
* **Track-level recording**: Separate audio files for user and bot
  * `on_track_audio_data` event handler
* **Turn-based recording**: Individual audio clips for each speaking turn
  * `on_user_turn_audio_data` and `on_bot_turn_audio_data` event handlers
* **Mono or stereo output**: Single channel mixing or two-channel separation
  * `num_channels=1` for mono; `num_channels=2` for stereo

## Basic Implementation

### Step 1: Create an Audio Buffer Processor

Initialize the audio buffer processor with your desired configuration:

```python  theme={null}
from pipecat.processors.audio.audio_buffer_processor import AudioBufferProcessor

# Create audio buffer processor with default settings
audiobuffer = AudioBufferProcessor(
    num_channels=1,               # 1 for mono, 2 for stereo (user left, bot right)
    enable_turn_audio=False,      # Enable per-turn audio recording
    user_continuous_stream=True,  # User has continuous audio stream
)
```

### Step 2: Add to Your Pipeline

Place the processor in your pipeline after all audio-producing components:

```python  theme={null}
pipeline = Pipeline(
    [
        transport.input(),
        stt,
        context_aggregator.user(),
        llm,
        tts,
        transport.output(),
        audiobuffer,          # Add after all audio components
        context_aggregator.assistant(),
    ]
)
```

### Step 3: Start Recording

Explicitly start recording when needed, typically when a session begins:

```python  theme={null}
@transport.event_handler("on_client_connected")
async def on_client_connected(transport, client):
    logger.info(f"Client connected")
    # Important: Start recording explicitly
    await audiobuffer.start_recording()
    # Continue with session initialization...
```

<Warning>
  You must call `start_recording()` explicitly to begin capturing audio. The
  processor won't record automatically when initialized.
</Warning>

### Step 4: Handle Audio Data

Register an event handler to process audio data:

```python  theme={null}
@audiobuffer.event_handler("on_audio_data")
async def on_audio_data(buffer, audio, sample_rate, num_channels):
    # Save or process the composite audio
    timestamp = datetime.datetime.now().strftime("%Y%m%d_%H%M%S")
    filename = f"recordings/conversation_{timestamp}.wav"

    # Create the WAV file
    with wave.open(filename, "wb") as wf:
        wf.setnchannels(num_channels)
        wf.setsampwidth(2)  # 16-bit audio
        wf.setframerate(sample_rate)
        wf.writeframes(audio)

    logger.info(f"Saved recording to {filename}")
```

<Tip>
  If recording separate tracks, you can use the `on_track_audio_data` event
  handler to save user and bot audio separately.
</Tip>

## Recording Longer Conversations

For conversations that last a few minutes, it may be sufficient to just buffer the audio in memory. However, for longer sessions, storing audio in memory poses two challenges:

1. **Memory Usage**: Long recordings can consume significant memory, leading to potential crashes or performance issues.
2. **Conversation Loss**: If the application crashes or the connection drops, you may lose all recorded audio.

Instead, use the `buffer_size` parameter to record audio in manageable segments. This allows you to periodically save audio data to disk or upload it to cloud storage, reducing memory usage and ensuring data persistence.
See an example of how to upload chunked audio to AWS cloud storage [here]().

### Chunked Recording

Set a reasonable `buffer_size` to trigger periodic uploads:

```python  theme={null}
# 30-second chunks (recommended for most use cases)
SAMPLE_RATE = 24000
CHUNK_DURATION = 30  # seconds
audiobuffer = AudioBufferProcessor(
    sample_rate=SAMPLE_RATE,
    buffer_size=SAMPLE_RATE * 2 * CHUNK_DURATION  # 2 bytes per sample (16-bit)
)

chunk_counter = 0

@audiobuffer.event_handler("on_track_audio_data")
async def on_chunk_ready(buffer, user_audio, bot_audio, sample_rate, num_channels):
    global chunk_counter

    # Upload or save individual chunks
    await upload_audio_chunk(f"user_chunk_{chunk_counter:03d}.wav", user_audio, sample_rate, 1)
    await upload_audio_chunk(f"bot_chunk_{chunk_counter:03d}.wav", bot_audio, sample_rate, 1)

    chunk_counter += 1
```

### Multipart Upload Strategy

For cloud storage, use multipart uploads to stream audio chunks. For example AWS cloud storage, use the [s3 multipart upload API](https://docs.aws.amazon.com/AmazonS3/latest/userguide/mpuoverview.html).

If you are rolling your own multipart upload code, consider the following:

**Conceptual Approach:**

1. **Initialize multipart upload** when recording starts
2. **Upload chunks as parts** when buffers fill (every \~30 seconds)
3. **Complete multipart upload** when recording ends
4. **Post-process** to create final WAV file(s), concatenate audio chunks

**Benefits:**

* Memory efficient for long sessions
* Fault tolerant (no data loss if connection drops)
* Enables real-time processing and analysis
* Parallel upload of multiple tracks

### \[Optional] Post-Processing Pipeline

If not using a managed multipart upload framework like AWS s3 multipart upload, concatenate audio chunks together to create final audio files.
This can be done with tools like FFmpeg:

**Concatenating Audio Files:**

```bash  theme={null}
# Method 1: Simple concatenation (same format)
ffmpeg -i "concat:chunk_001.wav|chunk_002.wav|chunk_003.wav" -acodec copy final.wav

# Method 2: Using file list (recommended for many chunks)
# Create filelist.txt with format:
# file 'chunk_001.wav'
# file 'chunk_002.wav'
# ...
ffmpeg -f concat -safe 0 -i filelist.txt -c copy final_recording.wav
```

**Automation Considerations:**

* Use sequence numbers in chunk filenames for proper ordering
* Include metadata (sample rate, channels, duration) with each chunk
* Implement retry logic for failed uploads
* Consider using cloud functions/lambdas for automatic post-processing

## Next Steps

<CardGroup cols={2}>
  <Card title="Try the Audio Recording Example" icon="code" iconType="duotone" href="https://github.com/pipecat-ai/pipecat/blob/main/examples/foundational/34-audio-recording.py">
    Explore a complete working example that demonstrates how to record and save
    both composite and track-level audio with Pipecat.
  </Card>

  <Card title="AudioBufferProcessor Reference" icon="book" iconType="duotone" href="/server/utilities/audio/audio-buffer-processor">
    Read the complete API reference documentation for advanced configuration
    options and event handlers.
  </Card>
</CardGroup>

Consider implementing audio recording in your application for quality assurance, training data collection, or creating conversation archives. The recorded audio can be stored locally, uploaded to cloud storage, or processed in real-time for further analysis.

---

# Source: https://docs.pipecat.ai/deployment/pipecat-cloud/rest-reference/endpoint/regions-list.md

> ## Documentation Index
> Fetch the complete documentation index at: https://docs.pipecat.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# List Regions

> Retrieve a list of all available regions for deploying agents and storing secrets.



## OpenAPI

````yaml GET /regions
openapi: 3.0.0
info:
  title: Pipecat Cloud - List Regions
  version: 1.0.0
  description: List available regions via Pipecat Cloud API
servers:
  - url: https://api.pipecat.daily.co/v1
    description: API server
security: []
paths:
  /regions:
    get:
      summary: List available regions
      description: >-
        Retrieve a list of all regions available for deploying agents and
        storing secrets. This endpoint can be accessed using either Clerk
        authentication or a Private API key.
      operationId: listRegions
      responses:
        '200':
          description: List of available regions
          content:
            application/json:
              schema:
                type: object
                properties:
                  regions:
                    type: array
                    items:
                      $ref: '#/components/schemas/Region'
              example:
                regions:
                  - code: us-west
                    display_name: US West (Oregon)
                  - code: us-east
                    display_name: US East (Virginia)
                  - code: eu-central
                    display_name: Europe (Frankfurt)
                  - code: ap-south
                    display_name: Asia Pacific (Mumbai)
        '401':
          description: Unauthorized - Invalid or missing authentication
          content:
            application/json:
              schema:
                $ref: '#/components/schemas/ErrorResponse'
              example:
                error: Unauthorized
                code: UNAUTHORIZED
        '500':
          description: Internal server error
          content:
            application/json:
              schema:
                $ref: '#/components/schemas/ErrorResponse'
              example:
                error: Internal server error
                code: INTERNAL_SERVER_ERROR
      security:
        - PrivateKeyAuth: []
components:
  schemas:
    Region:
      type: object
      properties:
        code:
          type: string
          description: Region code identifier (e.g., 'us-west', 'eu-central')
          example: us-west
        display_name:
          type: string
          description: Human-readable region name
          example: US West (Oregon)
      required:
        - code
        - display_name
    ErrorResponse:
      type: object
      properties:
        error:
          type: string
          description: Error message
        code:
          type: string
          description: Error code
  securitySchemes:
    PrivateKeyAuth:
      type: http
      scheme: bearer
      description: >-
        Authentication using a Pipecat Cloud Private API token.


        Generate a Private API key from your Dashboard (Settings > API Keys >
        Private > Create key) and include it as a Bearer token in the
        Authorization header.

````

---

# Source: https://docs.pipecat.ai/deployment/pipecat-cloud/guides/regions.md

# Source: https://docs.pipecat.ai/cli/cloud/regions.md

> ## Documentation Index
> Fetch the complete documentation index at: https://docs.pipecat.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# regions

> View available deployment regions

The `regions` command helps you discover which regions are available for deploying agents and storing secrets in Pipecat Cloud.

## list

List all available regions with their codes and display names.

**Usage:**

```shell  theme={null}
pipecat cloud regions list
```

This command displays a table of all regions where you can deploy agents and store secrets. Use the region codes shown in this list when specifying the `--region` flag in other commands.

**Example output:**

```
Code         Name
us-west      US West (Oregon)
us-east      US East (Virginia)
eu-central   Europe (Frankfurt)
ap-south     Asia Pacific (Mumbai)
```

## Using regions

Once you know the available region codes, you can use them with other commands:

**Deploy an agent to a specific region:**

```shell  theme={null}
pipecat cloud deploy my-agent my-image:latest --region eu-central
```

**Create secrets in a specific region:**

```shell  theme={null}
pipecat cloud secrets set my-secrets API_KEY=abc123 --region eu-central
```

**List agents in a specific region:**

```shell  theme={null}
pipecat cloud agent list --region us-east
```

<Info>
  Secrets and image pull secrets must be in the same region as the agents that
  use them. When deploying multi-region applications, create separate secret
  sets for each region.
</Info>

<Tip>
  Choose a region close to your users for optimal latency and performance.
</Tip>

---

# Source: https://docs.pipecat.ai/server/services/tts/resemble.md

> ## Documentation Index
> Fetch the complete documentation index at: https://docs.pipecat.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Resemble AI

> Text-to-speech service using Resemble AI's WebSocket streaming API with word-level timing

## Overview

`ResembleAITTSService` provides high-quality text-to-speech synthesis using Resemble AI's streaming WebSocket API with word-level timestamps and audio context management for handling multiple simultaneous synthesis requests with proper interruption support.

<CardGroup cols={2}>
  <Card title="Resemble AI TTS API Reference" icon="code" href="https://reference-server.pipecat.ai/en/latest/api/pipecat.services.resembleai.tts.html">
    Pipecat's API methods for Resemble AI TTS integration
  </Card>

  <Card title="Example Implementation" icon="play" href="https://github.com/pipecat-ai/pipecat/blob/main/examples/foundational/07zk-interruptible-resemble.py">
    Complete example with interruption handling
  </Card>

  <Card title="Resemble AI Documentation" icon="book" href="https://docs.resemble.ai/">
    Official Resemble AI API documentation
  </Card>

  <Card title="Sign up" icon="user" href="https://app.resemble.ai/">
    Sign up for a Resemble AI account
  </Card>
</CardGroup>

## Installation

To use Resemble AI services, install the required dependencies:

```bash  theme={null}
pip install "pipecat-ai[resemble]"
```

## Prerequisites

### Resemble AI Account Setup

Before using Resemble AI TTS services, you need:

1. **Resemble AI Account**: Sign up at [Resemble AI](https://app.resemble.ai)
2. **API Key**: Generate an API key from your [account settings](https://app.resemble.ai/account/api)
3. **Voice Selection**: Choose or create voice UUIDs from your [voice library](https://app.resemble.ai/hub/voices)

### Required Environment Variables

* `RESEMBLE_API_KEY`: Your Resemble AI API key for authentication

---

# Source: https://docs.pipecat.ai/server/utilities/daily/rest-helper.md

> ## Documentation Index
> Fetch the complete documentation index at: https://docs.pipecat.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# DailyRESTHelper

> Classes and methods for interacting with the Daily API to manage rooms and tokens

<Card title="Daily REST API Documentation" icon="link" href="https://docs.daily.co/reference/rest-api">
  For complete Daily REST API reference and additional details
</Card>

## Classes

### DailyRoomSipParams

Configuration for SIP (Session Initiation Protocol) parameters.

<ResponseField name="display_name" type="string" default="sw-sip-dialin">
  Display name for the SIP endpoint
</ResponseField>

<ResponseField name="video" type="boolean" default={false}>
  Whether video is enabled for SIP
</ResponseField>

<ResponseField name="sip_mode" type="string" default="dial-in">
  SIP connection mode
</ResponseField>

<ResponseField name="num_endpoints" type="integer" default={1}>
  Number of SIP endpoints
</ResponseField>

```python  theme={null}
from pipecat.transports.services.helpers.daily_rest import DailyRoomSipParams

sip_params = DailyRoomSipParams(
    display_name="conference-line",
    video=True,
    num_endpoints=2
)
```

### RecordingsBucketConfig

Configuration for storing Daily recordings in a custom S3 bucket.

<ResponseField name="bucket_name" type="string" required>
  Name of the S3 bucket for storing recordings
</ResponseField>

<ResponseField name="bucket_region" type="string" required>
  AWS region where the S3 bucket is located
</ResponseField>

<ResponseField name="assume_role_arn" type="string" required>
  ARN of the IAM role to assume for S3 access
</ResponseField>

<ResponseField name="allow_api_access" type="boolean" default={false}>
  Whether to allow API access to the recordings
</ResponseField>

```python  theme={null}
from pipecat.transports.services.helpers.daily_rest import RecordingsBucketConfig

bucket_config = RecordingsBucketConfig(
    bucket_name="my-recordings-bucket",
    bucket_region="us-west-2",
    assume_role_arn="arn:aws:iam::123456789012:role/DailyRecordingsRole",
    allow_api_access=True
)
```

### DailyRoomProperties

Properties that configure a Daily room's behavior and features.

<ResponseField name="exp" type="float" optional>
  Room expiration time as Unix timestamp (e.g., time.time() + 300 for 5 minutes)
</ResponseField>

<ResponseField name="enable_chat" type="boolean" default={false}>
  Whether chat is enabled in the room
</ResponseField>

<ResponseField name="enable_prejoin_ui" type="boolean" default={false}>
  Whether the prejoin lobby UI is enabled
</ResponseField>

<ResponseField name="enable_emoji_reactions" type="boolean" default={false}>
  Whether emoji reactions are enabled
</ResponseField>

<ResponseField name="eject_at_room_exp" type="boolean" default={false}>
  Whether to eject participants when room expires
</ResponseField>

<ResponseField name="enable_dialout" type="boolean" optional>
  Whether dial-out is enabled
</ResponseField>

<ResponseField name="enable_recording" type="string" optional>
  Recording settings ("cloud", "cloud-audio-only", "local", or "raw-tracks")
</ResponseField>

<ResponseField name="geo" type="string" optional>
  Geographic region for room
</ResponseField>

<ResponseField name="max_participants" type="number" optional>
  Maximum number of participants allowed in the room
</ResponseField>

<ResponseField name="recordings_bucket" type="RecordingsBucketConfig" optional>
  Configuration for custom S3 bucket recordings
</ResponseField>

<ResponseField name="sip" type="DailyRoomSipParams" optional>
  SIP configuration parameters
</ResponseField>

<ResponseField name="sip_uri" type="dict" optional>
  SIP URI configuration (returned by Daily)
</ResponseField>

<ResponseField name="start_video_off" type="boolean" default={false}>
  Whether the camera video is turned off by default
</ResponseField>

The class also includes a `sip_endpoint` property that returns the SIP endpoint URI if available.

`enable_recording` also supports `cloud-audio-only`, which records the call server-side and produces an audio-only MPEG-4 file with `.m4a` file extension and content type as `audio/mp4`. This recording setting behaves like `cloud`, except the `layout` options do not apply because there are no video tracks. Note: you can retrieve the resulting `.m4a` recordings via the [Daily REST API](https://docs.daily.co/reference/rest-api/recordings), in the same way you fetch `cloud` recording assets.

<Tip>
  If you're already using `enable_recording="cloud"` and want to switch to audio-only without changing your code, you can set `force_audio_only_recording: 1` on your Daily domain. This forces all cloud recordings to be audio-only (`.m4a` instead of `.mp4`) and skips recording video tracks even if they are present. This is useful when you want to transition to audio-only recordings immediately while avoiding an application redeploy. Switching `force_audio_only_recording: 0` will allow recording video tracks.
</Tip>

```python  theme={null}
import time
from pipecat.transports.services.helpers.daily_rest import (
    DailyRoomProperties,
    DailyRoomSipParams,
    RecordingsBucketConfig,
)

properties = DailyRoomProperties(
    exp=time.time() + 3600,  # 1 hour from now
    enable_chat=True,
    enable_emoji_reactions=True,
    enable_recording="cloud-audio-only",
    geo="us-west",
    max_participants=50,
    sip=DailyRoomSipParams(display_name="conference"),
    recordings_bucket=RecordingsBucketConfig(
        bucket_name="my-bucket",
        bucket_region="us-west-2",
        assume_role_arn="arn:aws:iam::123456789012:role/DailyRole"
    )
)

# Access SIP endpoint if available
if properties.sip_endpoint:
    print(f"SIP endpoint: {properties.sip_endpoint}")
```

### DailyRoomParams

Parameters for creating a new Daily room.

<ResponseField name="name" type="string" optional>
  Room name (if not provided, one will be generated)
</ResponseField>

<ResponseField name="privacy" type="string" default="public">
  Room privacy setting ("private" or "public")
</ResponseField>

<ResponseField name="properties" type="DailyRoomProperties">
  Room configuration properties
</ResponseField>

```python  theme={null}
import time
from pipecat.transports.services.helpers.daily_rest import (
    DailyRoomParams,
    DailyRoomProperties,
)

params = DailyRoomParams(
    name="team-meeting",
    privacy="private",
    properties=DailyRoomProperties(
        enable_chat=True,
        exp=time.time() + 7200  # 2 hours from now
    )
)
```

### DailyRoomObject

Response object representing a Daily room.

<ResponseField name="id" type="string">
  Unique room identifier
</ResponseField>

<ResponseField name="name" type="string">
  Room name
</ResponseField>

<ResponseField name="api_created" type="boolean">
  Whether the room was created via API
</ResponseField>

<ResponseField name="privacy" type="string">
  Room privacy setting
</ResponseField>

<ResponseField name="url" type="string">
  Complete room URL
</ResponseField>

<ResponseField name="created_at" type="string">
  Room creation timestamp in ISO 8601 format
</ResponseField>

<ResponseField name="config" type="DailyRoomProperties">
  Room configuration
</ResponseField>

```python  theme={null}
from pipecat.transports.services.helpers.daily_rest import (
    DailyRoomObject,
    DailyRoomProperties,
)

# Example of what a DailyRoomObject looks like when received
room = DailyRoomObject(
    id="abc123",
    name="team-meeting",
    api_created=True,
    privacy="private",
    url="https://your-domain.daily.co/team-meeting",
    created_at="2024-01-20T10:00:00.000Z",
    config=DailyRoomProperties(
        enable_chat=True,
        exp=1705743600
    )
)
```

### DailyMeetingTokenProperties

Properties for configuring a Daily meeting token.

<ResponseField name="room_name" type="string" optional>
  The room this token is valid for. If not set, token is valid for all rooms.
</ResponseField>

<ResponseField name="eject_at_token_exp" type="boolean" optional>
  Whether to eject user when token expires
</ResponseField>

<ResponseField name="eject_after_elapsed" type="integer" optional>
  Eject user after this many seconds
</ResponseField>

<ResponseField name="nbf" type="integer" optional>
  "Not before" timestamp - users cannot join before this time
</ResponseField>

<ResponseField name="exp" type="integer" optional>
  Expiration timestamp - users cannot join after this time
</ResponseField>

<ResponseField name="is_owner" type="boolean" optional>
  Whether token grants owner privileges
</ResponseField>

<ResponseField name="user_name" type="string" optional>
  User's display name in the meeting
</ResponseField>

<ResponseField name="user_id" type="string" optional>
  Unique identifier for the user (36 char limit)
</ResponseField>

<ResponseField name="enable_screenshare" type="boolean" optional>
  Whether user can share their screen
</ResponseField>

<ResponseField name="start_video_off" type="boolean" optional>
  Whether to join with video off
</ResponseField>

<ResponseField name="start_audio_off" type="boolean" optional>
  Whether to join with audio off
</ResponseField>

<ResponseField name="enable_recording" type="string" optional>
  Recording settings ("cloud", "cloud-audio-only", "local", or "raw-tracks")
</ResponseField>

<ResponseField name="enable_prejoin_ui" type="boolean" optional>
  Whether to show prejoin UI
</ResponseField>

<ResponseField name="start_cloud_recording" type="boolean" optional>
  Whether to start cloud recording when user joins
</ResponseField>

<ResponseField name="permissions" type="dict" optional>
  Initial default permissions for a non-meeting-owner participant
</ResponseField>

### DailyMeetingTokenParams

Parameters for creating a Daily meeting token.

<ResponseField name="properties" type="DailyMeetingTokenProperties">
  Token configuration properties
</ResponseField>

```python  theme={null}
from pipecat.transports.services.helpers.daily_rest import (
    DailyMeetingTokenParams,
    DailyMeetingTokenProperties,
)

token_params = DailyMeetingTokenParams(
    properties=DailyMeetingTokenProperties(
        user_name="John Doe",
        enable_screenshare=True,
        start_video_off=True,
        enable_recording="cloud-audio-only",
        start_cloud_recording=True,
        permissions={"canSend": ["video", "audio"]}
    )
)
```

<Warning>
  Recording type: ‎`cloud` will produce files with a ‎`.mp4` extension, while ‎`cloud-audio-only` will produce files with a ‎`.m4a` extension.
</Warning>

## Initialize DailyRESTHelper

Create a new instance of the Daily REST helper.

<ParamField path="daily_api_key" type="string" required>
  Your Daily API key
</ParamField>

<ParamField path="daily_api_url" type="string" default="https://api.daily.co/v1">
  The Daily API base URL
</ParamField>

<ParamField path="aiohttp_session" type="aiohttp.ClientSession" required>
  An aiohttp client session for making HTTP requests
</ParamField>

```python  theme={null}
helper = DailyRESTHelper(
    daily_api_key="your-api-key",
    aiohttp_session=session
)
```

## Create Room

Creates a new Daily room with specified parameters.

<ParamField path="params" type="DailyRoomParams" required>
  Room configuration parameters including name, privacy, and properties
</ParamField>

```python  theme={null}
# Create a room that expires in 1 hour
params = DailyRoomParams(
    name="my-room",
    privacy="private",
    properties=DailyRoomProperties(
        exp=time.time() + 3600,
        enable_chat=True
    )
)
room = await helper.create_room(params)
print(f"Room URL: {room.url}")
```

## Get Room From URL

Retrieves room information using a Daily room URL.

<ParamField path="room_url" type="string" required>
  The complete Daily room URL
</ParamField>

```python  theme={null}
room = await helper.get_room_from_url("https://your-domain.daily.co/my-room")
print(f"Room name: {room.name}")
```

## Get Token

Generates a meeting token for a specific room.

<ParamField path="room_url" type="string" required>
  The complete Daily room URL
</ParamField>

<ParamField path="expiry_time" type="float" default="3600">
  Token expiration time in seconds
</ParamField>

<ParamField path="eject_at_token_exp" type="bool" default="False">
  Whether to eject user when token expires
</ParamField>

<ParamField path="owner" type="bool" default="True">
  Whether the token should have owner privileges (overrides any setting in
  params)
</ParamField>

<ParamField path="params" type="DailyMeetingTokenParams" optional>
  Additional token configuration. Note that `room_name`, `exp`,
  `eject_at_token_exp`, and `is_owner` will be set based on the other function
  parameters.
</ParamField>

```python  theme={null}
# Basic token generation
token = await helper.get_token(
    room_url="https://your-domain.daily.co/my-room",
    expiry_time=1800,  # 30 minutes
    owner=True,
    eject_at_token_exp=True
)

# Advanced token generation with additional properties
token_params = DailyMeetingTokenParams(
    properties=DailyMeetingTokenProperties(
        user_name="John Doe",
        start_video_off=True
    )
)
token = await helper.get_token(
    room_url="https://your-domain.daily.co/my-room",
    expiry_time=1800,
    owner=False,
    eject_at_token_exp=True,
    params=token_params
)
```

## Delete Room By URL

Deletes a room using its URL.

<ParamField path="room_url" type="string" required>
  The complete Daily room URL
</ParamField>

```python  theme={null}
success = await helper.delete_room_by_url("https://your-domain.daily.co/my-room")
if success:
    print("Room deleted successfully")
```

## Delete Room By Name

Deletes a room using its name.

<ParamField path="room_name" type="string" required>
  The name of the Daily room
</ParamField>

```python  theme={null}
success = await helper.delete_room_by_name("my-room")
if success:
    print("Room deleted successfully")
```

## Get Name From URL

Extracts the room name from a Daily room URL.

<ParamField path="room_url" type="string" required>
  The complete Daily room URL
</ParamField>

```python  theme={null}
room_name = helper.get_name_from_url("https://your-domain.daily.co/my-room")
print(f"Room name: {room_name}")  # Outputs: "my-room"
```

---

# Source: https://docs.pipecat.ai/server/services/tts/rime.md

> ## Documentation Index
> Fetch the complete documentation index at: https://docs.pipecat.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Rime

> Text-to-speech service implementations using Rime AI

## Overview

Rime AI provides two TTS service implementations: `RimeTTSService` (WebSocket-based) with word-level timing and interruption support, and `RimeHttpTTSService` (HTTP-based) for simpler use cases. `RimeTTSService` is recommended for real-time interactive applications.

<CardGroup cols={2}>
  <Card title="Rime TTS API Reference" icon="code" href="https://reference-server.pipecat.ai/en/latest/api/pipecat.services.rime.tts.html">
    Pipecat's API methods for Rime TTS integration
  </Card>

  <Card title="Example Implementation" icon="play" href="https://github.com/pipecat-ai/pipecat/blob/main/examples/foundational/07q-interruptible-rime.py">
    Complete example with word timestamps
  </Card>

  <Card title="Rime Documentation" icon="book" href="https://docs.rime.ai/api-reference/endpoint/websockets-json">
    Official Rime WebSocket and HTTP API documentation
  </Card>

  <Card title="Voice Models" icon="microphone" href="https://docs.rime.ai/">
    Explore available voice models and features
  </Card>
</CardGroup>

## Installation

To use Rime services, install the required dependencies:

```bash  theme={null}
pip install "pipecat-ai[rime]"
```

## Prerequisites

### Rime Account Setup

Before using Rime TTS services, you need:

1. **Rime Account**: Sign up at [Rime AI](https://docs.rime.ai/)
2. **API Key**: Generate an API key from your account dashboard
3. **Voice Selection**: Choose from available voice models

### Required Environment Variables

* `RIME_API_KEY`: Your Rime API key for authentication

## Customizing Speech

`RimeTTSService` provides a set of helper methods for implementing Rime-specific customizations, meant to be used as part of text transformers. These include methods for spelling out text, adjusting speech rate, and modifying pitch. See the [Text Transformers for TTS](/guides/learn/text-to-speech#text-transformers-for-tts) section in the Text-to-Speech guide for usage examples.

### SPELL(text: str) -> str:

Implements [Rime's spell function](https://docs.rime.ai/api-reference/spell) to spell out text character by character.

```python  theme={null}
# Text transformers for TTS
# This will insert Rime's spell tags around the provided text.
async def spell_out_text(text: str, type: str) -> str:
    return RimeTTSService.SPELL(text)

tts = RimeTTSService(
    api_key=os.getenv("RIME_API_KEY"),
    text_transforms=[
        ("phone_number", spell_out_text),
    ],
)
```

### PAUSE\_TAG(seconds: float) -> str:

Implements [Rime's custom pause functionality](https://docs.rime.ai/api-reference/custom-pauses) to generate a properly formatted pause tag you can insert into the text.

```python  theme={null}
# Text transformers for TTS
# This will insert a one second pause after questions.
async def pause_after_questions(text: str, type: str) -> str:
    if text.endswith("?"):
        return f"{text}{RimeTTSService.PAUSE_TAG(1.0)}"
    return text

tts = RimeTTSService(
    api_key=os.getenv("RIME_API_KEY"),
    text_transforms=[
        ("sentence", pause_after_questions), # Only apply to sentence aggregations
    ],
)
```

### PRONOUNCE(self, text: str, word: str, phoneme: str) -> str:

Convenience method to support Rime's [custom pronunciations feature](https://docs.rime.ai/api-reference/custom-pronunciation). It takes a word and its desired phoneme representation, returning the text with the provided word replaced by the appropriate phoneme tag.

```python  theme={null}
# Text transformers for TTS
# This will a phoneme in place of the word "potato" to define how it
# should be pronounced.
async def maybe_say_potato_all_fancylike(text: str, type: str) -> str:
    if using_fancy_voice:
        return RimeTTSService.PRONOUNCE(text, "potato", "potato")
    else:
        return RimeTTSService.PRONOUNCE(text, "potato", "poteto")

tts = RimeTTSService(
    api_key=os.getenv("RIME_API_KEY"),
    text_transforms=[
        ("*", maybe_say_potato_all_fancylike), # Apply to all text
    ],
)
```

### INLINE\_SPEED(self, text: str, speed: float) -> str:

A convenience method to support Rime's [inline speed adjustment feature](https://docs.rime.ai/api-reference/speed). It will wrap the provided text in the `[]` tags and add the provided speed to the `inlineSpeedAlpha` field in the request metadata.

```python  theme={null}
# Text transformers for TTS
# This will make the word slow always be spoken more slowly.
async def slow_down_slow_words(text: str, type: str) -> str:
    return text.replace(
        "slow",
        RimeTTSService.INLINE_SPEED("slow", speed=0.5)
    )

tts = RimeTTSService(
    api_key=os.getenv("RIME_API_KEY"),
    text_transforms=[
        ("*", slow_down_slow_words), # Apply to all text
    ],
)
```

---

# Source: https://docs.pipecat.ai/server/services/tts/riva.md

# Source: https://docs.pipecat.ai/server/services/stt/riva.md

> ## Documentation Index
> Fetch the complete documentation index at: https://docs.pipecat.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# NVIDIA Riva

> Speech-to-text service implementation using NVIDIA Riva

## Overview

NVIDIA Riva provides two STT service implementations: `NvidiaSTTService` for real-time streaming transcription using Parakeet models, and `NvidiaSegmentedSTTService` for segmented transcription using Canary models with advanced language support and enterprise-grade accuracy.

<CardGroup cols={2}>
  <Card title="NVIDIA Riva STT API Reference" icon="code" href="https://reference-server.pipecat.ai/en/latest/api/pipecat.services.riva.stt.html">
    Pipecat's API methods for NVIDIA Riva STT integration
  </Card>

  <Card title="Example Implementation" icon="play" href="https://github.com/pipecat-ai/pipecat/blob/main/examples/foundational/07r-interruptible-riva-nim.py">
    Complete example with NVIDIA services integration
  </Card>

  <Card title="NVIDIA Riva Documentation" icon="book" href="https://docs.nvidia.com/deeplearning/riva/user-guide/docs/asr/asr-overview.html">
    Official NVIDIA Riva ASR documentation
  </Card>

  <Card title="NVIDIA Developer Portal" icon="microphone" href="https://developer.nvidia.com">
    Access API keys and Riva services
  </Card>
</CardGroup>

## Installation

To use NVIDIA Riva services, install the required dependency:

```bash  theme={null}
pip install "pipecat-ai[nvidia]"
```

## Prerequisites

### NVIDIA Riva Setup

Before using NVIDIA Riva STT services, you need:

1. **NVIDIA Developer Account**: Sign up at [NVIDIA Developer Portal](https://developer.nvidia.com)
2. **API Key**: Generate an NVIDIA API key for Riva services
3. **Model Selection**: Choose between Parakeet (streaming) and Canary (segmented) models

### Required Environment Variables

* `NVIDIA_API_KEY`: Your NVIDIA API key for authentication

---

# Source: https://docs.pipecat.ai/server/frameworks/rtvi/rtvi-observer.md

> ## Documentation Index
> Fetch the complete documentation index at: https://docs.pipecat.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# RTVI Observer

> Converting pipeline frames to RTVI protocol messages

The `RTVIObserver` translates Pipecat's internal pipeline events into standardized RTVI protocol messages. It monitors frame flow through the pipeline and generates corresponding client messages based on event types.

## Purpose

The `RTVIObserver` primarily serves to convert internal pipeline frames into to client-compatible RTVI messages. It is required for any application using RTVI as the client protocol to ensure proper communication of events such as speech start/stop, user transcript, bot output, metrics, and server messages.

## Automatic Setup

`RTVIObserver` is automatically created and attached when you create a `PipelineTask`. No manual setup is required for standard usage.

To customize the observer's behavior, pass `RTVIObserverParams` to the task:

```python  theme={null}
from pipecat.processors.frameworks.rtvi import RTVIObserverParams

task = PipelineTask(
    pipeline,
    rtvi_observer_params=RTVIObserverParams(
        bot_llm_enabled=False,
        metrics_enabled=False,
    ),
)
```

## Configuration

`RTVIObserverParams` accepts the following fields:

<ParamField path="bot_output_enabled" default="True">
  Indicates if bot output messages should be sent.
</ParamField>

<ParamField path="bot_llm_enabled" default="True">
  Indicates if the bot's LLM messages should be sent.
</ParamField>

<ParamField path="bot_tts_enabled" default="True">
  Indicates if the bot's TTS messages should be sent.
</ParamField>

<ParamField path="bot_speaking_enabled" default="True">
  Indicates if the bot's started/stopped speaking messages should be sent.
</ParamField>

<ParamField path="bot_audio_level_enabled" default="False">
  Indicates if bot's audio level messages should be sent.
</ParamField>

<ParamField path="user_llm_enabled" default="True">
  Indicates if the user's LLM input messages should be sent.
</ParamField>

<ParamField path="user_speaking_enabled" default="True">
  Indicates if the user's started/stopped speaking messages should be sent.
</ParamField>

<ParamField path="user_transcription_enabled" default="True">
  Indicates if user's transcription messages should be sent.
</ParamField>

<ParamField path="user_audio_level_enabled" default="False">
  Indicates if user's audio level messages should be sent.
</ParamField>

<ParamField path="metrics_enabled" default="True">
  Indicates if metrics messages should be sent.
</ParamField>

<ParamField path="system_logs_enabled" default="False">
  Indicates if system logs should be sent.
</ParamField>

<ParamField path="errors_enabled" default="True">
  ⚠️ **Deprecated**: Indicates if errors messages should be sent.
</ParamField>

<ParamField path="skip_aggregator_types" default="None">
  List of aggregation types to skip sending as tts/output messages.

  <Note>
    {" "}

    If using this to avoid sending secure information, be sure to also disable bot\_llm\_enabled
    to avoid leaking through LLM messages.
  </Note>
</ParamField>

<ParamField path="bot_output_transforms" default="None">
  A list of tuples to transform text just before sending it to TTS. Each tuple should be of the form `(aggregation_type, transform_function)`, where `aggregation_type` is a string (or `'*'` for all types), and `transform_function` is a callable that takes `(text, aggregation_type)` and returns the transformed text.

  **Example:**

  ```python  theme={null}
  def redact_sensitive(text, agg_type):
      # Example: redact numbers
      import re
      return re.sub(r"\d+", "[REDACTED]", text)

  bot_output_transforms = [
      ("credit_card", redact_sensitive),  # Only for 'credit_card' type
      ("*", lambda text, agg_type: text.upper()),  # For all types, make uppercase
  ]

  observer = RTVIObserver(
      rtvi,
      params=RTVIObserverParams(bot_output_transforms=bot_output_transforms),
  )
  ```
</ParamField>

<ParamField path="audio_level_period_secs" default="0.15">
  How often audio levels should be sent if enabled.
</ParamField>

## Frame Translation

The observer maps Pipecat's internal frames to RTVI protocol messages:

| Pipeline Frame              | RTVI Message                                |
| --------------------------- | ------------------------------------------- |
| **Speech Events**           |                                             |
| `UserStartedSpeakingFrame`  | `RTVIUserStartedSpeakingMessage`            |
| `UserStoppedSpeakingFrame`  | `RTVIUserStoppedSpeakingMessage`            |
| `BotStartedSpeakingFrame`   | `RTVIBotStartedSpeakingMessage`             |
| `BotStoppedSpeakingFrame`   | `RTVIBotStoppedSpeakingMessage`             |
| **Transcription**           |                                             |
| `TranscriptionFrame`        | `RTVIUserTranscriptionMessage(final=true)`  |
| `InterimTranscriptionFrame` | `RTVIUserTranscriptionMessage(final=false)` |
| **Bot Output**              |                                             |
| `AggregatedTextFrame`       | `RTVIBotOutputMessage`                      |
| **LLM Processing**          |                                             |
| `LLMFullResponseStartFrame` | `RTVIBotLLMStartedMessage`                  |
| `LLMFullResponseEndFrame`   | `RTVIBotLLMStoppedMessage`                  |
| `LLMTextFrame`              | `RTVIBotLLMTextMessage`                     |
| **TTS Events**              |                                             |
| `TTSStartedFrame`           | `RTVIBotTTSStartedMessage`                  |
| `TTSStoppedFrame`           | `RTVIBotTTSStoppedMessage`                  |
| `TTSTextFrame`              | `RTVIBotTTSTextMessage`                     |
| **Context/Metrics**         |                                             |
| `LLMContextFrame`           | `RTVIUserLLMTextMessage`                    |
| `MetricsFrame`              | `RTVIMetricsMessage`                        |
| `RTVIServerMessageFrame`    | `RTVIServerMessage`                         |

---

# Source: https://docs.pipecat.ai/server/frameworks/rtvi/rtvi-processor.md

> ## Documentation Index
> Fetch the complete documentation index at: https://docs.pipecat.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# RTVIProcessor

> Core coordinator for RTVI protocol communication

The `RTVIProcessor` manages bidirectional communication between clients and your Pipecat application. It processes client messages, handles service configuration, executes actions, and coordinates function calls.

## Initialization

`RTVIProcessor` is automatically added to your pipeline when you create a `PipelineTask`. Access it via `task.rtvi`:

```python  theme={null}
pipeline = Pipeline([
    transport.input(),
    stt,
    # ... other processors ...
    transport.output()
])

task = PipelineTask(pipeline)

# Access the processor
rtvi = task.rtvi
```

To provide a custom processor (e.g., for [Google RTVI](./google-rtvi-observer)):

```python  theme={null}
from pipecat.services.google.rtvi import GoogleRTVIProcessor

task = PipelineTask(pipeline, rtvi_processor=GoogleRTVIProcessor())
```

To disable RTVI entirely, set `enable_rtvi=False`.

## Readiness Protocol

### Client Ready State

Clients indicate readiness by sending a `client-ready` message, triggering the `on_client_ready` event in the processor:

```python  theme={null}
@rtvi.event_handler("on_client_ready")
async def on_client_ready(rtvi):
    # Handle client ready state
    await rtvi.set_bot_ready()
    # Initialize conversation
    await task.queue_frames([...])
```

### Bot Ready State

The server must mark the bot as ready before it can process client messages:

```python  theme={null}
await rtvi.set_bot_ready()
```

When marked ready, the bot sends a response containing:

* RTVI protocol version
* Current service configuration
* Available actions

## Services

Services represent configurable components of your application that clients can interact with.

### Registering Services

```python  theme={null}
# 1. Define option handler
async def handle_voice_option(processor, service, option):
    voice_id = option.value
    # Apply configuration change
    logger.info(f"Voice ID updated to: {voice_id}")

# 2. Create RTVIService
voice_service = RTVIService(
    name="voice",
    options=[
        RTVIServiceOption(
            name="voice_id",
            type="string",
            handler=handle_voice_option
        )
    ]
)

# 3. Register with processor
rtvi.register_service(voice_service)
```

### Option Types

Services support multiple data types for configuration:

```python  theme={null}
RTVIServiceOption(
    name="temperature",
    type="number",  # number, string, bool, array, object
    handler=handle_temperature
)
```

Option handlers receive:

* The processor instance
* The service name
* The option configuration with new value

## Actions

Actions are server-side functions that clients can trigger with arguments.

### Registering Actions

```python  theme={null}
# 1. Define handler function
async def handle_print_message(processor, service, arguments):
    message = arguments.get("message", "Default message")
    logger.info(f"Print action triggered with message: {message}")
    return True

# 2. Create and register RTVIAction
print_action = RTVIAction(
    service="conversation",
    action="print_message",
    arguments=[
        RTVIActionArgument(name="message", type="string")
    ],
    result="bool",
    handler=handle_print_message
)
rtvi.register_action(print_action)
```

### Action Arguments

Actions can accept typed arguments from clients:

```python  theme={null}
search_action = RTVIAction(
    service="knowledge",
    action="search",
    arguments=[
        RTVIActionArgument(name="query", type="string"),
        RTVIActionArgument(name="limit", type="number")
    ],
    result="array",
    handler=handle_search
)
```

## Function Calls

Handle LLM function calls with client interaction:

```python  theme={null}

await processor.handle_function_call(
    function_name=function_name,
    tool_call_id=tool_call_id,
    arguments=arguments,
)

await processor.handle_function_call(params)
```

The function call process:

1. LLM requests a function call
2. Processor notifies client with `llm-function-call` message
3. Client executes function and returns result
4. Result is passed back to LLM via `FunctionCallResultFrame`
5. Conversation continues

## Error Handling

Send error messages to clients:

```python  theme={null}
# General error
await processor.send_error("Invalid configuration")

# Request-specific error
await processor._send_error_response(request_id, "Invalid action arguments")
```

Error categories:

* Configuration errors
* Action execution errors
* Function call errors
* Protocol errors
* Fatal and non-fatal errors

## Bot Control

Manage bot state and handle interruptions:

```python  theme={null}
# Set bot as ready
await processor.set_bot_ready()

# Handle interruptions
await processor.interrupt_bot()
```

## Custom Messaging

Send custom messages from server to client:

```python  theme={null}
from pipecat.processors.frameworks.rtvi import RTVIServerMessageFrame

# Send a custom message
frame = RTVIServerMessageFrame(
    data={
        "type": "custom-event",
        "payload": {"key": "value"}
    }
)
await rtvi.push_frame(frame)
```

The client receives these messages through the `onServerMessage` callback or `serverMessage` event.

---

# Source: https://docs.pipecat.ai/client/rtvi-standard.md

> ## Documentation Index
> Fetch the complete documentation index at: https://docs.pipecat.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# The RTVI Standard

> An overview of the RTVI standard and its key features

The RTVI (Real-Time Voice and Video Inference) standard defines a set of message types and structures sent between clients and servers. It is designed to facilitate real-time interactions between clients and AI applications that require voice, video, and text communication. It provides a consistent framework for building applications that can communicate with AI models and the backends running those models in real-time.

<Note>
  This page documents version 1.0 of the RTVI standard, released in June 2025.
</Note>

## Key Features

<CardGroup cols={2}>
  <Card title="Connection Management" icon="plug">
    RTVI provides a flexible connection model that allows clients to connect to AI services and coordinate state.
  </Card>

  <Card title="Transcriptions" icon="text">
    The standard includes built-in support for real-time transcription of audio streams.
  </Card>

  <Card title="Client-Server Messaging" icon="comments">
    The standard defines a messaging protocol for sending and receiving messages between clients and servers, allowing for efficient communication of requests and responses.
  </Card>

  <Card title="Advanced LLM Interactions" icon="robot">
    The standard supports advanced interactions with large language models (LLMs), including context management, function call handline, and search results.
  </Card>

  <Card title="Service-Specific Insights" icon="lightbulb">
    RTVI supports events to provide insight into the input/output and state for typical services that exist in speech-to-speech workflows.
  </Card>

  <Card title="Metrics and Monitoring" icon="chart-line">
    RTVI provides mechanisms for collecting metrics and monitoring the performance of server-side services.
  </Card>
</CardGroup>

## Terms

* **Client**: The front-end application or user interface that interacts with the RTVI server.
* **Server**: The backend-end service that runs the AI framework and processes requests from the client.
* **User**: The end user interacting with the client application.
* **Bot**: The AI interacting with the user, technically an amalgamation of a large language model (LLM) and a text-to-speech (TTS) service.

## RTVI Message Format

The messages defined as part of the RTVI protocol adhere to the following format:

```json  theme={null}
{
    "id": string,
    "label": "rtvi-ai",
    "type": string,
    "data": unknown
}
```

<ParamField path="id" type="string" required={false}>
  A unique identifier for the message, used to correlate requests and responses.
</ParamField>

<ParamField path="label" type="string" required={true} default="rtvi-ai">
  A label that identifies this message as an RTVI message. This field is required and should always be set to `'rtvi-ai'`.
</ParamField>

<ParamField path="type" type="string" required={true}>
  The type of message being sent. This field is required and should be set to one of the predefined RTVI message types listed below.
</ParamField>

<ParamField path="data" type="unknown" required={false}>
  The payload of the message, which can be any data structure relevant to the message type.
</ParamField>

## RTVI Message Types

Following the above format, this section describes the various message types defined by the RTVI standard. Each message type has a specific purpose and structure, allowing for clear communication between clients and servers.

<Tip>
  Each message type below includes either a 🤖 or 🏄 emoji to denote whether the message is sent from the bot (🤖) or client (🏄).
</Tip>

### Connection Management

#### client-ready 🏄

Indicates that the client is ready to receive messages and interact with the server. Typically sent after the transport media channels have connected.

* **type**: `'client-ready'`
* **data**:
  * **version**: `string`

    The version of the RTVI standard being used. This is useful for ensuring compatibility between client and server implementations.

  * **about**: `AboutClient Object`

    An object containing information about the client, such as its rtvi-version, client library, and any other relevant metadata. The `AboutClient` object follows this structure:

    <Expandable title="AboutClient">
      <ParamField path="library" type="string" required />

      <ParamField path="library_version" type="string" />

      <ParamField path="platform" type="string" />

      <ParamField path="platform_version" type="string" />

      <ParamField path="platform_details" type="any">
        Any platform-specific details that may be relevant to the server. This could include information about the browser, operating system, or any other environment-specific data needed by the server. This field is optional and open-ended, so please be mindful of the data you include here and any security concerns that may arise from exposing sensitive or personal-identifiable information.
      </ParamField>
    </Expandable>

#### bot-ready 🤖

Indicates that the bot is ready to receive messages and interact with the client. Typically send after the transport media channels have connected.

* **type**: `'bot-ready'`
* **data**:
  * **version**: `string`

    The version of the RTVI standard being used. This is useful for ensuring compatibility between client and server implementations.

  * **about**: `any` (Optional)

    An object containing information about the server or bot. It's structure and value are both undefined by default. This provides flexibility to include any relevant metadata your client may need to know about the server at connection time, without any built-in security concerns. Please be mindful of the data you include here and any security concerns that may arise from exposing sensitive information.

#### disconnect-bot 🏄

Indicates that the client wishes to disconnect from the bot. Typically used when the client is shutting down or no longer needs to interact with the bot. Note: Disconnets should happen automatically when either the client or bot disconnects from the transport, so this message is intended for the case where a client may want to remain connected to the transport but no longer wishes to interact with the bot.

* **type**: `'disconnect-bot'`
* **data**: `undefined`

#### error 🤖

Indicates an error occurred during bot initialization or runtime.

* **type**: `'error'`
* **data**:
  * **message**: `string`

    Description of the error.

  * **fatal**: `boolean`

    Indicates if the error is fatal to the session.

### Transcription

#### user-started-speaking 🤖

Emitted when the user begins speaking

* **type**: `'user-started-speaking'`
* **data**: None

#### user-stopped-speaking 🤖

Emitted when the user stops speaking

* **type**: `'user-stopped-speaking'`
* **data**: None

#### bot-started-speaking 🤖

Emitted when the bot begins speaking

* **type**: `'bot-started-speaking'`
* **data**: None

#### bot-stopped-speaking 🤖

Emitted when the bot stops speaking

* **type**: `'bot-stopped-speaking'`
* **data**: None

#### user-transcription 🤖

Real-time transcription of user speech, including both partial and final results.

* **type**: `'user-transcription'`
* **data**:
  * **text**: `string`

    The transcribed text of the user.

  * **final**: `boolean`

    Indicates if this is a final transcription or a partial result.

  * **timestamp**: `string`

    The timestamp when the transcription was generated.

  * **user\_id**: `string`

    Identifier for the user who spoke.

#### bot-output 🤖

The best-effort representation of the bot's output text, including both spoken and unspoken text. In addition to transcriptions of spoken text, this message type may also include text that the bot outputs but does not speak (e.g., text sent to the client for display purposes only). Along with the text, this event includes a `spoken` flag to indicate whether the text was spoken by the bot or not and an `aggregated_by` field to indicate what the text represents (e.g. "sentence", "word", "code", "url").

* **type**: `'bot-output'`
* **data**:
  * **text**: `string`

    The output text from the bot.

  * **spoken**: `boolean`

    Indicates if this text was spoken by the bot.

  * **aggregated\_by**: `string`

    Indicates how the text was aggregated (e.g., "sentence", "word", "code", "url"). "sentence" and "word" are reserved aggregation types defined by the RTVI standard. Other aggregation types may be defined by custom text aggregators used by the server.

#### bot-transcription 🤖

<Warning>DEPRECATED in favor of `bot-output` in Pipecat version 0.0.95 and client-js version 1.5.0</Warning>

Transcription of the bot's speech. Note: This protocol currently does not match the user transcription format to support real-time timestamping for bot transcriptions. Rather, the event is typically sent for each sentence of the bot's response. This difference is currently due to limitations in TTS services which mostly do not support (or support well), accurate timing information. If/when this changes, this protocol may be updated to include the necessary timing information. For now, if you want to attempt real-time transcription to match your bot's speaking, you can try using the `bot-tts-text` message type.

* **type**: `'bot-transcription'`
* **data**:
  * **text**: `string`

    The transcribed text from the bot, typically aggregated at a per-sentence level.

### Client-Server Messaging

#### server-message 🤖

An arbitrary message sent from the server to the client. This can be used for custom interactions or commands. This message may be coupled with the `client-message` message type to handle responses from the client.

* **type**: `'server-message'`
* **data**: `any`

  The `data` can be any JSON-serializable object, formatted according to your own specifications.

#### client-message 🏄

An arbitrary message sent from the client to the server. This can be used for custom interactions or commands. This message may be coupled with the `server-response` message type to handle responses from the server.

* **type**: `'client-message'`
* **data**:

  * **t**: `string`
  * **d**: `unknown` (optional)

  The data payload should contain a `t` field indicating the type of message and an optional `d` field containing any custom, corresponding data needed for the message.

#### server-response 🤖

An message sent from the server to the client in response to a `client-message`. **IMPORTANT**: The `id` should match the `id` of the original `client-message` to correlate the response with the request.

* **type**: `'client-message'`
* **data**:

  * **t**: `string`
  * **d**: `unknown` (optional)

  The data payload should contain a `t` field indicating the type of message and an optional `d` field containing any custom, corresponding data needed for the message.

#### error-response 🤖

Error response to a specific client message. **IMPORTANT**: The `id` should match the `id` of the original `client-message` to correlate the response with the request.

* **type**: `'error-response'`
* **data**:
  * **error**: `string`

### Advanced LLM Interactions

#### send-text 🏄

A message sent from the client to the server to send text input to the LLM, appended to the user's context.

* **type**: `'send-text'`
* **data**:
  * **content**: `string`

    The text content to be appended to the user context.

  * **options**: `object` (optional)
    * **run\_immediately**: `boolean` (optional)

      If `true`, the pipeline should be interrupted and the LLM should process the input immediately after appending it to the context. Defaults to `true`.

    * **audio\_response**: `boolean` (optional)

      If `false`, the bot should bypass the TTS so that the bot does not respond to the text in audio. Defaults to `true`.

#### append-to-context 🏄

<Warning>DEPRECATED in favor of `send-text` in Pipecat version 0.0.86 and client-js version 0.0.4</Warning>

A message sent from the client to the server to append data to the context of the current llm conversation. This is useful for providing text-based content for the user or augmenting the context for the assistant.

* **type**: `'append-to-context'`
* **data**:
  * **role**: `"user"` | `"assistant"`

    The role the context should be appended to. Currently only supports `"user"` and `"assistant"`.

  * **content**: `unknown`

    The content to append to the context. This can be any data structure the llm understand.

  * **run\_immediately**: `boolean` (optional)

    Indicates whether the context should be run immediately after appending. Defaults to `false`. If set to `false`, the context will be appended but not executed until the next llm run.

#### llm-function-call 🤖

A function call request from the LLM, sent from the bot to the client. Note that for most cases, an LLM function call will be handled completely server-side. However, in the event that the call requires input from the client or the client needs to be aware of the function call, this message/response schema is required.

* **type**: `'llm-function-call'`
* **data**:
  * **function\_name**: `string`

    Name of the function to be called.

  * **tool\_call\_id**: `string`

    Unique identifier for this function call.

  * **args**: `Record<string, unknown>`

    Arguments to be passed to the function.

#### llm-function-call-result 🏄

The result of the function call requested by the LLM, returned from the client.

* **type**: `'llm-function-call-result'`
* **data**:
  * **function\_name**: `string`

    Name of the called function.

  * **tool\_call\_id**: `string`

    Identifier matching the original function call.

  * **args**: `Record<string, unknown>`

    Arguments that were passed to the function.

  * **result**: `Record<string, unknown> | string`

    The result returned by the function.

#### bot-llm-search-response 🤖

Search results from the LLM's knowledge base.

<Note>
  Currently, Google Gemini is the only LLM that supports built-in search. However, we expect other LLMs to follow suite, which is why this message type is defined as part of the RTVI standard. As more LLMs add support for this feature, the format of this message type may evolve to accommodate discrepancies.
</Note>

* **type**: `'bot-llm-search-response'`
* **data**:
  * **search\_result**: `string` (optional)

    Raw search result text.

  * **rendered\_content**: `string` (optional)

    Formatted version of the search results.

  * **origins**: `Array<Origin Object>`

    Source information and confidence scores for search results. The `Origin Object` follows this structure:

    ```json  theme={null}
    {
        "site_uri": string (optional),
        "site_title": string (optional),
        "results": Array<{
            "text": string,
            "confidence": number[]
        }>
    }
    ```

**Example:**

```json  theme={null}
"id": undefined
"label": "rtvi-ai"
"type": "bot-llm-search-response"
"data": {
    "origins": [
    {
        "results": [
            {
                "confidence": [0.9881149530410768],
                "text": "* Juneteenth: A Freedom Celebration is scheduled for June 18th from 12:00 pm to 2:00 pm."
            },
            {
                "confidence": [0.9692034721374512],
                "ext": "* A Juneteenth celebration at Fort Negley Park will take place on June 19th from 5:00 pm to 9:30 pm."
            }
        ],
        "site_title": "vanderbilt.edu",
        "site_uri": "https://vertexaisearch.cloud.google.com/grounding-api-redirect/AUZIYQHwif83VK9KAzrbMSGSBsKwL8vWfSfC9pgEWYKmStHyqiRoV1oe8j1S0nbwRg_iWgqAr9wUkiegu3ATC8Ll-cuE-vpzwElRHiJ2KgRYcqnOQMoOeokVpWqi"
    },
    {
        "results": [
            {
                "confidence": [0.6554043292999268],
                "text": "In addition to these events, Vanderbilt University is a large research institution with ongoing activities across many fields."
            }
        ],
        "site_title": "wikipedia.org",
        "site_uri": "https://vertexaisearch.cloud.google.com/grounding-api-redirect/AUZIYQESbF-ijx78QbaglrhflHCUWdPTD4M6tYOQigW5hgsHNctRlAHu9ktfPmJx7DfoP5QicE0y-OQY1cRl9w4Id0btiFgLYSKIm2-SPtOHXeNrAlgA7mBnclaGrD7rgnLIbrjl8DgUEJrrvT0CKzuo"
    }],
    "rendered_content": "<style>\n.container ... </div>\n</div>\n",
    "search_result": "Several events are happening at Vanderbilt University:\n\n* Juneteenth: A Freedom Celebration is scheduled for June 18th from 12:00 pm to 2:00 pm.\n* A Juneteenth celebration at Fort Negley Park will take place on June 19th from 5:00 pm to 9:30 pm.\n\n  In addition to these events, Vanderbilt University is a large research institution with ongoing activities across many fields.  For the most recent news, you should check Vanderbilt's official news website.\n"
}
```

### Service-Specific Insights

#### bot-llm-started 🤖

Indicates LLM processing has begun

* **type**: `bot-llm-started`
* **data**: None

#### bot-llm-stopped 🤖

Indicates LLM processing has completed

* **type**: `bot-llm-stopped`
* **data**: None

#### user-llm-text 🤖

Aggregated user input text that is sent to the LLM.

* **type**: `'user-llm-text'`
* **data**:
  * **text**: `string`

    The user's input text to be processed by the LLM.

#### bot-llm-text 🤖

Individual tokens streamed from the LLM as they are generated.

* **type**: `'bot-llm-text'`
* **data**:
  * **text**: `string`

    The token text from the LLM.

#### bot-tts-started 🤖

Indicates text-to-speech (TTS) processing has begun.

* **type**: `'bot-tts-started'`
* **data**: None

#### bot-tts-stopped 🤖

Indicates text-to-speech (TTS) processing has completed.

* **type**: `'bot-tts-stopped'`
* **data**: None

#### bot-tts-text 🤖

The per-token text output of the text-to-speech (TTS) service (what the TTS actually says).

* **type**: `'bot-tts-text'`
* **data**:
  * **text**: `string`

    The text representation of the generated bot speech.

### Metrics and Monitoring

#### metrics 🤖

Performance metrics for various processing stages and services. Each message will contain entries for one or more of the metrics types: `processing`, `ttfb`, `characters`.

* **type**: `'metrics'`
* **data**:
  * **processing**: \[See Below] (optional)

    Processing time metrics.

  * **ttfb**:  \[See Below] (optional)

    Time to first byte metrics.

  * **characters**:  \[See Below] (optional)

    Character processing metrics.

For each metric type, the data structure is an array of objects with the following structure:

* **processor**: `string`

  The name of the processor or service that generated the metric.

* **value**: `number`

  The value of the metric, typically in milliseconds or character count.

* **model**: `string` (optional)

  The model of the service that generated the metric, if applicable.

**Example:**

```json  theme={null}
{
  "type": "metrics",
  "data": {
    "processing": [
      {
        "model": "eleven_flash_v2_5",
        "processor": "ElevenLabsTTSService#0",
        "value": 0.0005140304565429688
      }
    ],
    "ttfb": [
      {
        "model": "eleven_flash_v2_5",
        "processor": "ElevenLabsTTSService#0",
        "value": 0.1573178768157959
      }
    ],
    "characters": [
      {
        "model": "eleven_flash_v2_5",
        "processor": "ElevenLabsTTSService#0",
        "value": 43
      }
    ]
  }
}
```

---

# Source: https://docs.pipecat.ai/server/services/stt/sambanova.md

# Source: https://docs.pipecat.ai/server/services/llm/sambanova.md

> ## Documentation Index
> Fetch the complete documentation index at: https://docs.pipecat.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# SambaNova

> LLM service implementation using SambaNova's API with OpenAI-compatible interface

## Overview

`SambaNovaLLMService` provides access to SambaNova's language models through an OpenAI-compatible interface. It inherits from `OpenAILLMService` and supports streaming responses, function calling, and context management with SambaNova's high-performance inference platform.

<CardGroup cols={2}>
  <Card title="SambaNova LLM API Reference" icon="code" href="https://reference-server.pipecat.ai/en/latest/api/pipecat.services.sambanova.llm.html">
    Pipecat's API methods for SambaNova integration
  </Card>

  <Card title="Example Implementation" icon="play" href="https://github.com/pipecat-ai/pipecat/blob/main/examples/foundational/14s-function-calling-sambanova.py">
    Complete example with function calling
  </Card>

  <Card title="SambaNova Documentation" icon="book" href="https://docs.sambanova.ai/">
    Official SambaNova API documentation and features
  </Card>

  <Card title="SambaNova Cloud" icon="microphone" href="https://cloud.sambanova.ai/?utm_source=pipecat&utm_medium=external&utm_campaign=cloud_signup">
    Access models and manage API keys
  </Card>
</CardGroup>

## Installation

To use SambaNova services, install the required dependencies:

```bash  theme={null}
pip install "pipecat-ai[sambanova]"
```

## Prerequisites

### SambaNova Account Setup

Before using SambaNova LLM services, you need:

1. **SambaNova Account**: Sign up at [SambaNova Cloud](https://cloud.sambanova.ai/?utm_source=pipecat\&utm_medium=external\&utm_campaign=cloud_signup)
2. **API Key**: Generate an API key from your account dashboard
3. **Model Selection**: Choose from available high-performance models

### Required Environment Variables

* `SAMBANOVA_API_KEY`: Your SambaNova API key for authentication

---

# Source: https://docs.pipecat.ai/server/services/tts/sarvam.md

# Source: https://docs.pipecat.ai/server/services/stt/sarvam.md

> ## Documentation Index
> Fetch the complete documentation index at: https://docs.pipecat.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Sarvam

> Speech-to-text service implementation using Sarvam AI's WebSocket-based streaming API

## Overview

`SarvamSTTService` provides real-time speech recognition using Sarvam AI's WebSocket API, supporting Indian language transcription with Voice Activity Detection (VAD) and multiple audio formats for high-accuracy speech recognition.

<CardGroup cols={2}>
  <Card title="Sarvam STT API Reference" icon="code" href="https://reference-server.pipecat.ai/en/latest/api/pipecat.services.sarvam.stt.html">
    Pipecat's API methods for Sarvam STT integration
  </Card>

  <Card title="Example Implementation" icon="play" href="https://github.com/pipecat-ai/pipecat/blob/main/examples/foundational/07z-interruptible-sarvam.py">
    Complete example with interruption handling
  </Card>

  <Card title="Sarvam Documentation" icon="book" href="https://docs.sarvam.ai/api-reference-docs/api-guides-tutorials/speech-to-text/overview">
    Official Sarvam AI STT documentation and features
  </Card>

  <Card title="Sarvam AI Platform" icon="microphone" href="https://dashboard.sarvam.ai/">
    Access API keys and speech models
  </Card>
</CardGroup>

## Installation

To use Sarvam services, install the required dependency:

```bash  theme={null}
pip install "pipecat-ai[sarvam]"
```

## Prerequisites

### Sarvam AI Account Setup

Before using Sarvam STT services, you need:

1. **Sarvam AI Account**: Sign up at [Sarvam AI](https://dashboard.sarvam.ai/)
2. **API Key**: Generate an API key from your account dashboard
3. **Model Access**: Access to Saarika (STT) or Saaras (STT-Translate) models

### Required Environment Variables

* `SARVAM_API_KEY`: Your Sarvam AI API key for authentication

---

# Source: https://docs.pipecat.ai/guides/fundamentals/saving-transcripts.md

> ## Documentation Index
> Fetch the complete documentation index at: https://docs.pipecat.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Saving Conversation Transcripts

> Learn how to collect and save conversation transcripts between users and your bot

## Overview

Recording transcripts of conversations between users and your bot is useful for debugging, analysis, and creating a record of interactions. Pipecat's turn events make it easy to collect both user and assistant messages as they occur.

## How It Works

Transcripts are collected using turn events on the context aggregators:

1. Capturing what the user says via `on_user_turn_stopped`
2. Capturing what the assistant says via `on_assistant_turn_stopped`
3. Each event provides the complete transcript for that turn
4. Allowing you to handle these events with custom logic

<Note>
  Turn events are emitted by the context aggregators (`LLMUserAggregator` and
  `LLMAssistantAggregator`), which are created as part of the
  `LLMContextAggregatorPair`.
</Note>

## Basic Implementation

### Step 1: Create Context Aggregators

First, create the context aggregator pair and get references to both aggregators:

```python  theme={null}
from pipecat.processors.aggregators.llm_response_universal import (
    LLMContextAggregatorPair,
    UserTurnStoppedMessage,
    AssistantTurnStoppedMessage,
)

# Create context aggregator pair
user_aggregator, assistant_aggregator = LLMContextAggregatorPair(context)
```

### Step 2: Add to Your Pipeline

Include the aggregators in your pipeline:

```python  theme={null}
pipeline = Pipeline(
    [
        transport.input(),
        stt,                              # Speech-to-text
        user_aggregator,
        llm,
        tts,                              # Text-to-speech
        transport.output(),
        assistant_aggregator,
    ]
)
```

### Step 3: Handle Turn Events

Register event handlers to capture transcripts when turns complete:

```python  theme={null}
@user_aggregator.event_handler("on_user_turn_stopped")
async def on_user_turn_stopped(aggregator, strategy, message: UserTurnStoppedMessage):
    print(f"[{message.timestamp}] user: {message.content}")

@assistant_aggregator.event_handler("on_assistant_turn_stopped")
async def on_assistant_turn_stopped(aggregator, message: AssistantTurnStoppedMessage):
    print(f"[{message.timestamp}] assistant: {message.content}")
```

<Tip>
  In addition to console logging, you can save transcripts to a database or file
  for later analysis.
</Tip>

## Next Steps

<CardGroup cols={2}>
  <Card title="Turn Events Reference" icon="book" iconType="duotone" href="/server/utilities/turn-management/turn-events">
    Learn about all available turn events and their parameters.
  </Card>

  <Card title="Transcriptions Reference" icon="scroll" iconType="duotone" href="/server/utilities/turn-management/transcriptions">
    See more examples for collecting and processing transcriptions.
  </Card>
</CardGroup>

Consider implementing transcript recording in your application for debugging during development and preserving important conversations in production. The transcript data can also be useful for analyzing conversation patterns and improving your bot's responses over time.

---

# Source: https://docs.pipecat.ai/deployment/pipecat-cloud/fundamentals/scaling.md

> ## Documentation Index
> Fetch the complete documentation index at: https://docs.pipecat.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Scaling

> Production-ready agent deployment and scaling strategies

Agentic applications demand near-instant agent availability to maintain user engagement and responsiveness. Pipecat Cloud manages the complexity of scaling these agent deployments in production environments, providing granular controls for compute resources and cost optimization.

## Core concepts

### Instances

An instance represents a single unit of compute that runs your agent.

Instance costs are determined by:

* [Active session](./active-sessions) runtime duration
* Warm instance maintenance time
* The compute profile specified in your deployment

Pipecat Cloud automatically provisions and manages agent instances to handle active sessions, ensuring that your deployment can scale to meet demand
within the limits of your deployment configuration.

### Instance pool

Making a deployment to Pipecat Cloud creates a managed pool of agent instances that:

* Routes requests to available agent instances
* Scales based on demand within configured limits
* Maintains optimal performance through auto-scaling

<Frame>
  <img src="https://mintcdn.com/daily/2bYrACcmgvvzC075/deployment/pipecat-cloud/images/scaling-pools.png?fit=max&auto=format&n=2bYrACcmgvvzC075&q=85&s=97e4482a7d258dce7916dfcae1fd5ae4" data-og-width="1661" width="1661" data-og-height="778" height="778" data-path="deployment/pipecat-cloud/images/scaling-pools.png" data-optimize="true" data-opv="3" srcset="https://mintcdn.com/daily/2bYrACcmgvvzC075/deployment/pipecat-cloud/images/scaling-pools.png?w=280&fit=max&auto=format&n=2bYrACcmgvvzC075&q=85&s=b38d2adc37be20e3c761699adff9916b 280w, https://mintcdn.com/daily/2bYrACcmgvvzC075/deployment/pipecat-cloud/images/scaling-pools.png?w=560&fit=max&auto=format&n=2bYrACcmgvvzC075&q=85&s=55ee4017da367d8814575d77d6d0ce22 560w, https://mintcdn.com/daily/2bYrACcmgvvzC075/deployment/pipecat-cloud/images/scaling-pools.png?w=840&fit=max&auto=format&n=2bYrACcmgvvzC075&q=85&s=803fc3826f172330a3e6ce6b09cef45e 840w, https://mintcdn.com/daily/2bYrACcmgvvzC075/deployment/pipecat-cloud/images/scaling-pools.png?w=1100&fit=max&auto=format&n=2bYrACcmgvvzC075&q=85&s=89b5def1129633a4c8bb07c6da401f8f 1100w, https://mintcdn.com/daily/2bYrACcmgvvzC075/deployment/pipecat-cloud/images/scaling-pools.png?w=1650&fit=max&auto=format&n=2bYrACcmgvvzC075&q=85&s=a16e6e9873908f6b1891a6ad77042683 1650w, https://mintcdn.com/daily/2bYrACcmgvvzC075/deployment/pipecat-cloud/images/scaling-pools.png?w=2500&fit=max&auto=format&n=2bYrACcmgvvzC075&q=85&s=9124b6598dfa1472794e288fa596f287 2500w" />
</Frame>

Developers can configure the upper and lower limit of a deployment's instance pool, providing a cost-effective way to handle varying loads.

### Minimum agents

`min-agents`

* Maintains specified number of warm agent instances to serve incoming requests
* Immediately ready to become active, reducing cold starts
* Defaults to `0` if unspecified

<Frame caption="Minimum number of agent instances maintained in a pool at all times.">
  <img src="https://mintcdn.com/daily/2bYrACcmgvvzC075/deployment/pipecat-cloud/images/scaling-min-instances.png?fit=max&auto=format&n=2bYrACcmgvvzC075&q=85&s=6f4d35f3d092ad22453232b2a465c9b5" data-og-width="1400" width="1400" data-og-height="778" height="778" data-path="deployment/pipecat-cloud/images/scaling-min-instances.png" data-optimize="true" data-opv="3" srcset="https://mintcdn.com/daily/2bYrACcmgvvzC075/deployment/pipecat-cloud/images/scaling-min-instances.png?w=280&fit=max&auto=format&n=2bYrACcmgvvzC075&q=85&s=8b3fee0482d1a92cdfb574e762bcfdb7 280w, https://mintcdn.com/daily/2bYrACcmgvvzC075/deployment/pipecat-cloud/images/scaling-min-instances.png?w=560&fit=max&auto=format&n=2bYrACcmgvvzC075&q=85&s=250af6e4f4f5384e67af85b1f6ff882f 560w, https://mintcdn.com/daily/2bYrACcmgvvzC075/deployment/pipecat-cloud/images/scaling-min-instances.png?w=840&fit=max&auto=format&n=2bYrACcmgvvzC075&q=85&s=31c4b901899fff68700416f6128226d5 840w, https://mintcdn.com/daily/2bYrACcmgvvzC075/deployment/pipecat-cloud/images/scaling-min-instances.png?w=1100&fit=max&auto=format&n=2bYrACcmgvvzC075&q=85&s=e9c2cfa2e6b2b6b24ddffb08043691c5 1100w, https://mintcdn.com/daily/2bYrACcmgvvzC075/deployment/pipecat-cloud/images/scaling-min-instances.png?w=1650&fit=max&auto=format&n=2bYrACcmgvvzC075&q=85&s=e86d55d222fbbd093bb3fe7c6e7c70d8 1650w, https://mintcdn.com/daily/2bYrACcmgvvzC075/deployment/pipecat-cloud/images/scaling-min-instances.png?w=2500&fit=max&auto=format&n=2bYrACcmgvvzC075&q=85&s=8b6cf6df8780bdf840de84a9e6301335 2500w" />
</Frame>

Developers specify a `min-agents` configuration to determines the number of agent instances that should be kept warm in their deployment pool. A warm instance is kept running and can immediately be used to serve an active session.

Maintaining a minimum number of agent instances is important to keep agent start times fast and reduce [cold starts](./scaling#cold-starts).

```shell  theme={null}
# Defaults to zero
pipecat cloud deploy [agent-name]

# Maintain 1 warm instance at all times
pipecat cloud deploy [agent-name] --min-agents 1
```

<Warning>
  Setting `--min-agents` to 1 or greater will incur charges even when the agent
  is not in use.
</Warning>

### Maximum agents

`max-agents`

* Sets hard limit on concurrent sessions
* Acts as a cost control / load mechanism
* Returns HTTP 429 when pool capacity is reached

<Frame caption="Maximum agents is the hard limit on the number of agents in your pool.">
  <img src="https://mintcdn.com/daily/2bYrACcmgvvzC075/deployment/pipecat-cloud/images/scaling-max-instances.png?fit=max&auto=format&n=2bYrACcmgvvzC075&q=85&s=b7bc369d9b4c08676d8858513385c089" data-og-width="1400" width="1400" data-og-height="778" height="778" data-path="deployment/pipecat-cloud/images/scaling-max-instances.png" data-optimize="true" data-opv="3" srcset="https://mintcdn.com/daily/2bYrACcmgvvzC075/deployment/pipecat-cloud/images/scaling-max-instances.png?w=280&fit=max&auto=format&n=2bYrACcmgvvzC075&q=85&s=9da4b371c48be68a0e2e2727c53cc446 280w, https://mintcdn.com/daily/2bYrACcmgvvzC075/deployment/pipecat-cloud/images/scaling-max-instances.png?w=560&fit=max&auto=format&n=2bYrACcmgvvzC075&q=85&s=a5e8a1c8b66059f2ddfd652c22ec640e 560w, https://mintcdn.com/daily/2bYrACcmgvvzC075/deployment/pipecat-cloud/images/scaling-max-instances.png?w=840&fit=max&auto=format&n=2bYrACcmgvvzC075&q=85&s=c523c0bba63e5e740f666e3751f7c289 840w, https://mintcdn.com/daily/2bYrACcmgvvzC075/deployment/pipecat-cloud/images/scaling-max-instances.png?w=1100&fit=max&auto=format&n=2bYrACcmgvvzC075&q=85&s=3f65bf67da5f18c3dffcf7d445c97bee 1100w, https://mintcdn.com/daily/2bYrACcmgvvzC075/deployment/pipecat-cloud/images/scaling-max-instances.png?w=1650&fit=max&auto=format&n=2bYrACcmgvvzC075&q=85&s=0541f4ae8e7edc3c93e207ab83075520 1650w, https://mintcdn.com/daily/2bYrACcmgvvzC075/deployment/pipecat-cloud/images/scaling-max-instances.png?w=2500&fit=max&auto=format&n=2bYrACcmgvvzC075&q=85&s=9340a249030f4c2e3a21d7de50026d53 2500w" />
</Frame>

<Note>
  Each deployment made to Pipecat Cloud has a maximum allowed pool
  size of 50. Please contact us at [help@daily.co](mailto:help@daily.co) or via
  [Discord](https://discord.gg/dailyco) if you require more capacity.
</Note>

Deployments can optionally be made with a `max-agents` configuration that limits the number of agent instances that your pool can contain.
This exists as a cost control measure, allowing developers to limit the total number of active sessions that can be run at any one time.

The maximum instance count is a hard limit, meaning requests made to a pool that is at capacity will receive a `429` response. See [starting sessions](./active-sessions#agent-capacity) for more information for how to handle this in your application code.

```shell  theme={null}
# Limit pool to 25 active sessions running concurrently
pipecat cloud deploy [agent-name] --max-agents 25
```

### Agent lifecycle

<Steps>
  <Step title="Pool Initialization">
    * Provisions the minimum number of warm agent instances based on `min-agents` configuration (defaults to `0`)
    * Listens for session requests to route to available agent instances
  </Step>

  <Step title="Session Assignment">
    * ✅ If a warm instance is available, the session will be assigned to that instance
    * ⏳ If no warm agent instances are available, and your pool is not at capacity, a new instance will be provisioned to handle the request (e.g a cold-start)
    * ❌ If your pool is at capacity, your application will receive a `429` response from the start request
  </Step>

  <Step title="Auto-scaling">
    * The Pipecat Cloud auto-scaler determines if additional warm agent instances should be created to support further requests.
    * Once a session concludes, agent instances are either returned to the pool to serve another session or discarded
  </Step>
</Steps>

<Note>
  <b>Your are billed for warm agent instances</b>, even if they are not handling
  active sessions. Developers should consider their deployment strategy when
  cost optimizing, adjusting the minimum and maximum instance count accordingly.
  See [current pricing](https://www.daily.co/pricing/pipecat-cloud) for details.
</Note>

## Cold-starts

A cold start may occur when an active session request is made and no warm agent instances are available in the pool to handle it. In this case, Pipecat Cloud will provision a new instance to handle the request.

Cold starts require additional time to provision the instance and load the agent instances, which may result in a delay for the user. To minimize cold starts, you can configure your pool to maintain a minimum number of warm agent instances at all times.

<Frame>
  <img src="https://mintcdn.com/daily/2bYrACcmgvvzC075/deployment/pipecat-cloud/images/scaling-cold-starts.png?fit=max&auto=format&n=2bYrACcmgvvzC075&q=85&s=d05a17b5f2ab8684d795fb3fc6ab13bc" data-og-width="1400" width="1400" data-og-height="778" height="778" data-path="deployment/pipecat-cloud/images/scaling-cold-starts.png" data-optimize="true" data-opv="3" srcset="https://mintcdn.com/daily/2bYrACcmgvvzC075/deployment/pipecat-cloud/images/scaling-cold-starts.png?w=280&fit=max&auto=format&n=2bYrACcmgvvzC075&q=85&s=07d3edfcd6f58b64f67089a23cfd474f 280w, https://mintcdn.com/daily/2bYrACcmgvvzC075/deployment/pipecat-cloud/images/scaling-cold-starts.png?w=560&fit=max&auto=format&n=2bYrACcmgvvzC075&q=85&s=536a8bce2db0e612a01ae92164414114 560w, https://mintcdn.com/daily/2bYrACcmgvvzC075/deployment/pipecat-cloud/images/scaling-cold-starts.png?w=840&fit=max&auto=format&n=2bYrACcmgvvzC075&q=85&s=4cc93e505426b1165a57994444429e6e 840w, https://mintcdn.com/daily/2bYrACcmgvvzC075/deployment/pipecat-cloud/images/scaling-cold-starts.png?w=1100&fit=max&auto=format&n=2bYrACcmgvvzC075&q=85&s=2014c404f946d52a96b170c9a0a22834 1100w, https://mintcdn.com/daily/2bYrACcmgvvzC075/deployment/pipecat-cloud/images/scaling-cold-starts.png?w=1650&fit=max&auto=format&n=2bYrACcmgvvzC075&q=85&s=20839a2c89da8631a15ba0591ef2beec 1650w, https://mintcdn.com/daily/2bYrACcmgvvzC075/deployment/pipecat-cloud/images/scaling-cold-starts.png?w=2500&fit=max&auto=format&n=2bYrACcmgvvzC075&q=85&s=16040fc16eababec62cdf70e4eb3a037 2500w" />
</Frame>

Pipecat Cloud aims to mitigate cold starts as much as possible through [auto-scaling](#auto-scaling).

<AccordionGroup>
  <Accordion title="How long does a cold start take?">
    The exact duration of a cold start depends on multiple factors, such as the size of your agent image and the complexity of your agent. In general, cold starts take around 10 seconds. Once you have a running instance of a deployed image, new sessions start immediately with no cold start delay.

    For developers, especially during test and development, using scale-to-zero with cold starts is a good place to start as it saves money. Once you understand your utilization pattern, you can build a strategy around optimizing start times by maintaining warm agent instances.
  </Accordion>

  <Accordion title="Mitigation strategies">
    To avoid cold starts, you can:

    * Adjust the number of warm agent instances (`min-agents`) in your pool to ensure that there are always agent instances available to handle requests.
    * Adjust your maximum instance count (`max-agents`) and issue capacity notifications in your application.
  </Accordion>
</AccordionGroup>

### Scale-to-zero

For some deployments, using a minimum instance count of 0 is preferable (e.g. while in development.) Since you are only charged for warm agents instances and active sessions, this can be a cost-effective way to manage deployments where fast start times are not required.

When the minimum instance count is set to 0, the pool will scale down to 0 agent instances when there are no active sessions. Idle agent instances are maintained for 5 minutes before being terminated.

<Tip>
  Scale-to-zero is not recommended for production deployments where immediate
  response is required.
</Tip>

## Auto-scaling

Pipecat Cloud performs auto-scaling by default on all deployments. Auto-scaling is accomplished through the following mechanisms:

* Scaling up based on request velocity
* Maintaining efficiency within max-agents limit
* Scaling down to min-agents (or zero) during low usage
* Supporting burst workloads automatically

### Auto-Scaling Buffer

<Note>
  <b>Pipecat Cloud maintains a free auto-scaling buffer</b> in addition to your
  paid reserved agent instances. This saves you from over-provisioning warm
  agent instances while still ensuring fast response times during traffic
  increases.
</Note>

When your traffic increases and you have active sessions running, our system automatically:

* Proactively provisions additional idle agent instances based on your current usage patterns
* Provides these buffer agent instances at no additional cost to you
* Ensures you can continue handling traffic spikes even when all your paid warm agents are in use

For example, with multiple active sessions:

* The system is already spinning up additional buffer agent instances in the background
* These buffer agent instances become available within \~10 seconds
* You can continue calling the `/start` endpoint without worrying about configuring additional capacity

This approach means:

* You can set a lower `min-agents` value than your peak traffic requirements
* You'll still avoid cold starts in most scenarios
* You get better cost efficiency without sacrificing performance

The only scenario where you need to consider higher `min-agents` values is for extremely rapid traffic spikes (tens or hundreds of calls per second) where the buffer can't be provisioned fast enough.

### Scaling Philosophy

Our scaling system is designed to minimize the need for manual capacity planning. Here's how we recommend thinking about scaling:

* **Start simple:** We encourage you to set `min-agents` to 0 initially and test how the system performs for your specific use case. Many applications work well without any pre-warmed agent instances.

* **Optimize as needed:** We work hard to make cold starts rare and as fast as possible so that, for many applications, you don't have to worry about warm instances at all.

* **Tune for traffic patterns:** If you have spiky workloads with bursty traffic patterns, setting an appropriate `min-agents` value can help prevent cold starts during critical periods. Consider scheduling higher `min-agents` values only during your peak usage hours.

Our goal is to make scaling decisions as simple as possible while giving you the controls you need for optimization when required.

## Updating scaling configuration

You can update your deployment's configuration at any time via the CLI or Pipecat Cloud Dashboard.

```shell  theme={null}
pipecat cloud deploy [agent-name] [image] --min-agents 1 --max-agents 5
```

Please note that changing your scaling parameters will not disrupt any active sessions.
If you reduce your max instance count below the number of currently active sessions, you will still be billed for the duration of those sessions.

## Capacity Planning

Effective capacity planning is crucial for production deployments to ensure your agents respond immediately.

Pipecat Cloud auto-scales your agents. For most cases, the only action you need to take is to set the `--min-agents` parameter to 1. However, if your application experiences fluctuations in traffic, you may need to plan for additional warm capacity to ensure your agents are always ready to respond immediately.

<Card title="Capacity Planning Guide" icon="calculator" href="../guides/capacity-planning">
  See our guide for calculating reserved agents, understanding warm capacity,
  and implementing scaling strategies for production.
</Card>

## Usage summary

Pipecat Cloud bills based on:

* **Active session minutes**: Time your agents spend handling live sessions
* **Reserved session minutes**: Time your warm agent instances are kept running, even when idle

An active session starts when you call the `/start` endpoint (or CLI or SDK equivalent) and ends when your agent's pipeline shuts down.

Reserved session minutes are optional and controlled by setting `--min-agents` in your deployment configuration.

<Note>
  Both active and reserved session time is measured to the second and billed in
  minutes.
</Note>

---

# Source: https://docs.pipecat.ai/deployment/pipecat-cloud/rest-reference/endpoint/secret-create-update.md

> ## Documentation Index
> Fetch the complete documentation index at: https://docs.pipecat.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Create or Update Secrets

> Create or update a secret set and its values



## OpenAPI

````yaml PUT /secrets/{setName}
openapi: 3.0.0
info:
  title: Pipecat Cloud - Create/Update Secret Set
  version: 1.0.0
  description: Create or update a secret set via Pipecat Cloud Private API
servers:
  - url: https://api.pipecat.daily.co/v1
    description: API server
security:
  - PrivateKeyAuth: []
paths:
  /secrets/{setName}:
    put:
      summary: Create or update a secret set
      description: >-
        Create a new secret set or update an existing one. Supports both regular
        secrets (multiple key-value pairs) and image pull secrets.
      operationId: upsertSecretSet
      parameters:
        - name: setName
          in: path
          required: true
          description: Name of the secret set to create or update
          schema:
            type: string
            pattern: ^[a-z0-9]([-a-z0-9]*[a-z0-9])?(\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*$
            minLength: 3
            maxLength: 63
      requestBody:
        required: true
        content:
          application/json:
            schema:
              oneOf:
                - $ref: '#/components/schemas/RegularSecretsRequest'
                - $ref: '#/components/schemas/ImagePullSecretRequest'
            examples:
              regularSecrets:
                summary: Regular secrets
                description: Create or update a set with multiple key-value pairs
                value:
                  secrets:
                    - secretKey: API_KEY
                      secretValue: your-api-key
                    - secretKey: DATABASE_URL
                      secretValue: postgresql://user:pass@host:5432/db
                  region: us-west
              imagePullSecret:
                summary: Image pull secret
                description: Create an image pull secret
                value:
                  isImagePullSecret: true
                  host: https://index.docker.io/v1/
                  secretValue: your-docker-auth-token
                  region: us-west
      responses:
        '200':
          description: Secret set successfully created or updated
          content:
            application/json:
              schema:
                type: object
                properties:
                  status:
                    type: string
                    enum:
                      - OK
                  region:
                    type: string
                    description: The region where the secret set was created or exists
                required:
                  - status
                  - region
              example:
                status: OK
                region: us-west
        '400':
          description: Bad request - Invalid parameters or operation not allowed
          content:
            application/json:
              schema:
                $ref: '#/components/schemas/ErrorResponse'
              examples:
                cannotUpdateImagePullSecret:
                  summary: Cannot update existing image pull secrets
                  value:
                    error: >-
                      Cannot update existing image pull secrets. Delete and
                      recreate instead.
                    code: GENERIC_BAD_REQUEST
                cannotChangeSecretType:
                  summary: Cannot change secret type
                  value:
                    error: >-
                      Cannot update existing regular secrets to image pull
                      secrets
                    code: GENERIC_BAD_REQUEST
                regionMismatch:
                  summary: Region mismatch
                  value:
                    error: >-
                      Secret already exists in region 'us-west'. Cannot change
                      region to 'us-east'.
                    code: GENERIC_BAD_REQUEST
                invalidRegion:
                  summary: Invalid region access
                  value:
                    error: Invalid region or organization does not have access
                    code: GENERIC_BAD_REQUEST
                validationError:
                  summary: Validation error
                  value:
                    error: Invalid request format
                    code: GENERIC_BAD_REQUEST
        '401':
          description: Unauthorized - Invalid or missing API key
          content:
            application/json:
              schema:
                $ref: '#/components/schemas/ErrorResponse'
              example:
                error: Unauthorized
                code: UNAUTHORIZED
        '500':
          description: Internal server error
          content:
            application/json:
              schema:
                $ref: '#/components/schemas/ErrorResponse'
      security:
        - PrivateKeyAuth: []
components:
  schemas:
    RegularSecretsRequest:
      type: object
      properties:
        secrets:
          type: array
          items:
            type: object
            properties:
              secretKey:
                type: string
                description: Name of the secret key
              secretValue:
                type: string
                description: Value of the secret
            required:
              - secretKey
              - secretValue
          minItems: 1
          description: Array of secret key-value pairs
        region:
          type: string
          description: >-
            The region where the secret set will be created. If not specified,
            defaults to the organization's default region. Secrets must be in
            the same region as the agents that use them. Cannot be changed after
            creation.
          example: us-west
      required:
        - secrets
    ImagePullSecretRequest:
      type: object
      properties:
        isImagePullSecret:
          type: boolean
          enum:
            - true
          description: Must be true for image pull secrets
        host:
          type: string
          format: uri
          description: >-
            Host URL. A trailing slash will be added automatically if not
            present.
          example: https://index.docker.io/v1/
        secretValue:
          type: string
          description: Authentication token or credentials
        region:
          type: string
          description: >-
            The region where the secret set will be created. If not specified,
            defaults to the organization's default region. Secrets must be in
            the same region as the agents that use them.
          example: us-west
      required:
        - isImagePullSecret
        - host
        - secretValue
    ErrorResponse:
      type: object
      properties:
        error:
          type: string
          description: Error message
        code:
          type: string
          description: Error code
  securitySchemes:
    PrivateKeyAuth:
      type: http
      scheme: bearer
      description: >-
        Authentication requires a Pipecat Cloud Private API token.


        Generate a Private API key from your Dashboard (Settings > API Keys >
        Private > Create key) and include it as a Bearer token in the
        Authorization header.

````

---

# Source: https://docs.pipecat.ai/deployment/pipecat-cloud/rest-reference/endpoint/secret-delete-secret-set.md

> ## Documentation Index
> Fetch the complete documentation index at: https://docs.pipecat.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Delete Entire Secret Set

> Delete an entire secret set by its name. This operation removes the secret set and all associated key-value pairs.



## OpenAPI

````yaml DELETE /secrets/{setName}
openapi: 3.0.0
info:
  title: Pipecat Cloud - Delete Secret Set
  version: 1.0.0
  description: Delete an entire secret set via Pipecat Cloud Private API
servers:
  - url: https://api.pipecat.daily.co/v1
    description: API server
security:
  - PrivateKeyAuth: []
paths:
  /secrets/{setName}:
    delete:
      summary: Delete an entire secret set
      description: Permanently removes a secret set and all its associated secrets.
      operationId: deleteSecretSet
      parameters:
        - name: setName
          in: path
          required: true
          description: Name of the secret set to delete
          schema:
            type: string
            pattern: ^[a-z0-9]([-a-z0-9]*[a-z0-9])?(\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*$
            minLength: 3
            maxLength: 63
      responses:
        '200':
          description: Secret set successfully deleted
          content:
            application/json:
              schema:
                type: object
                properties:
                  status:
                    type: string
                    enum:
                      - OK
              example:
                status: OK
        '401':
          description: Unauthorized - Invalid or missing API key
          content:
            application/json:
              schema:
                $ref: '#/components/schemas/ErrorResponse'
              example:
                error: Unauthorized
                code: UNAUTHORIZED
        '404':
          description: Secret set not found
          content:
            application/json:
              schema:
                $ref: '#/components/schemas/ErrorResponse'
              example:
                error: Set not found
                code: NOT_FOUND
        '500':
          description: Internal server error
          content:
            application/json:
              schema:
                $ref: '#/components/schemas/ErrorResponse'
      security:
        - PrivateKeyAuth: []
components:
  schemas:
    ErrorResponse:
      type: object
      properties:
        error:
          type: string
          description: Error message
        code:
          type: string
          description: Error code
  securitySchemes:
    PrivateKeyAuth:
      type: http
      scheme: bearer
      description: >-
        Authentication requires a Pipecat Cloud Private API token.


        Generate a Private API key from your Dashboard (Settings > API Keys >
        Private > Create key) and include it as a Bearer token in the
        Authorization header.

````

---

# Source: https://docs.pipecat.ai/deployment/pipecat-cloud/rest-reference/endpoint/secret-delete-specific-secret.md

> ## Documentation Index
> Fetch the complete documentation index at: https://docs.pipecat.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Delete Specific Secret From Set

> Delete a specific secret from a set by its key. This operation removes the key-value pair from the specified secret set.



## OpenAPI

````yaml DELETE /secrets/{setName}/{secretKey}
openapi: 3.0.0
info:
  title: Pipecat Cloud - Delete Secret Key
  version: 1.0.0
  description: Delete a specific secret key from a secret set via Pipecat Cloud Private API
servers:
  - url: https://api.pipecat.daily.co/v1
    description: API server
security:
  - PrivateKeyAuth: []
paths:
  /secrets/{setName}/{secretKey}:
    delete:
      summary: Delete a specific secret from a set
      description: >-
        Remove a specific secret key from a secret set. Cannot be used with
        image pull secrets.
      operationId: deleteSecretFromSet
      parameters:
        - name: setName
          in: path
          required: true
          description: Name of the secret set
          schema:
            type: string
            pattern: ^[a-z0-9]([-a-z0-9]*[a-z0-9])?(\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*$
            minLength: 3
            maxLength: 63
        - name: secretKey
          in: path
          required: true
          description: Name of the secret key to delete
          schema:
            type: string
      responses:
        '200':
          description: Secret successfully deleted
          content:
            application/json:
              schema:
                type: object
                properties:
                  status:
                    type: string
                    enum:
                      - OK
              example:
                status: OK
        '401':
          description: Unauthorized - Invalid or missing API key
          content:
            application/json:
              schema:
                $ref: '#/components/schemas/ErrorResponse'
              example:
                error: Unauthorized
                code: UNAUTHORIZED
        '404':
          description: Secret not found
          content:
            application/json:
              schema:
                $ref: '#/components/schemas/ErrorResponse'
              example:
                error: Secret not found
                code: NOT_FOUND
        '500':
          description: Internal server error
          content:
            application/json:
              schema:
                $ref: '#/components/schemas/ErrorResponse'
      security:
        - PrivateKeyAuth: []
components:
  schemas:
    ErrorResponse:
      type: object
      properties:
        error:
          type: string
          description: Error message
        code:
          type: string
          description: Error code
  securitySchemes:
    PrivateKeyAuth:
      type: http
      scheme: bearer
      description: >-
        Authentication requires a Pipecat Cloud Private API token.


        Generate a Private API key from your Dashboard (Settings > API Keys >
        Private > Create key) and include it as a Bearer token in the
        Authorization header.

````

---

# Source: https://docs.pipecat.ai/deployment/pipecat-cloud/rest-reference/endpoint/secret-list-all.md

> ## Documentation Index
> Fetch the complete documentation index at: https://docs.pipecat.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# List All Secret Sets

> Retrieve a list of all secret sets in your organization with their name and secret type.



## OpenAPI

````yaml GET /secrets
openapi: 3.0.0
info:
  title: Pipecat Cloud - List Secret Sets
  version: 1.0.0
  description: List all secret sets via Pipecat Cloud Private API
servers:
  - url: https://api.pipecat.daily.co/v1
    description: API server
security:
  - PrivateKeyAuth: []
paths:
  /secrets:
    get:
      summary: List all secret sets
      description: Retrieve a list of all secret sets for the organization.
      operationId: listSecretSets
      responses:
        '200':
          description: List of secret sets
          content:
            application/json:
              schema:
                type: object
                properties:
                  sets:
                    type: array
                    items:
                      $ref: '#/components/schemas/SecretSetSummary'
              example:
                sets:
                  - name: my-app-secrets
                    type: secret
                    region: us-west
                  - name: dockerhub-credentials
                    type: imagePullSecret
                    region: us-west
        '401':
          description: Unauthorized - Invalid or missing API key
          content:
            application/json:
              schema:
                $ref: '#/components/schemas/ErrorResponse'
              example:
                error: Unauthorized
                code: UNAUTHORIZED
        '500':
          description: Internal server error
          content:
            application/json:
              schema:
                $ref: '#/components/schemas/ErrorResponse'
      security:
        - PrivateKeyAuth: []
components:
  schemas:
    SecretSetSummary:
      type: object
      properties:
        name:
          type: string
          description: Name of the secret set
        type:
          type: string
          enum:
            - secret
            - imagePullSecret
          description: Type of the secret set
        region:
          type: string
          description: The region where the secret set is stored
    ErrorResponse:
      type: object
      properties:
        error:
          type: string
          description: Error message
        code:
          type: string
          description: Error code
  securitySchemes:
    PrivateKeyAuth:
      type: http
      scheme: bearer
      description: >-
        Authentication requires a Pipecat Cloud Private API token.


        Generate a Private API key from your Dashboard (Settings > API Keys >
        Private > Create key) and include it as a Bearer token in the
        Authorization header.

````

---

# Source: https://docs.pipecat.ai/deployment/pipecat-cloud/rest-reference/endpoint/secret-list-one.md

> ## Documentation Index
> Fetch the complete documentation index at: https://docs.pipecat.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Get Secret Set Details

> Retrieve key and value pairs for a specific secret set.



## OpenAPI

````yaml GET /secrets/{setName}
openapi: 3.0.0
info:
  title: Pipecat Cloud - Get Secret Set
  version: 1.0.0
  description: Get a specific secret set via Pipecat Cloud Private API
servers:
  - url: https://api.pipecat.daily.co/v1
    description: API server
security:
  - PrivateKeyAuth: []
paths:
  /secrets/{setName}:
    get:
      summary: Get a specific secret set
      description: >-
        Retrieve the keys (not values) of a specific secret set. Image pull
        secrets cannot be retrieved.
      operationId: getSecretSet
      parameters:
        - name: setName
          in: path
          required: true
          description: Name of the secret set to retrieve
          schema:
            type: string
            pattern: ^[a-z0-9]([-a-z0-9]*[a-z0-9])?(\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*$
            minLength: 3
            maxLength: 63
      responses:
        '200':
          description: Secret set details
          content:
            application/json:
              schema:
                type: object
                properties:
                  region:
                    type: string
                    description: The region where the secret set is stored
                  secrets:
                    type: array
                    items:
                      $ref: '#/components/schemas/SecretKey'
              example:
                region: us-west
                secrets:
                  - fieldName: API_KEY
                  - fieldName: DATABASE_URL
                  - fieldName: JWT_SECRET
        '400':
          description: Bad request - Cannot retrieve image pull secrets
          content:
            application/json:
              schema:
                $ref: '#/components/schemas/ErrorResponse'
              example:
                error: Cannot retrieve image pull secrets by name
                code: GENERIC_BAD_REQUEST
        '401':
          description: Unauthorized - Invalid or missing API key
          content:
            application/json:
              schema:
                $ref: '#/components/schemas/ErrorResponse'
              example:
                error: Unauthorized
                code: UNAUTHORIZED
        '404':
          description: Secret set not found
          content:
            application/json:
              schema:
                $ref: '#/components/schemas/ErrorResponse'
              example:
                error: Set not found
                code: NOT_FOUND
        '500':
          description: Internal server error
          content:
            application/json:
              schema:
                $ref: '#/components/schemas/ErrorResponse'
      security:
        - PrivateKeyAuth: []
components:
  schemas:
    SecretKey:
      type: object
      properties:
        fieldName:
          type: string
          description: Name of the secret key (values are not returned for security)
    ErrorResponse:
      type: object
      properties:
        error:
          type: string
          description: Error message
        code:
          type: string
          description: Error code
  securitySchemes:
    PrivateKeyAuth:
      type: http
      scheme: bearer
      description: >-
        Authentication requires a Pipecat Cloud Private API token.


        Generate a Private API key from your Dashboard (Settings > API Keys >
        Private > Create key) and include it as a Bearer token in the
        Authorization header.

````

---

# Source: https://docs.pipecat.ai/deployment/pipecat-cloud/fundamentals/secrets.md

# Source: https://docs.pipecat.ai/cli/cloud/secrets.md

> ## Documentation Index
> Fetch the complete documentation index at: https://docs.pipecat.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# secrets

> Secret sets and secret management commands

The `secrets` command group helps you manage sensitive information for your agent deployments. You can create and manage secret sets (key-value pairs) and image pull secrets (for private Docker registries).

## list

List secret sets and image pull secrets for active namespace / organization.

If provided with a valid secret set name, will show the keys of that set (values are hidden.)

**Usage:**

```shell  theme={null}
pipecat cloud secrets list [ARGS] [OPTIONS]
```

**Arguments:**

<ParamField path="name" type="string">
  Name of the secret set list keys for. Must be a valid string identifier.
</ParamField>

**Options:**

<ParamField path="--sets / -s" type="boolean" default="true">
  Show secret sets only. Filter out image pull secrets from the results.
</ParamField>

<ParamField path="--organization / -o" type="string">
  Organization to list secrets for. If not provided, uses the current
  organization from your configuration.
</ParamField>

<ParamField path="--region / -r" type="string">
  Filter secrets by region. Only secrets in the specified region will be shown.
  If not provided, secrets from all regions are listed.
</ParamField>

## set

Create or update a secret set with the given name and key-value pairs. Secrets can be passed directly as key value pairs or loaded from a file.

**Usage:**

```shell  theme={null}
pipecat cloud secrets set [ARGS] [OPTIONS]
```

**Arguments:**

<ParamField path="name" type="string" required>
  Name of the secret set to create or modify. Must be a valid string identifier
  containing only characters, numbers, and hyphens.
</ParamField>

<ParamField path="secrets" type="string">
  List of secret key-value pairs e.g. `KEY1=value1 KEY2="value with spaces"`.

  See [this note](/deployment/pipecat-cloud/fundamentals/secrets#special-characters) on using special
  characters in secret values.

  Example:

  ```shell  theme={null}
  pipecat cloud secrets set my-secrets 'API_KEY=123 API_KEY_2="value with spaces"'
  ```
</ParamField>

**Options:**

<ParamField path="--file / -f" type="string">
  Relative path to a file with a list of secret key-value pairs. Each line in
  the file should be in the format `KEY=value`.

  Example:

  ```shell  theme={null}
  pipecat cloud secrets set my-secrets --file .env
  ```
</ParamField>

<ParamField path="--skip / -s" type="boolean" default="false">
  Skip confirmations and proceed with the operation.
</ParamField>

<ParamField path="--organization / -o" type="string">
  Organization to create/update the secret set in. If not provided, uses the
  current organization from your configuration.
</ParamField>

<ParamField path="--region / -r" type="string">
  Region where the secret set will be stored. If not specified, uses your
  organization's default region (typically `us-west`). Secrets must be in the
  same region as the agents that use them.
</ParamField>

**Example:**

Create a secret set in a specific region:

```shell  theme={null}
pipecat cloud secrets set my-secrets API_KEY=abc123 --region eu-central
```

## unset

Removes a specific secret key from a secret set.

**Usage:**

```shell  theme={null}
pipecat cloud secrets unset [ARGS] [OPTIONS]
```

**Arguments:**

<ParamField path="name" type="string" required>
  Name of the secret set to remove the secret from.
</ParamField>

<ParamField path="secret_key" type="string" required>
  The key of the secret to remove from the set.
</ParamField>

Example:

```shell  theme={null}
pipecat cloud secrets unset my-secret-set SOME_KEY
```

**Options:**

<ParamField path="--skip / -s" type="boolean" default="false">
  Skip confirmations and proceed with the operation.
</ParamField>

<ParamField path="--organization / -o" type="string">
  Organization containing the secret set. If not provided, uses the current
  organization from your configuration.
</ParamField>

## delete

Deletes an entire secret set.

**Usage:**

```shell  theme={null}
pipecat cloud secrets delete [ARGS] [OPTIONS]
```

**Arguments:**

<ParamField path="name" type="string" required>
  Name of the secret set to delete. This action is irreversible.
</ParamField>

**Options:**

<ParamField path="--skip / -s" type="boolean" default="false">
  Skip confirmations and proceed with the operation.
</ParamField>

<ParamField path="--organization / -o" type="string">
  Organization containing the secret set. If not provided, uses the current
  organization from your configuration.
</ParamField>

## image-pull-secret

Creates credentials for pulling images from private Docker registries.

This command encodes and securely stores your image repository credentials. These credentials are used with the [deploy](./deploy) command when pulling images from private repositories. If you don't provide credentials directly, the command will prompt you for input and offer the option to encode them in base64 format for additional security.

**Usage:**

```shell  theme={null}
pipecat cloud secrets image-pull-secret [ARGS] [OPTIONS]
```

**Arguments:**

<ParamField path="name" type="string" required>
  Name of the credentials set to create or modify. Must be a valid string
  identifier.
</ParamField>

<ParamField path="host" type="string" required>
  Host address of the image repository e.g. `https://index.docker.io/v1/`.
</ParamField>

<ParamField path="credentials" type="string" default="prompts for input">
  Credentials for the image repository in the form of `username:password`.

  Will prompt you for the value if not provided.

  Example:

  ```shell  theme={null}
  pipecat cloud secrets image-pull-secret my-registry-creds https://index.docker.io/v1/ my-username:my-password
  ```
</ParamField>

**Options:**

<ParamField path="--encode" type="boolean" default="true">
  Encode the credentials in base64 format.
</ParamField>

<ParamField path="--organization / -o" type="string">
  Organization to create the image pull secret in. If not provided, uses the
  current organization from your configuration.
</ParamField>

<ParamField path="--region / -r" type="string">
  Region where the image pull secret will be stored. If not specified, uses your
  organization's default region (typically `us-west`). Image pull secrets must
  be in the same region as the agents that use them.
</ParamField>

**Examples:**

Create an image pull secret in a specific region:

```shell  theme={null}
pipecat cloud secrets image-pull-secret my-registry-creds https://index.docker.io/v1/ --region eu-central
```

***

<Card title="More about secrets" icon="key" href="/deployment/pipecat-cloud/fundamentals/secrets">
  Learn more about managing application secrets
</Card>

---

# Source: https://docs.pipecat.ai/deployment/pipecat-cloud/security/security-and-compliance.md

> ## Documentation Index
> Fetch the complete documentation index at: https://docs.pipecat.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Security Guide

> Securing your agents on Pipecat Cloud

This guide addresses security considerations throughout your journey with Pipecat Cloud, from account creation to running production agents. Each section focuses on practical security measures you can take and explains how Pipecat Cloud protects your agents.

## Account & Organization Security

### How do I secure access to my Pipecat Cloud account?

Pipecat Cloud implements different authentication mechanisms depending on how you interact with the platform:

| Interface | Authentication Method          | Security Notes                              |
| --------- | ------------------------------ | ------------------------------------------- |
| Dashboard | Email + OTP                    | Two-factor authentication by default        |
| CLI       | JWT via Clerk                  | Token-based auth with automatic expiration  |
| REST API  | Bearer token (public API keys) | Scoped to specific organizations and agents |

<Tip>
  For CLI usage, we recommend running `pipecat cloud auth login` regularly to
  refresh your credentials.
</Tip>

### How do I manage access for my team?

Organizations provide a secure way to collaborate with team members:

* Create separate organizations for different teams or environments
* Invite members with appropriate permission levels
* Use organization-specific API keys for automation

Each organization has its own isolated environment with separate resources, agents, and secrets.

## Development Security

### How do I keep API keys and secrets secure during development?

Never include sensitive credentials in your agent code or images:

```python  theme={null}
# DON'T do this
api_key = "sk_1234567890abcdef"  # Hardcoded keys are a security risk

# DO this instead
import os
api_key = os.environ.get("SERVICE_API_KEY")  # Use environment variables
```

During local development, use environment files (`.env`) and ensure they're in your `.gitignore`.

For deployment, use Pipecat Cloud's secrets management:

```bash  theme={null}
# Create a secret set with your API keys
pipecat cloud secrets set my-agent-secrets \
  OPENAI_API_KEY=sk_... \
  ANTHROPIC_API_KEY=sk_...

# Reference in deployment
pipecat cloud deploy my-agent my-image:latest --secret-set my-agent-secrets
```

<Info>
  Secrets are encrypted at rest and in transit, then securely injected as
  environment variables in your agent's runtime.
</Info>

<Warning>
  When you update or rotate API keys in a secret set, you must redeploy your
  agent to apply these changes. Updated secrets are not automatically propagated
  to running deployments.
</Warning>

### Is my agent code secure during development?

Your local development environment and version control security are your responsibility. We recommend:

* Using private repositories
* Implementing branch protection
* Scanning code for security issues
* Never committing secrets or credentials

## Deployment Security

### How do I secure my agent container?

Follow these container security best practices:

1. Start with our [official base image](https://github.com/daily-co/pipecat-cloud-images/tree/main/pipecat-base) for a pre-hardened configuration
2. Include only necessary components to reduce attack surface
3. Keep dependencies updated to patch vulnerabilities
4. Use container scanning tools before deployment

### Are my private container images secure?

If you use private container registries, Pipecat Cloud supports secure image pulls:

```bash  theme={null}
# Create an image pull secret for private registry
pipecat cloud secrets image-pull-secret dockerhub https://index.docker.io/v1/

# Alternatively, use base64 encoding for credentials
pipecat cloud secrets image-pull-secret dockerhub https://index.docker.io/v1/ --base64encode

# Use it during deployment
pipecat cloud deploy my-agent my-private-repo/image:latest --credentials dockerhub
```

Credentials are securely stored and only used during image pulls.

<Tip>
  The optional `--base64encode` flag provides an additional layer of obfuscation
  when entering credentials in environments where terminal history or session
  logging might be a concern.
</Tip>

### How does Pipecat Cloud protect my agent code?

Your deployed code remains private and protected through multiple layers:

* Containers run in isolated Kubernetes pods
* Network policies restrict cross-agent communication
* Each organization has separate namespaces
* Access controls prevent unauthorized access to running containers

## Runtime Security

### Can other customers access my agents or data?

Pipecat Cloud implements strong multi-tenant isolation:

* Organizational namespacing creates logical boundaries between customers
* Network policies prevent cross-namespace communication
* Kubernetes RBAC controls limit access to running pods
* Authentication is required for all API access

### How are my agents protected at runtime?

Pipecat Cloud provides several runtime security measures:

* **Network Controls**: Restrict egress to necessary services
* **Access Monitoring**: Track authentication and access patterns
* **Secret Injection**: Environment variables for secure credential access

### Who can start sessions with my agents?

Only clients with valid API keys for your organization can start agent sessions. For security best practices:

1. **Use server-side code** to start agent sessions, keeping API keys secure on your server
2. **Never expose API keys** in client-side code

```bash  theme={null}
# Create a public API key for client applications
pipecat cloud organizations keys create

# Associate with specific agents
pipecat cloud organizations keys use
```

<Warning>
  While called "public API keys," these should still be treated as sensitive
  credentials and not shared with end-users or embedded in client-side code.
  They are "public" only in the sense that they have limited permissions
  compared to admin keys.
</Warning>

We recommend implementing a server-side endpoint that starts sessions on behalf of your users, similar to how you might handle other sensitive API operations.

## Data Security

### Is my data encrypted?

Pipecat Cloud encrypts your data throughout the platform:

* **In Transit**: TLS 1.3 encryption for all API communications
* **At Rest**: AES-256 encryption for stored data
* **Database**: Hardware-level encryption for database content, such as public API keys which developers control the rotation of

### Where is my data stored?

All Pipecat Cloud infrastructure is currently located in the US (AWS us-west-2 region). Contact us at `help@daily.co` if you have specific data residency requirements.

### What happens to my data when agents terminate?

Ephemeral data used during agent sessions is removed when the session ends. By default, stdout and stderr logs are stored in Pipecat Cloud. You have control over what is logged and where the logging is stored. Refer to [Logging & Observability](../fundamentals/logging) for more information.

## Monitoring & Incident Response

### How does Pipecat Cloud detect security issues?

We maintain comprehensive logging and monitoring:

* Administrative API calls are logged and retained for 1 year
* Authentication attempts and access patterns are monitored
* System events are tracked for security analysis

<Info>
  Request access to [Daily's Trust Center](https://trust.daily.co/) to learn
  more about our incident response and security practices.
</Info>

### How do I report security vulnerabilities?

If you discover a potential security vulnerability:

1. Email us at [disclosures@daily.co](mailto:disclosures@daily.co) (we'll acknowledge within one week)
2. Provide reasonable time for us to resolve the issue before disclosure
3. Make a good faith effort to avoid violating privacy or degrading service

<Note>
  Reporting security findings directly to Daily helps to ensure continued
  protection for all users.
</Note>

## Security Responsibilities

Security in cloud environments works best as a partnership. Pipecat Cloud follows a shared responsibility model where we secure the underlying infrastructure and platform, while you maintain responsibility for your application code and configuration. Understanding these boundaries helps ensure comprehensive protection for your agents and data.

### What security aspects does Pipecat Cloud handle?

Pipecat Cloud is responsible for:

* Infrastructure and platform security
* Network security and isolation
* Secret storage and injection
* Authentication systems
* Platform monitoring and logging

### What security aspects am I responsible for?

As a customer, you're responsible for:

* Application code security
* API key management
* Agent configuration security
* Following container security best practices
* Proper secrets management

<Tip>
  While Pipecat Cloud secures log storage, you should ensure your agent code
  doesn't log sensitive information like credentials, personal data, or
  proprietary content in stdout/stderr streams.
</Tip>

## Additional Resources

For security-related questions or to report security issues:

* **Security Issues**: Email [disclosures@daily.co](mailto:disclosures@daily.co) to report potential vulnerabilities
* **General Support**: Contact [help@daily.co](mailto:help@daily.co) for security questions

---

# Source: https://docs.pipecat.ai/server/services/analytics/sentry.md

> ## Documentation Index
> Fetch the complete documentation index at: https://docs.pipecat.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Sentry Metrics

> Performance monitoring integration with Sentry for Pipecat frame processors

## Overview

`SentryMetrics` extends `FrameProcessorMetrics` to provide performance monitoring integration with Sentry. It tracks Time to First Byte (TTFB) and processing duration metrics for frame processors, enabling real-time performance monitoring and error tracking for your Pipecat applications.

<CardGroup cols={2}>
  <Card title="Sentry Metrics API Reference" icon="code" href="https://reference-server.pipecat.ai/en/latest/api/pipecat.processors.metrics.sentry.html">
    Pipecat's API methods for Sentry metrics integration
  </Card>

  <Card title="Example Implementation" icon="play" href="https://github.com/pipecat-ai/pipecat/blob/main/examples/foundational/47-sentry-metrics.py">
    Browse examples using Sentry metrics
  </Card>

  <Card title="Sentry Documentation" icon="book" href="https://docs.sentry.io/platforms/python/">
    Official Sentry Python SDK documentation
  </Card>

  <Card title="Sentry Platform" icon="chart-line" href="https://sentry.io/">
    Access performance monitoring and error tracking
  </Card>
</CardGroup>

## Installation

To use Sentry analytics services, install the required dependencies:

```bash  theme={null}
pip install "pipecat-ai[sentry]"
```

## Prerequisites

### Sentry Account Setup

Before using Sentry metrics services, you need:

1. **Sentry Account**: Sign up at [Sentry Platform](https://sentry.io/)
2. **Project Setup**: Create a project and obtain your DSN
3. **SDK Initialization**: Configure Sentry SDK in your application
4. **Metrics Configuration**: Set up performance monitoring and error tracking

### Required Configuration

* **Sentry DSN**: Your project's Data Source Name for authentication
* **Traces Sample Rate**: Configure performance monitoring sampling
* **SDK Initialization**: Initialize Sentry before using metrics

### Key Features

* **Performance Monitoring**: Track TTFB and processing duration metrics
* **Error Tracking**: Automatic error capture and reporting
* **Frame Processor Metrics**: Monitor individual processor performance
* **Real-time Analytics**: Live performance data and alerting

---

# Source: https://docs.pipecat.ai/server/utilities/service-switchers/service-switcher.md

> ## Documentation Index
> Fetch the complete documentation index at: https://docs.pipecat.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# ServiceSwitcher

> Dynamically switch between different service instances at runtime using configurable strategies

## Overview

`ServiceSwitcher` is a specialized parallel pipeline that enables dynamic switching between multiple service instances at runtime. This is useful when you need to change between different STT providers, TTS providers, or other frame processors based on user preferences, costs, performance requirements, or other runtime conditions.

The switcher uses a strategy pattern to determine which service is active. Currently, a manual switching strategy is provided, allowing explicit control over which service handles frames at any given time.

## How It Works

`ServiceSwitcher` wraps multiple services in a parallel pipeline where only the active service processes frames. Each service is "sandwiched" between two filters that check if it's the currently active service before allowing frames to pass through. When you switch services, the filters update to redirect frame flow to the newly active service.

## Constructor

```python  theme={null}
from pipecat.pipeline.service_switcher import ServiceSwitcher, ServiceSwitcherStrategyManual

switcher = ServiceSwitcher(
    services=[stt_service1, stt_service2],
    strategy_type=ServiceSwitcherStrategyManual
)
```

<ParamField path="services" type="List[FrameProcessor]" required>
  List of service instances to switch between. Can be any frame processors (STT,
  TTS, or custom processors).
</ParamField>

<ParamField path="strategy_type" type="Type[ServiceSwitcherStrategy]" required>
  The strategy class to use for switching logic. Pass the class itself, not an
  instance.
</ParamField>

## Switching Strategies

### ServiceSwitcherStrategyManual

The manual strategy allows explicit control over which service is active by pushing `ManuallySwitchServiceFrame` frames into the pipeline.

**Initial State**: The first service in the list is active by default.

**Switching**: Push a `ManuallySwitchServiceFrame` with the desired service instance.

## Usage Examples

### Switching Between TTS Services

```python  theme={null}
from pipecat.frames.frames import ManuallySwitchServiceFrame
from pipecat.pipeline.service_switcher import ServiceSwitcher, ServiceSwitcherStrategyManual
from pipecat.services.elevenlabs.tts import ElevenLabsTTSService
from pipecat.services.cartesia.tts import CartesiaTTSService

# Create TTS services
elevenlabs = ElevenLabsTTSService(api_key=os.getenv("ELEVENLABS_API_KEY"), voice_id=os.getenv("ELEVENLABS_VOICE_ID"))
cartesia = CartesiaTTSService(api_key=os.getenv("CARTESIA_API_KEY"), voice_id=os.getenv("CARTESIA_VOICE_ID"))

# Create switcher with both services
tts_switcher = ServiceSwitcher(
    services=[elevenlabs, cartesia],
    strategy_type=ServiceSwitcherStrategyManual
)

# Use in pipeline
pipeline = Pipeline([
    transport.input(),
    stt,
    context_aggregator.user(),
    llm,
    tts_switcher,
    transport.output(),
    context_aggregator.assistant()
])

# Later, switch to Cartesia
await task.queue_frame(ManuallySwitchServiceFrame(service=cartesia))
```

---

# Source: https://docs.pipecat.ai/deployment/pipecat-cloud/guides/session-api.md

> ## Documentation Index
> Fetch the complete documentation index at: https://docs.pipecat.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Session API

> Send HTTP requests to your running Pipecat Cloud sessions

The Session API lets you send HTTP requests directly to your running Pipecat Cloud agents. This enables real-time control and data exchange with active sessions, such as updating conversation context, triggering actions, or retrieving session state.

## How It Works

1. [Start a session](/deployment/pipecat-cloud/fundamentals/active-sessions) with your agent
2. Capture the `sessionId` header from the start response
3. Send HTTP requests to your session using the session endpoint

## Endpoint Format

```
https://api.pipecat.daily.co/v1/public/{service_name}/sessions/{session_id}/{path}
```

| Parameter      | Description                                   |
| -------------- | --------------------------------------------- |
| `service_name` | The name of your deployed agent               |
| `session_id`   | The `sessionId` value from the start response |
| `path`         | The endpoint path you defined in your bot     |

### Supported Methods

* `GET`
* `POST`
* `PUT`
* `PATCH`
* `DELETE`
* `OPTIONS`
* `HEAD`

## Authentication

Include your Pipecat Cloud public API key in the `Authorization` header:

```bash  theme={null}
Authorization: Bearer pk_...
```

## Setting Up Your Bot

To handle Session API requests, define endpoints in your `bot.py` file using the `app` object from `pipecatcloud_system`:

```python  theme={null}
from pipecatcloud_system import app
from pipecat.runner.types import PipecatRunnerArguments

# Define your API endpoints
@app.get("/status")
async def get_status():
    return {"status": "active", "message": "Bot is running"}

@app.post("/update-context")
async def update_context(data: dict):
    # Handle context updates
    return {"updated": True}

# Your main bot function
async def bot(args: PipecatRunnerArguments):
    # Bot implementation
    pass
```

<Note>Requires base image version `0.1.2` or later.</Note>

## Examples

### Get Session Status

Define an endpoint that returns the current session state:

```python  theme={null}
from pipecatcloud_system import app

session_data = {"messages": [], "user_name": None}

@app.get("/status")
async def get_status():
    return {
        "message_count": len(session_data["messages"]),
        "user_name": session_data["user_name"]
    }
```

Call the endpoint:

```bash  theme={null}
curl -X GET \
  'https://api.pipecat.daily.co/v1/public/my-agent/sessions/57af5437-97a2-4646-9873-a5c5935bd705/status' \
  -H 'Authorization: Bearer pk_...'
```

### Update Conversation Context

Define an endpoint to inject information into the conversation:

```python  theme={null}
from pipecatcloud_system import app
from pydantic import BaseModel

class ContextUpdate(BaseModel):
    user_name: str
    preferences: dict

@app.post("/context")
async def update_context(update: ContextUpdate):
    # Update your bot's context with the new information
    # This could modify the LLM system prompt, add to conversation history, etc.
    return {"status": "context updated", "user_name": update.user_name}
```

Call the endpoint:

```bash  theme={null}
curl -X POST \
  'https://api.pipecat.daily.co/v1/public/my-agent/sessions/57af5437-97a2-4646-9873-a5c5935bd705/context' \
  -H 'Authorization: Bearer pk_...' \
  -H 'Content-Type: application/json' \
  -d '{"user_name": "Alice", "preferences": {"language": "Spanish"}}'
```

### Trigger an Action

Define an endpoint to trigger bot behavior mid-session:

```python  theme={null}
from pipecatcloud_system import app

@app.post("/speak")
async def trigger_speech(data: dict):
    message = data.get("message", "Hello!")
    # Queue the message for your TTS pipeline
    # Implementation depends on your bot architecture
    return {"queued": True, "message": message}
```

## Important Notes

* **Startup latency**: If you call the Session API before your bot finishes initializing, the request may take longer while waiting for the bot to become available.
* **Session scope**: Each endpoint call is routed to the specific session identified by `session_id`. Different sessions run independently.
* **Error handling**: If a session has ended or the session ID is invalid, you'll receive an error response.

## Getting Help

If you have questions about the Session API, reach out on [Discord](https://discord.gg/pipecat).

---

# Source: https://docs.pipecat.ai/deployment/pipecat-cloud/sdk-reference/session-arguments.md

> ## Documentation Index
> Fetch the complete documentation index at: https://docs.pipecat.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Session Arguments

> Understanding the arguments received by bot entry points

When creating agents with Pipecat Cloud, your `bot()` entry point function receives different types of arguments depending on the session type. These classes represent the structure of those arguments.

## PipecatSessionArguments

Standard Pipecat Cloud agent session arguments, used for basic sessions.

```python  theme={null}
from pipecatcloud.agent import PipecatSessionArguments

def bot(args: PipecatSessionArguments):
    print(f"Session ID: {args.session_id}")
    print(f"Custom data: {args.body}")
```

### Properties

<ParamField path="session_id" type="Optional[str]">
  The unique identifier for the current session.
</ParamField>

<ParamField path="body" type="Any">
  The custom data passed to the agent via the session parameters.
</ParamField>

## DailySessionArguments

Arguments for sessions that involve Daily WebRTC rooms for voice/video interaction.

```python  theme={null}
from pipecat.runner.types import DailyRunnerArguments

def bot(args: DailyRunnerArguments):
    print(f"Session ID: {args.session_id}")
    print(f"Daily room URL: {args.room_url}")
    print(f"Daily token: {args.token}")
    print(f"Custom data: {args.body}")
```

### Properties

<ParamField path="session_id" type="Optional[str]">
  The unique identifier for the current session.
</ParamField>

<ParamField path="room_url" type="str">
  The URL for the Daily room.
</ParamField>

<ParamField path="token" type="str">
  The authentication token for the Daily room.
</ParamField>

<ParamField path="body" type="Any">
  The custom data passed to the agent via the session parameters.
</ParamField>

## WebSocketSessionArguments

Arguments for sessions that use WebSocket connections for real-time communication.

```python  theme={null}
from pipecat.runner.types import WebSocketRunnerArguments

async def bot(args: WebSocketRunnerArguments):
    print(f"Session ID: {args.session_id}")
    await args.websocket.send_text("Hello from the agent!")
```

### Properties

<ParamField path="session_id" type="Optional[str]">
  The unique identifier for the current session.
</ParamField>

<ParamField path="websocket" type="WebSocket">
  The FastAPI WebSocket connection used to communicate with the client.
</ParamField>

---

# Source: https://docs.pipecat.ai/guides/learn/session-initialization.md

> ## Documentation Index
> Fetch the complete documentation index at: https://docs.pipecat.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Session Initialization

> Learn how to set up connections between users and your Pipecat voice AI bot

Before your voice AI bot can start processing audio and generating responses, you need to establish a connection between the user and your bot. This process is called **session initialization** - it's how users and bots find each other and establish a communication channel for real-time audio exchange.

## Understanding the Architecture

Session initialization involves multiple components working together:

* **Runner**: A FastAPI server that handles incoming connection requests and manages session setup
* **Pipecat Bot**: Your voice AI application running as a separate server-side service
* **Client Application**: The user-facing app (web browser, mobile app, etc.)

The runner acts as the coordinator, setting up the necessary resources and starting bot instances, while the Pipecat bot handles the actual voice AI processing.

## Development Runner

For most development and many production use cases, Pipecat provides a **development runner** that handles all the session initialization complexity for you. Instead of building FastAPI servers and managing WebRTC connections yourself, you focus on your bot logic while the runner handles the infrastructure.

### Using the Development Runner

Your bot needs a single entry point function that the runner will call:

```python  theme={null}
from pipecat.runner.types import RunnerArguments

async def bot(runner_args: RunnerArguments):
    """Main bot entry point called by the development runner."""

    # Create your transport based on the runner arguments
    transport = SmallWebRTCTransport(
        params=TransportParams(
            audio_in_enabled=True,
            audio_out_enabled=True,
        ),
        webrtc_connection=runner_args.webrtc_connection,
    )

    # Run your bot logic
    await run_bot(transport)

if __name__ == "__main__":
    from pipecat.runner.run import main
    main()
```

Then start your bot with different connection types:

```bash  theme={null}
# P2P WebRTC (opens browser interface)
python bot.py -t webrtc

# Daily room-based WebRTC
python bot.py -t daily

# Telephony (requires ngrok or similar)
python bot.py -t twilio -x your_domain.ngrok.io
```

where `-t` specifies the transport type (e.g., `webrtc`, `daily`, `twilio`) and `-x` is the optional proxy domain for telephony.

The development runner automatically:

* Creates the FastAPI server
* Sets up the appropriate endpoints
* Handles connection management
* Starts your bot instances
* Provides a web interface (for WebRTC)

<Tip>
  Learn more about building with the development runner in the [runner
  guide](/server/utilities/runner/guide).
</Tip>

## Connection Types Under the Hood

While the development runner handles the complexity, understanding the three connection patterns helps you choose the right approach and debug issues:

### 1. P2P WebRTC Connections

**What happens:**

1. Runner serves a web interface at `http://localhost:7860/client`
2. When you open the page and connect, browser creates a WebRTC offer
3. Runner receives the offer, establishes connection, starts your bot
4. Browser and bot communicate directly via WebRTC

**When to use:** Direct client connections, embedded applications, local development

### 2. Room-Based WebRTC (Daily)

**What happens:**

<Steps>
  <Step title="Room Request">
    User visits the client application and clicks to start a session
  </Step>

  <Step title="Room Creation">
    Runner calls Daily's API to create a room and tokens using
    `pipecat.runner.daily.configure()`
  </Step>

  <Step title="Parallel Join">
    Both user's browser and your bot join the same Daily room
  </Step>

  <Step title="Media Handshake">
    Once media streams are established, browser sends `client_ready` message
  </Step>

  <Step title="Bot Activation">
    Your bot receives the event and starts the conversation
  </Step>
</Steps>

**When to use:** Video calls, group sessions, production deployments

<Note>
  Room-based WebRTC can also be used for SIP or PSTN connections, which require
  different connection patterns. Refer to the [telephony
  guide](/guides/telephony/overview) for details.
</Note>

### 3. WebSocket Connections (Telephony)

**What happens:**

1. Telephony provider (Twilio, etc.) receives a phone call
2. Provider connects to your runner's webhook endpoint
3. Runner accepts WebSocket connection and parses telephony-specific messages
4. Your bot starts immediately with the parsed connection data

**When to use:** Phone bots, telephony integrations

## Starting Conversations

How and when your bot begins talking depends on the connection type:

### Immediate Start (P2P WebRTC, WebSocket)

These connections are ready immediately, so you can start talking right after connection:

```python  theme={null}
@transport.event_handler("on_client_connected")
async def on_client_connected(transport, client):
    logger.info("Client connected - starting conversation")
    messages.append({
        "role": "system",
        "content": "Say hello and introduce yourself."
    })
    await task.queue_frames([LLMRunFrame()])
```

### Handshake Required (Client/Server Room-based WebRTC)

For client/server applications using room-based WebRTC, a handshake ensures both sides are ready and the client won't miss the opening message:

```python  theme={null}
@rtvi.event_handler("on_client_ready")
async def on_client_ready(rtvi):
    await rtvi.set_bot_ready()  # Confirm readiness to client
    # Start the conversation
    await task.queue_frames([LLMRunFrame()])
```

<Note>
  For client/server room-based connections, waiting for `on_client_ready` is
  crucial - starting too early can cause the client to miss part of the initial
  message.
</Note>

## Process Isolation

Each session runs its own dedicated bot instance for:

* **Resource Management**: Dedicated CPU and memory per session
* **Error Isolation**: One session crash doesn't affect others
* **Clean Cleanup**: Resources automatically freed when sessions end

The development runner handles this process management automatically.

## Custom Runners: When You Need More Control

The development runner works for most cases, but sometimes you need custom behavior - specific authentication, custom endpoints, or integration with existing systems. For these cases, you can create your own FastAPI runner.

The development runner source code ([available on GitHub](https://github.com/pipecat-ai/pipecat/blob/main/src/pipecat/runner/run.py)) provides excellent examples for:

**Daily Integration Example:**

```python  theme={null}
from pipecat.runner.daily import configure

@app.post("/start")
async def start_bot(background_tasks: BackgroundTasks):
    async with aiohttp.ClientSession() as session:
        room_url, token = await configure(session)

        # Start bot instance
        background_tasks.add_task(run_bot, room_url, token)

        return {"room_url": room_url, "token": token}
```

**WebSocket Telephony Example:**

```python  theme={null}
from pipecat.runner.utils import parse_telephony_websocket

@app.websocket("/ws")
async def websocket_endpoint(websocket: WebSocket):
    await websocket.accept()

    # Parse provider-specific messages
    transport_type, call_data = await parse_telephony_websocket(websocket)

    # Start bot with parsed data
    await run_telephony_bot(websocket, transport_type, call_data)
```

<Tip>
  Refer to the development runner source code to understand these patterns
  before building custom runners. It handles many edge cases and provides
  battle-tested implementations.
</Tip>

## Key Takeaways

* **Start with the development runner** for fastest development and learning
* **Understand connection types** to choose the right approach for your use case
* **Handle startup timing correctly** - immediate start vs. handshake patterns matter
* **Plan for process isolation** - one bot instance per session is the recommended pattern
* **Reference the source code** when building custom runners for production

## What's Next

Now that you understand session initialization, let's explore the different transport options and how to configure them for your specific needs.

<Card title="Pipeline & Frame Processing" icon="arrow-right" href="/guides/learn/pipeline">
  Learn how Pipecat's pipeline architecture orchestrates frame processing for
  voice AI applications
</Card>

---

# Source: https://docs.pipecat.ai/deployment/pipecat-cloud/rest-reference/endpoint/session-proxy.md

> ## Documentation Index
> Fetch the complete documentation index at: https://docs.pipecat.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Session API

> Send HTTP requests directly to your running Pipecat Cloud agent sessions.

Send HTTP requests to endpoints defined in your running bot. Supports `GET`, `POST`, `PUT`, `PATCH`, `DELETE`, `OPTIONS`, and `HEAD` methods.

## Request Headers

Headers are forwarded to your bot with these exceptions:

* `host` - Excluded
* `content-length` - Excluded
* `authorization` - Excluded (authentication is handled by the API gateway)

<Note>
  Requires base image version `0.1.2` or later. See the [Session API
  guide](/deployment/pipecat-cloud/guides/session-api) for setup instructions
  and examples.
</Note>


## OpenAPI

````yaml GET /{serviceName}/sessions/{sessionId}/{path}
openapi: 3.0.0
info:
  title: Pipecat Cloud - Session API
  version: 1.0.0
  description: Send HTTP requests directly to running Pipecat Cloud agent sessions
servers:
  - url: https://api.pipecat.daily.co/v1/public
    description: Public API server
security:
  - PublicKeyAuth: []
paths:
  /{serviceName}/sessions/{sessionId}/{path}:
    get:
      summary: Send GET request to session
      description: >-
        Proxies a GET request to an endpoint defined in your running bot. The
        response is returned directly from the bot.
      operationId: sessionProxyGet
      parameters:
        - $ref: '#/components/parameters/ServiceName'
        - $ref: '#/components/parameters/SessionId'
        - $ref: '#/components/parameters/Path'
      responses:
        '200':
          description: Successful response from the bot
          content:
            application/json:
              schema:
                $ref: '#/components/schemas/ProxiedResponse'
        '400':
          $ref: '#/components/responses/ServiceNotAvailable'
        '401':
          $ref: '#/components/responses/Unauthorized'
        '404':
          $ref: '#/components/responses/NotFound'
        '500':
          $ref: '#/components/responses/InternalServerError'
      security:
        - PublicKeyAuth: []
components:
  parameters:
    ServiceName:
      name: serviceName
      in: path
      required: true
      description: Name of the deployed agent
      schema:
        type: string
      example: my-agent
    SessionId:
      name: sessionId
      in: path
      required: true
      description: Session ID returned from the start endpoint
      schema:
        type: string
        format: uuid
      example: 57af5437-97a2-4646-9873-a5c5935bd705
    Path:
      name: path
      in: path
      required: true
      description: The endpoint path defined in your bot using the `@app` decorator
      schema:
        type: string
      example: status
  schemas:
    ProxiedResponse:
      type: object
      description: >-
        Response from your bot endpoint. The schema depends on what your bot
        returns.
      additionalProperties: true
      example:
        status: active
        message_count: 5
        user_name: Alice
    ErrorResponse:
      type: object
      properties:
        error:
          type: string
          description: Error message describing what went wrong
        code:
          type: string
          description: Error code for programmatic handling
      required:
        - error
        - code
  responses:
    ServiceNotAvailable:
      description: Service deployment is not available
      content:
        application/json:
          schema:
            $ref: '#/components/schemas/ErrorResponse'
          example:
            error: >-
              Attempt to start agent when deployment is not in ready state. Is
              your image pull secret valid?
            code: PCC-1001
    Unauthorized:
      description: Invalid or missing API key
      content:
        application/json:
          schema:
            $ref: '#/components/schemas/ErrorResponse'
          example:
            error: Unauthorized / token expired
            code: '401'
    NotFound:
      description: Service or session not found
      content:
        application/json:
          schema:
            $ref: '#/components/schemas/ErrorResponse'
          example:
            error: Service not found
            code: '404'
    InternalServerError:
      description: Internal server error
      content:
        application/json:
          schema:
            $ref: '#/components/schemas/ErrorResponse'
          example:
            error: >-
              Internal server error. Please check logs for more information or
              contact support.
            code: '500'
  securitySchemes:
    PublicKeyAuth:
      type: http
      scheme: bearer
      description: >-
        Authentication using a Pipecat Cloud public API key.


        Generate a public API key from your Dashboard (Settings > API Keys >
        Public > Create key) and include it as a Bearer token in the
        Authorization header.

````

---

# Source: https://docs.pipecat.ai/deployment/pipecat-cloud/sdk-reference/sessions.md

> ## Documentation Index
> Fetch the complete documentation index at: https://docs.pipecat.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Session Management

> Managing agent sessions with the Pipecat Cloud Python SDK

The session management classes allow you to start and interact with agent sessions.

## Session

The `Session` class is the primary way to start and interact with agent sessions.

```python  theme={null}
from pipecatcloud.session import Session, SessionParams

session = Session(
    agent_name="my-agent",
    api_key="pk_...",
    params=SessionParams(use_daily=True)
)
```

### Constructor Parameters

<ParamField path="agent_name" type="string" required>
  Name of the deployed agent to interact with.
</ParamField>

<ParamField path="api_key" type="string" required>
  Public API key for authentication.
</ParamField>

<ParamField path="params" type="SessionParams">
  Optional parameters to configure the session.
</ParamField>

### Methods

<ResponseField name="start()" type="async">
  Starts a new session with the specified agent.

  **Returns**

  A dictionary containing session information. If `use_daily` is True, includes `dailyRoom` URL and `dailyToken`.

  **Raises**

  `AgentStartError`: If the session fails to start due to missing API key, agent not found, agent not ready, or capacity limits.
</ResponseField>

## SessionParams

The `SessionParams` class allows you to configure a session.

```python  theme={null}
from pipecatcloud.session import SessionParams

params = SessionParams(
    data={"custom_field": "value"},
    use_daily=True,
    daily_room_properties={"enable_recording": "cloud"}
)
```

### Parameters

<ParamField path="data" type="Dict[str, Any]" default="None">
  Optional dictionary of data to pass to the agent. Must be JSON-serializable.
</ParamField>

<ParamField path="use_daily" type="boolean" default="False">
  If True, creates a Daily WebRTC room for the session, enabling voice
  interaction.
</ParamField>

<ParamField path="daily_room_properties" type="Dict[str, Any]" default="None">
  Optional dictionary of properties to configure the Daily room. Only used when
  `use_daily=True`.

  See [Daily API
  documentation](https://docs.daily.co/reference/rest-api/rooms/config) for
  available properties.
</ParamField>

---

# Source: https://docs.pipecat.ai/server/utilities/audio/silero-vad-analyzer.md

> ## Documentation Index
> Fetch the complete documentation index at: https://docs.pipecat.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# SileroVADAnalyzer

> Voice Activity Detection analyzer using the Silero VAD ONNX model

## Overview

`SileroVADAnalyzer` is a Voice Activity Detection (VAD) analyzer that uses the Silero VAD ONNX model to detect speech in audio streams. It provides high-accuracy speech detection with efficient processing using ONNX runtime.

## Installation

The Silero VAD analyzer requires additional dependencies:

```bash  theme={null}
pip install "pipecat-ai[silero]"
```

## Constructor Parameters

<ParamField path="sample_rate" type="int" default="None">
  Audio sample rate in Hz. Must be either 8000 or 16000.
</ParamField>

<ParamField path="params" type="VADParams" default="VADParams()">
  Voice Activity Detection parameters object

  <Expandable title="properties">
    <ParamField path="confidence" type="float" default="0.7">
      Confidence threshold for speech detection. Higher values make detection more strict. Must be between 0 and 1.
    </ParamField>

    <ParamField path="start_secs" type="float" default="0.2">
      Time in seconds that speech must be detected before transitioning to SPEAKING state.
    </ParamField>

    <ParamField path="stop_secs" type="float" default="0.8">
      Time in seconds of silence required before transitioning back to QUIET state.
    </ParamField>

    <ParamField path="min_volume" type="float" default="0.6">
      Minimum audio volume threshold for speech detection. Must be between 0 and 1.
    </ParamField>
  </Expandable>
</ParamField>

## Usage Example

```python  theme={null}
context = LLMContext(messages)
user_aggregator, assistant_aggregator = LLMContextAggregatorPair(
    context,
    user_params=LLMUserAggregatorParams(
        vad_analyzer=SileroVADAnalyzer(params=VADParams(stop_secs=0.2)),
    ),
)
```

## Technical Details

### Sample Rate Requirements

The analyzer supports two sample rates:

* 8000 Hz (256 samples per frame)
* 16000 Hz (512 samples per frame)

Model Management

* Uses ONNX runtime for efficient inference
* Automatically resets model state every 5 seconds to manage memory
* Runs on CPU by default for consistent performance
* Includes built-in model file

## Notes

* High-accuracy speech detection
* Efficient ONNX-based processing
* Automatic memory management
* Thread-safe for pipeline processing
* Built-in model file included
* CPU-optimized inference
* Supports 8kHz and 16kHz audio

---

# Source: https://docs.pipecat.ai/server/services/video/simli.md

> ## Documentation Index
> Fetch the complete documentation index at: https://docs.pipecat.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Simli

> Real-time AI avatar video generation service using WebRTC streaming

## Overview

`SimliVideoService` integrates with Simli to create real-time AI avatar video experiences using WebRTC streaming. The service processes audio input to generate synchronized avatar video and audio output, handling real-time streaming, audio resampling, and conversation interruptions for engaging conversational AI applications.

<CardGroup cols={2}>
  <Card title="Simli Video API Reference" icon="code" href="https://reference-server.pipecat.ai/en/latest/api/pipecat.services.simli.video.html">
    Pipecat's API methods for Simli video integration
  </Card>

  <Card title="Example Implementation" icon="play" href="https://github.com/pipecat-ai/pipecat/blob/main/examples/foundational/27-simli-layer.py">
    Complete example with avatar streaming
  </Card>

  <Card title="Simli Documentation" icon="book" href="https://docs.simli.com/overview">
    Official Simli API documentation and guides
  </Card>

  <Card title="Simli Platform" icon="microphone" href="https://www.simli.com/">
    Access avatar faces and manage API keys
  </Card>
</CardGroup>

## Installation

To use Simli services, install the required dependency:

```bash  theme={null}
pip install "pipecat-ai[simli]"
```

## Prerequisites

### Simli Account Setup

Before using Simli video services, you need:

1. **Simli Account**: Sign up at [Simli Platform](https://www.simli.com/)
2. **API Key**: Generate an API key from your account dashboard
3. **Face Selection**: Choose or create avatar faces for video generation
4. **WebRTC Setup**: Configure real-time streaming capabilities

### Required Environment Variables

* `SIMLI_API_KEY`: Your Simli API key for authentication
* `SIMLI_FACE_ID`: ID of your avatar face

---

# Source: https://docs.pipecat.ai/server/services/transport/small-webrtc.md

# Source: https://docs.pipecat.ai/client/react-native/transports/small-webrtc.md

# Source: https://docs.pipecat.ai/client/js/transports/small-webrtc.md

# Source: https://docs.pipecat.ai/client/ios/transports/small-webrtc.md

# Source: https://docs.pipecat.ai/client/android/transports/small-webrtc.md

> ## Documentation Index
> Fetch the complete documentation index at: https://docs.pipecat.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Small WebRTC Transport

> WebRTC implementation for Android

The Small WebRTC transport implementation enables real-time audio communication with the Small WebRTC Pipecat transport, using a direct WebRTC connection.

## Installation

Add the transport dependency to your `build.gradle`:

```gradle  theme={null}
implementation "ai.pipecat:small-webrtc-transport:0.3.7"
```

## Usage

Create a client:

```kotlin  theme={null}
val transport = SmallWebRTCTransport.Factory(context, baseUrl)

val options = RTVIClientOptions(
    params = RTVIClientParams(baseUrl = null),
    enableMic = true,
    enableCam = true
)

val client = RTVIClient(transport, callbacks, options)

client.start().withCallback {
    // ...
}
```

## Resources

<CardGroup cols={2}>
  <Card horizontal title="Demo" icon="play" href="https://github.com/pipecat-ai/pipecat-examples/tree/main/p2p-webrtc/video-transform/client/android">
    Demo App
  </Card>

  <Card horizontal title="Source" icon="github" href="https://github.com/pipecat-ai/pipecat-client-android-transports/">
    Client Transports
  </Card>
</CardGroup>

<Card title="Pipecat Android Client Reference" icon="book-open" href="https://docs-android.rtvi.ai/">
  Complete API documentation for the Pipecat Android client.
</Card>

---

# Source: https://docs.pipecat.ai/server/utilities/smart-turn/smart-turn-overview.md

> ## Documentation Index
> Fetch the complete documentation index at: https://docs.pipecat.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Smart Turn Overview

> Advanced conversational turn detection powered by the smart-turn model

## Overview

Smart Turn Detection is an advanced feature in Pipecat that determines when a user has finished speaking and the bot should respond. Unlike basic Voice Activity Detection (VAD) which only detects speech vs. non-speech, Smart Turn Detection uses a machine learning model to recognize natural conversational cues like intonation patterns and linguistic signals.

<CardGroup cols={3}>
  <Card title="Smart Turn Model" icon="github" href="https://github.com/pipecat-ai/smart-turn">
    Open source model for advanced conversational turn detection. Contribute to
    model training and development.
  </Card>

  <Card title="Data Collector" icon="microphone" href="https://turn-training.pipecat.ai/">
    Contribute conversational data to improve the smart-turn model
  </Card>

  <Card title="Data Classifier" icon="check-circle" href="https://smart-turn-dataset.pipecat.ai/">
    Help classify turn completion patterns in conversations
  </Card>
</CardGroup>

Pipecat provides `LocalSmartTurnAnalyzerV3` which runs inference locally using ONNX. This is the recommended approach due to the fast CPU inference times in Smart Turn v3.

## Installation

```bash  theme={null}
pip install "pipecat-ai[local-smart-turn-v3]"
```

The Smart Turn model weights are bundled with Pipecat, so no need to download these separately.

## Integration with User Turn Strategies

Smart Turn Detection is integrated into your application by configuring a `TurnAnalyzerUserTurnStopStrategy` with `LocalSmartTurnAnalyzerV3` in your context aggregator:

```python  theme={null}
from pipecat.audio.turn.smart_turn.local_smart_turn_v3 import LocalSmartTurnAnalyzerV3
from pipecat.audio.vad.silero import SileroVADAnalyzer
from pipecat.audio.vad.vad_analyzer import VADParams
from pipecat.processors.aggregators.llm_response_universal import (
    LLMContextAggregatorPair,
    LLMUserAggregatorParams,
)
from pipecat.transports.base_transport import TransportParams
from pipecat.turns.user_stop import TurnAnalyzerUserTurnStopStrategy
from pipecat.turns.user_turn_strategies import UserTurnStrategies

transport = SmallWebRTCTransport(
    webrtc_connection=webrtc_connection,
    params=TransportParams(
        audio_in_enabled=True,
    ),
)

user_aggregator, assistant_aggregator = LLMContextAggregatorPair(
    context,
    user_params=LLMUserAggregatorParams(
        user_turn_strategies=UserTurnStrategies(
            stop=[TurnAnalyzerUserTurnStopStrategy(
                turn_analyzer=LocalSmartTurnAnalyzerV3()
            )]
        ),
        vad_analyzer=SileroVADAnalyzer(params=VADParams(stop_secs=0.2)),
    ),
)
```

<Tip>
  Smart Turn Detection requires VAD to be enabled and works best when the VAD analyzer is set to a short `stop_secs` value. We recommend 0.2 seconds.

  ```python  theme={null}
  vad_analyzer=SileroVADAnalyzer(params=VADParams(stop_secs=0.2))
  ```
</Tip>

## Configuration

The `SmartTurnParams` class configures turn detection behavior:

<ParamField path="stop_secs" type="float" default="3.0">
  Duration of silence in seconds required before triggering a silence-based end
  of turn
</ParamField>

<ParamField path="pre_speech_ms" type="float" default="0.0">
  Amount of audio (in milliseconds) to include before speech is detected
</ParamField>

<ParamField path="max_duration_secs" type="float" default="8.0">
  Maximum allowed segment duration in seconds. For segments longer than this
  value, a rolling window is used.
</ParamField>

## Local Smart Turn

The `LocalSmartTurnAnalyzerV3` runs inference locally. Version 3 of the model supports fast CPU inference on ordinary cloud instances.

### Constructor Parameters

<ParamField path="smart_turn_model_path" type="Optional[str]" default="None">
  Path to the Smart Turn v3 ONNX file containing the model weights. Download this from
  [https://huggingface.co/pipecat-ai/smart-turn-v3/tree/main](https://huggingface.co/pipecat-ai/smart-turn-v3/tree/main)

  This parameter is optional, as Pipecat includes a copy of the model internally, and this
  is used if the path is unset.
</ParamField>

<ParamField path="sample_rate" type="Optional[int]" default="None">
  Audio sample rate (will be set by the transport if not provided)
</ParamField>

<ParamField path="params" type="SmartTurnParams" default="SmartTurnParams()">
  Configuration parameters for turn detection
</ParamField>

### Example

```python  theme={null}
from pipecat.audio.turn.smart_turn.local_smart_turn_v3 import LocalSmartTurnAnalyzerV3
from pipecat.audio.vad.silero import SileroVADAnalyzer
from pipecat.audio.vad.vad_analyzer import VADParams
from pipecat.processors.aggregators.llm_response_universal import (
    LLMContextAggregatorPair,
    LLMUserAggregatorParams,
)
from pipecat.transports.base_transport import TransportParams
from pipecat.turns.user_stop import TurnAnalyzerUserTurnStopStrategy
from pipecat.turns.user_turn_strategies import UserTurnStrategies

# Create the transport
transport = SmallWebRTCTransport(
    webrtc_connection=webrtc_connection,
    params=TransportParams(
        audio_in_enabled=True,
        audio_out_enabled=True,
    ),
)

# Configure Smart Turn Detection via user turn strategies
user_aggregator, assistant_aggregator = LLMContextAggregatorPair(
    context,
    user_params=LLMUserAggregatorParams(
        user_turn_strategies=UserTurnStrategies(
            stop=[TurnAnalyzerUserTurnStopStrategy(
                turn_analyzer=LocalSmartTurnAnalyzerV3()
            )]
        ),
        vad_analyzer=SileroVADAnalyzer(params=VADParams(stop_secs=0.2)),
    ),
)
```

## How It Works

Smart Turn Detection continuously analyzes audio streams to identify natural turn completion points:

1. **Audio Buffering**: The system continuously buffers audio with timestamps, maintaining a small buffer of pre-speech audio.
2. **VAD Processing**: Voice Activity Detection (using the Silero model) detects when there is a pause in the user's speech.
3. **Smart Turn Analysis**: When VAD detects a pause in speech, the Smart Turn model analyzes the audio from the most recent 8 seconds of the user's turn, and makes a decision about whether the turn is complete or incomplete.

The system includes a fallback mechanism: if a turn is classified as incomplete but silence continues for longer than `stop_secs`, the turn is automatically marked as complete.

## Notes

* The model supports 23 languages, see the [source repository](https://github.com/pipecat-ai/smart-turn) for more details
* You can adjust the `stop_secs` parameter based on your application's needs for responsiveness
* Smart Turn generally provides a more natural conversational experience but is computationally more intensive than simple VAD
* `LocalSmartTurnAnalyzerV3` is designed to run on CPU, and inference can be performed on low-cost cloud instances in under 100ms. However, by installing the `onnxruntime-gpu` dependency, you can achieve higher performance by making use of GPU inference.

---

# Source: https://docs.pipecat.ai/deployment/pipecat-cloud/guides/smart-turn.md

> ## Documentation Index
> Fetch the complete documentation index at: https://docs.pipecat.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Smart Turn Detection

> Advanced conversational turn detection

## Overview

Smart Turn Detection uses an advanced machine learning model to determine when a user has finished speaking and your bot should respond. Unlike basic Voice Activity Detection (VAD) which only detects speech vs. non-speech, Smart Turn Detection recognizes natural conversational cues like intonation patterns and linguistic signals for more natural conversations.

<CardGroup cols={2}>
  <Card title="GitHub Repository" icon="github" href="https://github.com/pipecat-ai/smart-turn">
    Open source model for conversational turn detection
  </Card>

  <Card title="Model weights" icon="face-smiling-hands" href="https://huggingface.co/pipecat-ai/smart-turn-v3/tree/main">
    ONNX weights file for Smart Turn v3
  </Card>
</CardGroup>

## Key Benefits

* **Natural conversations**: More human-like turn-taking patterns
* **Free to use**: The model is fully open source
* **Scalable**: Smart Turn v3 supports fast CPU inference directly inside your Pipecat Cloud instance

## Quick Start

To enable Smart Turn Detection in your Pipecat Cloud bot, configure a `TurnAnalyzerUserTurnStopStrategy` with `LocalSmartTurnAnalyzerV3` in your context aggregator.

The model weights are bundled with Pipecat, so there's no need to download them separately.

```python  theme={null}
from pipecat.audio.turn.smart_turn.local_smart_turn_v3 import LocalSmartTurnAnalyzerV3
from pipecat.audio.vad.silero import SileroVADAnalyzer
from pipecat.audio.vad.vad_analyzer import VADParams
from pipecat.processors.aggregators.llm_response_universal import (
    LLMContextAggregatorPair,
    LLMUserAggregatorParams,
)
from pipecat.transports.daily.transport import DailyParams, DailyTransport
from pipecat.turns.user_stop import TurnAnalyzerUserTurnStopStrategy
from pipecat.turns.user_turn_strategies import UserTurnStrategies

async def main(room_url: str, token: str):
    transport = DailyTransport(
        room_url,
        token,
        "Voice AI Bot",
        DailyParams(
            audio_in_enabled=True,
            audio_out_enabled=True,
        ),
    )

    # Configure Smart Turn Detection via user turn strategies
    user_aggregator, assistant_aggregator = LLMContextAggregatorPair(
        context,
        user_params=LLMUserAggregatorParams(
            user_turn_strategies=UserTurnStrategies(
                stop=[TurnAnalyzerUserTurnStopStrategy(
                    turn_analyzer=LocalSmartTurnAnalyzerV3()
                )]
            ),
            vad_analyzer=SileroVADAnalyzer(params=VADParams(stop_secs=0.2)),
        ),
    )

    # Continue with your pipeline setup...
```

<Tip>
  Smart Turn Detection requires VAD to be enabled with `stop_secs=0.2`. This
  value mimics the training data and allows Smart Turn to dynamically adjust
  timing based on the model's predictions.
</Tip>

## How It Works

1. **Audio Analysis**: The system continuously analyzes incoming audio for speech patterns
2. **VAD Processing**: Voice Activity Detection segments audio into speech and silence
3. **Turn Classification**: When VAD detects a pause, the ML model analyzes the speech segment for natural completion cues
4. **Smart Response**: The model determines if the turn is complete or if the user is likely to continue speaking

## Training Data Collection

The smart-turn model is trained on real conversational data collected through these applications. Help us improve the model by contributing your own data or classifying existing data:

<CardGroup cols={2}>
  <Card title="Data Collector" icon="microphone" href="https://turn-training.pipecat.ai/">
    Contribute conversational data to improve the model
  </Card>

  <Card title="Data Classifier" icon="check-circle" href="https://smart-turn-dataset.pipecat.ai/">
    Help classify turn completion patterns in conversations
  </Card>
</CardGroup>

## More information

For more details on Smart Turn, see the following links:

<CardGroup cols={2}>
  <Card title="Smart Turn Overview" href="../../../server/utilities/smart-turn/smart-turn-overview">
    More details about the Pipecat Smart Turn integration
  </Card>
</CardGroup>

---

# Source: https://docs.pipecat.ai/server/services/stt/soniox.md

> ## Documentation Index
> Fetch the complete documentation index at: https://docs.pipecat.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Soniox

> Speech-to-text service implementation using Soniox's WebSocket API

## Overview

`SonioxSTTService` provides real-time speech-to-text transcription using Soniox's WebSocket API with support for over 60 languages, custom context, multiple languages in the same conversation, and advanced features for accurate multilingual transcription.

<CardGroup cols={2}>
  <Card title="Soniox STT API Reference" icon="code" href="https://pipecat-docs.readthedocs.io/en/latest/api/pipecat.services.soniox.stt.html">
    Pipecat's API methods for Soniox STT integration
  </Card>

  <Card title="Example Implementation" icon="play" href="https://github.com/pipecat-ai/pipecat/blob/main/examples/foundational/07za-interruptible-soniox.py">
    Complete example with interruption handling
  </Card>

  <Card title="Soniox Documentation" icon="book" href="https://soniox.com/docs/speech-to-text/get-started">
    Official Soniox documentation and features
  </Card>

  <Card title="Soniox Console" icon="microphone" href="https://console.soniox.com/">
    Access multilingual models and API keys
  </Card>
</CardGroup>

## Installation

To use Soniox services, install the required dependencies:

```bash  theme={null}
pip install "pipecat-ai[soniox]"
```

## Prerequisites

### Soniox Account Setup

Before using Soniox STT services, you need:

1. **Soniox Account**: Sign up at [Soniox Console](https://console.soniox.com/)
2. **API Key**: Generate an API key from your console dashboard
3. **Language Selection**: Choose from 60+ supported languages and models

### Required Environment Variables

* `SONIOX_API_KEY`: Your Soniox API key for authentication

---

# Source: https://docs.pipecat.ai/server/utilities/audio/soundfile-mixer.md

> ## Documentation Index
> Fetch the complete documentation index at: https://docs.pipecat.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# SoundfileMixer

> Audio mixer for combining real-time audio with sound files

## Overview

`SoundfileMixer` is an audio mixer that combines incoming audio with audio from files. It supports multiple audio file formats through the soundfile library and can handle runtime volume adjustments and sound switching.

## Installation

The soundfile mixer requires additional dependencies:

```bash  theme={null}
pip install "pipecat-ai[soundfile]"
```

## Constructor Parameters

<ParamField path="sound_files" type="Mapping[str, str]" required>
  Dictionary mapping sound names to file paths. Files must be mono (single channel).
</ParamField>

<ParamField path="default_sound" type="str" required>
  Name of the default sound to play (must be a key in sound\_files).
</ParamField>

<ParamField path="volume" type="float" default="0.4">
  Initial volume for the mixed sound. Values typically range from 0.0 to 1.0, but can go higher.
</ParamField>

<ParamField path="loop" type="bool" default="true">
  Whether to loop the sound file when it reaches the end.
</ParamField>

## Control Frames

<ParamField path="MixerUpdateSettingsFrame" type="Frame">
  Updates mixer settings at runtime

  <Expandable title="properties">
    <ParamField path="sound" type="str">
      Changes the current playing sound (must be a key in sound\_files)
    </ParamField>

    <ParamField path="volume" type="float">
      Updates the mixing volume
    </ParamField>

    <ParamField path="loop" type="bool">
      Updates whether the sound should loop
    </ParamField>
  </Expandable>
</ParamField>

<ParamField path="MixerEnableFrame" type="Frame">
  Enables or disables the mixer

  <Expandable title="properties">
    <ParamField path="enable" type="bool">
      Whether mixing should be enabled
    </ParamField>
  </Expandable>
</ParamField>

## Usage Example

```python  theme={null}
# Initialize mixer with sound files
mixer = SoundfileMixer(
    sound_files={"office": "office_ambience.wav"},
    default_sound="office",
    volume=2.0,
)

# Add to transport
transport = DailyTransport(
    room_url,
    token,
    "Audio Bot",
    DailyParams(
        audio_out_enabled=True,
        audio_out_mixer=mixer,
    ),
)

# Control mixer at runtime
await task.queue_frame(MixerUpdateSettingsFrame({"volume": 0.5}))
await task.queue_frame(MixerEnableFrame(False))  # Disable mixing
await task.queue_frame(MixerEnableFrame(True))   # Enable mixing
```

## Notes

* Supports any audio format that soundfile can read
* Automatically resamples audio files to match output sample rate
* Files must be mono (single channel)
* Thread-safe for pipeline processing
* Can dynamically switch between multiple sound files
* Volume can be adjusted in real-time
* Mixing can be enabled/disabled on demand

---

# Source: https://docs.pipecat.ai/guides/learn/speech-input.md

> ## Documentation Index
> Fetch the complete documentation index at: https://docs.pipecat.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Speech Input & Turn Detection

> Learn how Pipecat detects user turns using VAD, transcriptions, and turn detection models

A key to natural conversations is properly detecting when the user starts and stops speaking. This is more nuanced than simply detecting audio; a brief pause doesn't always mean the user is done talking.

## Overview

Pipecat uses [user turn strategies](/server/utilities/turn-management/user-turn-strategies) to determine when user turns start and end. These strategies can use different techniques:

**For detecting turn start:**

* Voice Activity Detection (VAD): triggers when speech is detected
* Transcription-based (fallback): triggers when transcription is received but VAD didn't detect speech
* Minimum words: waits for a minimum number of spoken words before triggering

**For detecting turn end:**

* Transcription-based: analyzes transcription to determine when the user is done
* Turn detection model: uses AI to understand if the user has finished their thought

Custom strategies can also be implemented for specific use cases. By combining these techniques, you can create responsive yet natural conversations that don't interrupt users mid-sentence or wait too long after they've finished.

## Voice Activity Detection (VAD)

### What VAD Does

VAD is responsible for detecting when a user starts and stops speaking. Pipecat uses the [Silero VAD](https://github.com/snakers4/silero-vad), an open-source model that runs locally on CPU with minimal overhead.

**Performance characteristics:**

* Processes 30+ms audio chunks in less than 1ms
* Runs on a single CPU thread
* Minimal system resource impact

### VAD Configuration

VAD is configured through `VADParams` in the `LLMContextAggregatorPair`:

```python  theme={null}
from pipecat.audio.vad.silero import SileroVADAnalyzer
from pipecat.audio.vad.vad_analyzer import VADParams
from pipecat.processors.aggregators.llm_response_universal import (
    LLMContextAggregatorPair,
    LLMUserAggregatorParams,
)

vad_analyzer = SileroVADAnalyzer(
    params=VADParams(
        confidence=0.7,      # Minimum confidence for voice detection
        start_secs=0.2,      # Time to wait before confirming speech start
        stop_secs=0.8,       # Time to wait before confirming speech stop
        min_volume=0.6,      # Minimum volume threshold
    )
)

user_aggregator, assistant_aggregator = LLMContextAggregatorPair(
    context,
    user_params=LLMUserAggregatorParams(vad_analyzer=vad_analyzer)),
)
```

### Key Parameters

**`start_secs` (default: 0.2)**

* How long a user must speak before VAD confirms speech has started
* Lower values = more responsive, but may trigger on brief sounds
* Higher values = less sensitive, but may miss quick utterances, like "yes", "no", or "ok"

**`stop_secs` (default: 0.8)**

* How much silence must be detected before confirming speech has stopped
* Critical for turn-taking behavior
* Modified automatically when using turn detection

**`confidence` and `min_volume`**

* Generally work well with defaults
* Only adjust after extensive testing with your specific audio conditions

<Warning>
  Changing confidence and min\_volume requires careful profiling to ensure
  optimal performance across different audio environments and use cases.
</Warning>

## User Turn Detection

While VAD detects speech vs. silence, it can't understand linguistic context. A pause doesn't mean the user is done. User turn strategies interpret VAD signals and transcriptions to determine actual turn boundaries.

### How It Works

1. **Turn Start**: When VAD detects speech (or transcription arrives), the start strategy emits `UserStartedSpeakingFrame` and optionally triggers an interruption
2. **Turn End**: When the stop strategy determines the user is done, it emits `UserStoppedSpeakingFrame`

<Note>
  VAD also emits its own frames (`VADUserStartedSpeakingFrame`,
  `VADUserStoppedSpeakingFrame`) which indicate raw speech/silence detection.
  These are inputs to the turn strategies, not the final turn decisions.
</Note>

### Detecting Turn End

Turn end detection uses transcription to understand what the user said:

**Transcription-based (Default)**: Analyzes transcription to determine when the user has finished speaking.

```python  theme={null}
from pipecat.turns.user_stop import TranscriptionUserTurnStopStrategy

stop_strategy = TranscriptionUserTurnStopStrategy()
```

**Smart Turn Model**: Uses an AI model to analyze audio and determine if the user has finished their thought.

```python  theme={null}
from pipecat.audio.turn.smart_turn.local_smart_turn_v3 import LocalSmartTurnAnalyzerV3
from pipecat.turns.user_stop import TurnAnalyzerUserTurnStopStrategy

stop_strategy = TurnAnalyzerUserTurnStopStrategy(
    turn_analyzer=LocalSmartTurnAnalyzerV3()
)
```

When using Smart Turn, configure VAD with a low `stop_secs` (0.2) so the model can analyze speech quickly.

<CardGroup cols={2}>
  <Card title="User Turn Strategies" icon="arrows-turn-to-dots" href="/server/utilities/turn-management/user-turn-strategies">
    Complete reference for start and stop strategies
  </Card>

  <Card title="Smart Turn Overview" icon="brain" href="/server/utilities/smart-turn/smart-turn-overview">
    Smart Turn model implementation guide
  </Card>
</CardGroup>

### Interruptions

Interruptions stop the bot when the user starts speaking. This is controlled by the `enable_interruptions` parameter on start strategies (enabled by default).

When a user turn starts with interruptions enabled:

1. Bot immediately stops speaking
2. Pending audio and text is cleared
3. Pipeline ready for new user input

To disable interruptions:

```python  theme={null}
from pipecat.turns.user_start import VADUserTurnStartStrategy

start_strategy = VADUserTurnStartStrategy(enable_interruptions=False)
```

<Note>
  Keep interruptions enabled (default) for natural conversations. This enables
  users to interrupt the bot mid-sentence, just like human conversations.
</Note>

## Best Practices

### Optimal Configuration

**For most voice AI use cases:**

```python  theme={null}
from pipecat.audio.turn.smart_turn.local_smart_turn_v3 import LocalSmartTurnAnalyzerV3
from pipecat.processors.aggregators.llm_response_universal import (
    LLMContextAggregatorPair,
    LLMUserAggregatorParams,
)
from pipecat.turns.user_stop import TurnAnalyzerUserTurnStopStrategy
from pipecat.turns.user_turn_strategies import UserTurnStrategies

# Responsive VAD with turn detection
vad_params = VADParams(
    start_secs=0.2,
    stop_secs=0.2 if using_turn_detection else 0.8,
)

transport = YourTransport(
    params=TransportParams(),
)

# Configure Smart Turn via user turn strategies (if using turn detection)
user_aggregator, assistant_aggregator = LLMContextAggregatorPair(
    context,
    user_params=LLMUserAggregatorParams(
        user_turn_strategies=UserTurnStrategies(
            stop=[TurnAnalyzerUserTurnStopStrategy(
                turn_analyzer=LocalSmartTurnAnalyzerV3()
            )]
        ) if using_smart_turn_detection else None,
        vad_analyzer=SileroVADAnalyzer(params=vad_params),
    ),
)

task = PipelineTask(pipeline)
```

### Performance Considerations

* **Use local VAD**: 150-200ms faster than remote VAD services
* **Tune for your use case**: Test with real audio conditions
* **Monitor CPU usage**: VAD adds minimal overhead but monitor in production
* **Consider turn detection**: Improves conversation quality but adds complexity

## Key Takeaways

* **VAD detects speech activity** but turn detection understands conversation context
* **Configuration affects user experience** - tune parameters for your specific use case
* **System frames coordinate behavior** - enable interruptions and natural turn-taking
* **Local processing is faster** - Silero VAD provides low-latency speech detection
* **Turn detection improves quality** - but requires careful VAD configuration

## What's Next

Now that you understand how speech input is detected and processed, let's explore how that audio gets converted to text through speech recognition.

<Card title="Speech to Text" icon="arrow-right" href="/guides/learn/speech-to-text">
  Learn how to configure speech recognition in your voice AI pipeline
</Card>

---

# Source: https://docs.pipecat.ai/guides/learn/speech-to-text.md

> ## Documentation Index
> Fetch the complete documentation index at: https://docs.pipecat.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Speech to Text

> Learn how to configure speech recognition to convert user audio into text in your Pipecat pipeline

**Speech to Text (STT)** services are responsible for converting user audio into text transcriptions. They receive audio input from users and provide real-time transcriptions that your bot can process and respond to.

## Pipeline Placement

STT processors must be positioned correctly in your pipeline to receive and process audio frames:

```python  theme={null}
pipeline = Pipeline([
    transport.input(),             # Creates InputAudioRawFrames
    stt,                           # Processes audio → creates TranscriptionFrames
    context_aggregator.user(),     # Uses transcriptions for context
    llm,
    tts,
    transport.output(),
])
```

**Placement requirements:**

* **After `transport.input()`**: STT needs `InputAudioRawFrame`s from the transport
* **Before context processing**: Transcriptions must be available for context aggregation
* **Before LLM processing**: Text must be ready for language model input

## STT Service Types

Pipecat provides two types of STT services based on how they process audio:

### 1. STTService (Streaming)

**How it works:**

* Establishes a WebSocket connection to the STT provider
* Continuously streams audio for real-time transcription
* Lower latency due to persistent connection

### 2. SegmentedSTTService (HTTP-based)

**How it works:**

* Uses local VAD (Voice Activity Detection) to chunk speech
* Sends audio segments to STT service as wav files
* Higher latency due to segmentation and HTTP POST requests

<Note>
  STT services are modular and can be swapped out with no additional overhead.
  You can easily switch between streaming and segmented services based on your
  needs.
</Note>

## Supported STT Services

Pipecat supports a wide range of STT providers to fit different needs and budgets:

<Card title="Supported STT Services" icon="list" href="/server/services/supported-services#speech-to-text">
  View the complete list of supported speech-to-text providers
</Card>

Popular options include:

<CardGroup cols={3}>
  <Card title="Deepgram" icon="microphone" href="/server/services/stt/deepgram">
    Fast, accurate streaming STT with excellent real-time performance
  </Card>

  <Card title="Speechmatics" icon="waveform" href="/server/services/stt/speechmatics">
    Advanced speech recognition with strong accent and dialect handling
  </Card>

  <Card title="AssemblyAI" icon="brain" href="/server/services/stt/assemblyai">
    AI-powered transcription with speaker diarization and sentiment analysis
  </Card>

  <Card title="Gladia" icon="sparkles" href="/server/services/stt/gladia">
    High-performance STT with multilingual support and custom models
  </Card>

  <Card title="Azure Speech" icon="microsoft" href="/server/services/stt/azure">
    Microsoft's enterprise-grade STT service with extensive language support
  </Card>

  <Card title="Google Speech-to-Text" icon="google" href="/server/services/stt/google">
    Reliable transcription with strong language model integration
  </Card>
</CardGroup>

## STT Configuration

### Service-Specific Configuration

Each STT service has its own customization options. Refer to specific service documentation for details:

<Card title="Individual STT Services" icon="screwdriver-wrench" href="/server/services/supported-services#speech-to-text">
  Explore configuration options for each supported STT provider
</Card>

For example, let's look at configuring the **DeepgramSTTService** using the `LiveOptions` class:

```python  theme={null}
from deepgram import LiveOptions
from pipecat.services.deepgram.stt import DeepgramSTTService
from pipecat.transcriptions.language import Language

# Configure using LiveOptions for full control
live_options = LiveOptions(
    model="nova-2",
    language=Language.EN_US,
    interim_results=True,        # Enable interim transcripts
    smart_format=True,           # Enable punctuation and formatting
    punctuate=True,              # Add punctuation
    profanity_filter=True,       # Filter profanity
    vad_events=False,            # Use pipeline VAD instead
)

stt = DeepgramSTTService(
    api_key=os.getenv("DEEPGRAM_API_KEY"),
    live_options=live_options,
)
```

### STTService Base Class Configuration

All STT services inherit from the STTService base class. The base class has base configuration options which are set with smart defaults:

```python  theme={null}
stt = YourSTTService(
    # Service-specific options...
    audio_passthrough=True,      # Pass audio frames downstream (recommended)
    sample_rate=16000,           # Audio sample rate (better set in PipelineParams)
)
```

**Key options:**

* **`audio_passthrough=True`**: Allows audio frames to continue downstream to other processors (like audio recording)
* **`sample_rate`**: Audio sampling rate - best practice is to **set the `audio_in_sample_rate` in `PipelineParams` for consistency**

<Warning>
  Setting `audio_passthrough=False` will stop audio frames from being passed
  downstream, which may break audio recording or other audio-dependent
  processors.
</Warning>

### Pipeline-Level Audio Configuration

Instead of setting sample rates on individual services, configure them pipeline-wide:

```python  theme={null}
task = PipelineTask(
    pipeline,
    params=PipelineParams(
        audio_in_sample_rate=16000,   # All input processors use this rate
        audio_out_sample_rate=24000,  # All output processors use this rate
    ),
)
```

This ensures all audio processors use consistent sample rates without manual configuration.

<Tip>
  Always set audio sample rates in `PipelineParams` to avoid mismatches between
  different audio processors. This simplifies configuration and ensures
  consistent audio quality across your pipeline.
</Tip>

## Best Practices

### Enable Interim Results

When available, enable interim transcripts for better user experience:

```python  theme={null}
stt = DeepgramSTTService(
    api_key=os.getenv("DEEPGRAM_API_KEY"),
    live_options=LiveOptions(
      interim_results=True,
    )
)
```

**Benefits:**

* Notifies context aggregation that more text is coming
* Prevents premature LLM completions
* Enables interruption detection
* Improves conversation flow

### Enable Punctuation and Formatting

Use smart formatting when available:

```python  theme={null}
stt = DeepgramSTTService(
    api_key=os.getenv("DEEPGRAM_API_KEY"),
    live_options=LiveOptions(
        smart_format=True,     # Adds punctuation and capitalization
        profanity_filter=True, # Optional content filtering
    )
)
```

**Benefits:**

* Professional-looking transcripts
* Better LLM comprehension
* Eliminates post-processing needs
* Improved context understanding

### Use Local VAD

While many STT services provide Voice Activity Detection, use Pipecat's local Silero VAD for better performance:

```python  theme={null}
from pipecat.audio.vad.silero import SileroVADAnalyzer

# Configure in context aggregator
user_aggregator, assistant_aggregator = LLMContextAggregatorPair(
    context,
    user_params=LLMUserAggregatorParams(
        vad_analyzer=SileroVADAnalyzer(params=VADParams(stop_secs=0.2)),
    ),
)
```

**Advantages:**

* **150-200ms faster** speech detection (no network round trip)
* More responsive conversation flow
* Better interruption handling
* Reduced latency overall

## Key Takeaways

* **Pipeline placement matters** - STT must come after transport input, before context processing
* **Service types differ** - streaming services have lower latency than segmented
* **Services are modular** - easily swap providers without code changes
* **Best practices improve performance** - use interim results, formatting, and local VAD
* **Configuration affects quality** - proper setup significantly impacts transcription accuracy

## What's Next

Now that you understand speech recognition, let's explore how to manage conversation context and memory in your voice AI bot.

<Card title="Context Management" icon="arrow-right" href="/guides/learn/context-management">
  Learn how to handle conversation history and context in your pipeline
</Card>

---

# Source: https://docs.pipecat.ai/server/services/tts/speechmatics.md

# Source: https://docs.pipecat.ai/server/services/stt/speechmatics.md

> ## Documentation Index
> Fetch the complete documentation index at: https://docs.pipecat.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Speechmatics

> Speech-to-text service implementation using Speechmatics' real-time transcription STT API

## Overview

`SpeechmaticsSTTService` enables real-time speech transcription using Speechmatics' WebSocket API with partial and final results, speaker diarization, and end of utterance detection (VAD) for comprehensive conversation analysis.

<Note>
  Since Speechmatics provides its own user turn start and end detection, you
  should use `ExternalUserTurnStrategies` to let Speechmatics handle turn
  management. See [User Turn
  Strategies](/server/utilities/turn-management/user-turn-strategies) for
  configuration details.
</Note>

<CardGroup cols={2}>
  <Card title="Speechmatics STT API Reference" icon="code" href="https://reference-server.pipecat.ai/en/latest/api/pipecat.services.speechmatics.stt.html">
    Pipecat's API methods for Speechmatics STT integration
  </Card>

  <Card title="Example Implementation" icon="play" href="https://github.com/pipecat-ai/pipecat/blob/main/examples/foundational/07a-interruptible-speechmatics.py">
    Complete example with interruption handling
  </Card>

  <Card title="Speechmatics Documentation" icon="book" href="https://docs.speechmatics.com/rt-api-ref">
    Official Speechmatics documentation and features
  </Card>

  <Card title="Speaker Diarization Guide" icon="microphone" href="https://docs.speechmatics.com/speech-to-text/features/diarization#speaker-diarization">
    Learn about separating different speakers in audio
  </Card>
</CardGroup>

## Installation

To use Speechmatics services, install the required dependencies:

```bash  theme={null}
pip install "pipecat-ai[speechmatics]"
```

## Prerequisites

### Speechmatics Account Setup

Before using Speechmatics STT services, you need:

1. **Speechmatics Account**: Sign up at [Speechmatics](https://www.speechmatics.com/)
2. **API Key**: Generate an API key from your account dashboard
3. **Feature Selection**: Configure transcription features like speaker diarization

### Select Endpoint

Speechmatics STT supports the following endpoints (defaults to `EU2`):

| Region | Environment   | STT Endpoint                     | Access                    |
| ------ | ------------- | -------------------------------- | ------------------------- |
| EU     | EU1           | `wss://neu.rt.speechmatics.com/` | Self-Service / Enterprise |
| EU     | EU2 (Default) | `wss://eu2.rt.speechmatics.com/` | Self-Service / Enterprise |
| US     | US1           | `wss://wus.rt.speechmatics.com/` | Enterprise                |

### Required Environment Variables

* `SPEECHMATICS_API_KEY`: Your Speechmatics API key for authentication
* `SPEECHMATICS_RT_URL`: Speechmatics endpoint URL (optional, defaults to EU2)

## End of Turn detection

The Speechmatics STT service supports Pipecat's own end of turn detection (Silero VAD and Smart Turn) without any additional configuration. When using Pipecat's features, the `turn_detection_mode` must be set to `TurnDetectionMode.EXTERNAL` (which is the default).

### Default mode

By default, Speechmatics uses signals from Pipecat's VAD / smart turn detection as input to trigger the end of turn and finalization of the current transcript segment. This provides a seamless integration where Pipecat's voice activity detection and turn detection work in conjunction with Speechmatics' real-time processing capabilities.

<Note>
  If you wish to use features such as focussing on or ignoring other speakers,
  then you may see benefit from using `TurnDetectionMode.ADAPTIVE` or
  `TurnDetectionMode.SMART_TURN` modes.
</Note>

### Adaptive End of Turn detection

This mode looks at the content of the speech, pace of speaking and other acoustic information (using VAD) to determine when the user has finished speaking. This is especially important when using the plugin's ability to focus on a specific speaker and not have other speakers interrupt the agent / conversation.

To use this mode, set the `turn_detection_mode` to `TurnDetectionMode.ADAPTIVE` in your STT configuration. You must also remove any other VAD / smart turn features within Pipecat to ensure that there is not a conflict.

```python  theme={null}
transport_params = TransportParams(
    audio_in_enabled=True,
    audio_out_enabled=True,
    # vad_analyzer=... <- REMOVE (use Speechmatics' built-in VAD)
    # turn_analyzer=... <- REMOVE (use Speechmatics' built-in end-of-turn detection)
)

...

stt = SpeechmaticsSTTService(
    api_key=os.getenv("SPEECHMATICS_API_KEY"),
    params=SpeechmaticsSTTService.InputParams(
        language=Language.EN,
        turn_detection_mode=SpeechmaticsSTTService.TurnDetectionMode.ADAPTIVE,
        speaker_active_format="<{speaker_id}>{text}</{speaker_id}>",
    ),
)
```

### Smart Turn detection

Further to `ADAPTIVE`, Speechmatics also provides its own smart turn detection which combines VAD and the use of Smart Turn v3 from Pipecat. This can be enabled by setting the `turn_detection_mode` parameter to `TurnDetectionMode.SMART_TURN`.

```python  theme={null}
transport_params = TransportParams(
    audio_in_enabled=True,
    audio_out_enabled=True,
    # vad_analyzer=... <- REMOVE (use Speechmatics' built-in VAD)
    # turn_analyzer=... <- REMOVE (use Speechmatics' built-in end-of-turn detection)
)

...

stt = SpeechmaticsSTTService(
    api_key=os.getenv("SPEECHMATICS_API_KEY"),
    params=SpeechmaticsSTTService.InputParams(
        language=Language.EN,
        turn_detection_mode=SpeechmaticsSTTService.TurnDetectionMode.SMART_TURN,
        speaker_active_format="<{speaker_id}>{text}</{speaker_id}>",
    ),
)
```

## Speaker Diarization

Speechmatics STT supports speaker diarization, which separates out different speakers in the audio. The identity of each speaker is returned in the TranscriptionFrame objects in the `user_id` attribute.

If `speaker_active_format` or `speaker_passive_format` are provided, then the text output for the TranscriptionFrame will be formatted to this specification. Your system context can then be updated to include information about this format to understand which speaker spoke which words. The passive format is optional and is used when the engine has been told to focus on specific speakers and other speakers will then be formatted using the `speaker_passive_format` format.

* `speaker_active_format` -> the formatter for active speakers
* `speaker_passive_format` -> the formatter for passive / background speakers

Examples:

* `<{speaker_id}>{text}</{speaker_id}>` -> `<S1>Good morning.</S1>`.
* `@{speaker_id}: {text}` -> `@S1: Good morning.`.

### Available attributes

| Attribute          | Description                                 | Example                         |
| ------------------ | ------------------------------------------- | ------------------------------- |
| `speaker_id`       | The label of the speaker                    | `S1`                            |
| `text` / `content` | The transcribed text                        | `Good morning.`                 |
| `ts`               | The timestamp of the transcription          | `2025-09-15T19:47:29.096+00:00` |
| `start_time`       | The start time of the transcription segment | `0.0`                           |
| `end_time`         | The end time of the transcription segment   | `2.5`                           |
| `lang`             | The language of the transcription segment   | `en`                            |

## Speaker Lock

In conjunction with speaker diarization, it is possible to decide at the start or during a conversation to focus on a specific speaker, ignore or retain words from other speakers, or implicitly ignore one or more speakers altogether.

In the example below, the following will happen:

* `S1` will be transcribed as normal and drive the end of turn and the conversation flow
* `S2` will be ignored completely
* All other speakers' words will be transcribed and emitted as tagged segments, but ONLY when a speaker in focus also speaks

What this means is that if `S3` says "Hello", then it is not until `S1` speaks again that the transcription will be emitted.

```python  theme={null}
stt = SpeechmaticsSTTService(
    api_key=os.getenv("SPEECHMATICS_API_KEY"),
    params=SpeechmaticsSTTService.InputParams(
        language=Language.EN,
        focus_speakers=["S1"],
        ignore_speakers=["S2"],
        focus_mode=SpeechmaticsSTTService.SpeakerFocusMode.RETAIN,
        speaker_active_format="<{speaker_id}>{text}</{speaker_id}>",
    ),
)
```

## Language Support

<Note>
  Refer to the [Speechmatics
  docs](https://docs.speechmatics.com/introduction/supported-languages) for more
  information on supported languages.
</Note>

Speechmatics STT supports the following languages and regional variants.

Setting a language can be done using the `language` parameter when creating the STT object. The exception to this is English / Mandarin which has the code `cmn_en`.

| Language Code  | Description | Locales                   |
| -------------- | ----------- | ------------------------- |
| `Language.AR`  | Arabic      | -                         |
| `Language.BA`  | Bashkir     | -                         |
| `Language.EU`  | Basque      | -                         |
| `Language.BE`  | Belarusian  | -                         |
| `Language.BG`  | Bulgarian   | -                         |
| `Language.BN`  | Bengali     | -                         |
| `Language.YUE` | Cantonese   | -                         |
| `Language.CA`  | Catalan     | -                         |
| `Language.HR`  | Croatian    | -                         |
| `Language.CS`  | Czech       | -                         |
| `Language.DA`  | Danish      | -                         |
| `Language.NL`  | Dutch       | -                         |
| `Language.EN`  | English     | `en-US`, `en-GB`, `en-AU` |
| `Language.EO`  | Esperanto   | -                         |
| `Language.ET`  | Estonian    | -                         |
| `Language.FA`  | Persian     | -                         |
| `Language.FI`  | Finnish     | -                         |
| `Language.FR`  | French      | -                         |
| `Language.GL`  | Galician    | -                         |
| `Language.DE`  | German      | -                         |
| `Language.EL`  | Greek       | -                         |
| `Language.HE`  | Hebrew      | -                         |
| `Language.HI`  | Hindi       | -                         |
| `Language.HU`  | Hungarian   | -                         |
| `Language.IA`  | Interlingua | -                         |
| `Language.IT`  | Italian     | -                         |
| `Language.ID`  | Indonesian  | -                         |
| `Language.GA`  | Irish       | -                         |
| `Language.JA`  | Japanese    | -                         |
| `Language.KO`  | Korean      | -                         |
| `Language.LV`  | Latvian     | -                         |
| `Language.LT`  | Lithuanian  | -                         |
| `Language.MS`  | Malay       | -                         |
| `Language.MT`  | Maltese     | -                         |
| `Language.CMN` | Mandarin    | `cmn-Hans`, `cmn-Hant`    |
| `Language.MR`  | Marathi     | -                         |
| `Language.MN`  | Mongolian   | -                         |
| `Language.NO`  | Norwegian   | -                         |
| `Language.PL`  | Polish      | -                         |
| `Language.PT`  | Portuguese  | -                         |
| `Language.RO`  | Romanian    | -                         |
| `Language.RU`  | Russian     | -                         |
| `Language.SK`  | Slovakian   | -                         |
| `Language.SL`  | Slovenian   | -                         |
| `Language.ES`  | Spanish     | -                         |
| `Language.SV`  | Swedish     | -                         |
| `Language.SW`  | Swahili     | -                         |
| `Language.TA`  | Tamil       | -                         |
| `Language.TH`  | Thai        | -                         |
| `Language.TR`  | Turkish     | -                         |
| `Language.UG`  | Uyghur      | -                         |
| `Language.UK`  | Ukrainian   | -                         |
| `Language.UR`  | Urdu        | -                         |
| `Language.VI`  | Vietnamese  | -                         |
| `Language.CY`  | Welsh       | -                         |

For bilingual transcription, use the `language` and `domain` parameters as follows:

| Language Code | Description        | Domain Options |
| ------------- | ------------------ | -------------- |
| `cmn_en`      | English / Mandarin | -              |
| `en_ms`       | English / Malay    | -              |
| `Language.ES` | English / Spanish  | `bilingual-en` |
| `en_ta`       | English / Tamil    | -              |

## Usage Examples

Examples are included in the Pipecat project:

* Using Speechmatics STT service -> [07a-interruptible-speechmatics.py](https://github.com/pipecat-ai/pipecat/blob/main/examples/foundational/07a-interruptible-speechmatics.py)
* Using Speechmatics STT service with VAD -> [07a-interruptible-speechmatics-vad.py](https://github.com/pipecat-ai/pipecat/blob/main/examples/foundational/07a-interruptible-speechmatics-vad.py)
* Transcribing with Speechmatics STT -> [13h-speechmatics-transcription.py](https://github.com/pipecat-ai/pipecat/blob/main/examples/foundational/13h-speechmatics-transcription.py)

Sample projects:

* Guess Who -> [Guess Who](https://github.com/sam-s10s/pipecat-guess-who)
* Guess Who Board Game -> [Guess Who](https://github.com/sam-s10s/pipecat-guess-who-irl)

### Basic Configuration

Initialize the `SpeechmaticsSTTService` and use it in a pipeline:

```python  theme={null}
from pipecat.services.speechmatics.stt import SpeechmaticsSTTService
from pipecat.transcriptions.language import Language

# Configure service
stt = SpeechmaticsSTTService(
    api_key="your-api-key",
    params=SpeechmaticsSTTService.InputParams(
        language=Language.FR,
    )
)

# Use in pipeline
pipeline = Pipeline([
    transport.input(),
    stt,
    context_aggregator.user(),
    llm,
    tts,
    transport.output(),
    context_aggregator.assistant()
])
```

### With Diarization

This will enable diarization and also only go to the LLM if words are spoken from the first speaker (`S1`). Words from other speakers are transcribed but only sent when the first speaker speaks. When using the `TurnDetectionMode.ADAPTIVE` or `TurnDetectionMode.SMART_TURN` options, this will use the speaker diarization to determine when a speaker is speaking. You will need to disable VAD options within the selected transport object to ensure this works correctly (see [07b-interruptible-speechmatics-vad.py](https://github.com/pipecat-ai/pipecat/blob/main/examples/foundational/07b-interruptible-speechmatics-vad.py) as an example).

Initialize the `SpeechmaticsSTTService` and use it in a pipeline:

```python  theme={null}
from pipecat.services.speechmatics.stt import SpeechmaticsSTTService
from pipecat.transcriptions.language import Language

# Configure service
stt = SpeechmaticsSTTService(
    api_key="your-api-key",
    params=SpeechmaticsSTTService.InputParams(
        language=Language.EN,
        turn_detection_mode=SpeechmaticsSTTService.TurnDetectionMode.ADAPTIVE,
        focus_speakers=["S1"],
        speaker_active_format="<{speaker_id}>{text}</{speaker_id}>",
        speaker_passive_format="<PASSIVE><{speaker_id}>{text}</{speaker_id}></PASSIVE>",
    )
)

# Use in pipeline
pipeline = Pipeline([
    transport.input(),
    stt,
    context_aggregator.user(),
    llm,
    tts,
    transport.output(),
    context_aggregator.assistant()
])
```

## Additional Notes

* **Connection Management**: Automatically handles WebSocket connections and reconnections
* **Sample Rate**: The default sample rate of `16000` in `pcm_s16le` format
* **VAD Integration**: Optionally supports Speechmatics' built-in VAD and end of utterance detection

---

# Source: https://docs.pipecat.ai/deployment/pipecat-cloud/rest-reference/endpoint/start.md

> ## Documentation Index
> Fetch the complete documentation index at: https://docs.pipecat.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Start an Agent session

> Start a new session with a deployed agent.

This endpoint starts a new instance of a deployed agent. You can optionally create a Daily room for the service to connect to.


## OpenAPI

````yaml POST /{agentName}/start
openapi: 3.0.0
info:
  title: Pipecat Cloud
  version: 1.0.0
  description: Public API for Pipecat Cloud services
servers:
  - url: https://api.pipecat.daily.co/v1/public
    description: Public API server
security:
  - PublicKeyAuth: []
paths:
  /{agentName}/start:
    post:
      summary: Start a service instance
      operationId: startService
      parameters:
        - name: agentName
          in: path
          required: true
          description: Name of the agent to start
          schema:
            type: string
      requestBody:
        required: true
        content:
          application/json:
            schema:
              $ref: '#/components/schemas/StartRequest'
      responses:
        '200':
          description: Service started successfully
          content:
            application/json:
              schema:
                $ref: '#/components/schemas/StartResponse'
        '400':
          description: Invalid request
          content:
            application/json:
              schema:
                $ref: '#/components/schemas/ErrorResponse'
      security:
        - PublicKeyAuth: []
components:
  schemas:
    StartRequest:
      type: object
      properties:
        createDailyRoom:
          type: boolean
          default: false
          description: Whether to create a Daily room for the service instance
          example: true
        dailyRoomProperties:
          type: object
          description: >-
            Optional configuration for the Daily room. Only used when
            createDailyRoom is true.


            See [Daily API
            documentation](https://docs.daily.co/reference/rest-api/rooms/config)
            for supported properties.
          example:
            enable_recording: cloud
        dailyMeetingTokenProperties:
          type: object
          description: >-
            Optional configuration for the Daily meeting token. Only used when
            createDailyRoom is true.


            See [Daily API
            documentation](https://docs.daily.co/reference/rest-api/meeting-tokens/config)
            for supported properties.
          example:
            is_owner: true
            enable_auto_recording: true
        enableDefaultIceServers:
          type: boolean
          default: false
          description: >-
            Whether to use the default STUN/TURN ICE servers provided by Pipecat
            Cloud.
          example: true
        transport:
          type: string
          description: Explicitly specify which transport the bot should use.
          enum:
            - daily
            - webrtc
          example: webrtc
        body:
          type: object
          description: >-
            Arbitrary user data / configuration object to pass to the service
            instance. Accessible as the first parameter of the `bot` method or
            custom entry point.


            See the [`Starting Sessions`
            docs](../../fundamentals/active-sessions#passing-data) for more
            information.
          example:
            foo: bar
    StartResponse:
      type: object
      properties:
        dailyRoom:
          type: string
          description: URL of the created Daily room (if createDailyRoom was set to true)
          example: >-
            https://cloud-354b1860310a444ba0f528b6a2f467c4.daily.co/qZbK9nyXCDCmQ0zDDZta
        dailyToken:
          type: string
          description: Token to access the Daily room (if createDailyRoom was set to true)
          example: eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9...
        sessionId:
          type: string
          description: Session ID for the created session
          example: 639f91d8-d511-4677-a83b-bd7564d5d92f
        iceConfig:
          type: object
          description: >-
            ICE server configuration (only included when enableDefaultIceServers
            is true)
          properties:
            iceServers:
              type: array
              items:
                type: object
                properties:
                  urls:
                    type: array
                    items:
                      type: string
                    description: Array of STUN/TURN server URLs
                  username:
                    type: string
                    description: Authentication username for TURN servers
                  credential:
                    type: string
                    description: Authentication credential for TURN servers
          example:
            iceServers:
              - urls:
                  - stun:stun.cloudflare.com:3478
                  - turn:turn.cloudflare.com:3478?transport=udp
                  - turn:turn.cloudflare.com:3478?transport=tcp
                  - turns:turn.cloudflare.com:5349?transport=tcp
                username: g061f2543c0c7aab7d4b087ad407709...
                credential: e6330e0a539d4cfe9cf9f01b9eb2f...
    ErrorResponse:
      type: object
      properties:
        error:
          type: string
          description: Error message
        code:
          type: string
          description: Error code
  securitySchemes:
    PublicKeyAuth:
      type: http
      scheme: bearer
      description: Authentication using a Pipecat Cloud API token.

````

---

# Source: https://docs.pipecat.ai/deployment/pipecat-cloud/rest-reference/endpoint/stop.md

> ## Documentation Index
> Fetch the complete documentation index at: https://docs.pipecat.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Stop an Agent session

> Stop a running agent session and clean up its resources.



## OpenAPI

````yaml DELETE /agents/{agentName}/sessions/{sessionId}
openapi: 3.0.0
info:
  title: Pipecat Cloud - Stop Agent Session
  version: 1.0.0
  description: Stop an agent session via Pipecat Cloud Private API
servers:
  - url: https://api.pipecat.daily.co/v1
    description: API server
security:
  - PrivateKeyAuth: []
paths:
  /agents/{agentName}/sessions/{sessionId}:
    delete:
      summary: Stop an agent session
      description: Stops an active session of a deployed agent and cleans up its resources.
      operationId: stopSession
      parameters:
        - name: agentName
          in: path
          required: true
          description: Name of the agent
          schema:
            type: string
        - name: sessionId
          in: path
          required: true
          description: UUID of the session to stop
          schema:
            type: string
            format: uuid
      responses:
        '200':
          description: Session stopped successfully
          content:
            application/json:
              schema:
                $ref: '#/components/schemas/StopResponse'
        '400':
          description: Invalid request (e.g., invalid session ID format)
          content:
            application/json:
              schema:
                $ref: '#/components/schemas/ErrorResponse'
        '404':
          description: Service or session not found
          content:
            application/json:
              schema:
                $ref: '#/components/schemas/ErrorResponse'
      security:
        - PrivateKeyAuth: []
components:
  schemas:
    StopResponse:
      type: object
      properties:
        status:
          type: string
          description: Status of the stop operation
          example: terminated
        session_id:
          type: string
          format: uuid
          description: Session ID of the stopped session
          example: 639f91d8-d511-4677-a83b-bd7564d5d92f
    ErrorResponse:
      type: object
      properties:
        error:
          type: string
          description: Error message
        code:
          type: string
          description: Error code
  securitySchemes:
    PrivateKeyAuth:
      type: http
      scheme: bearer
      description: >-
        Authentication requires a Pipecat Cloud Private API token.


        Generate a Private API key from your Dashboard (Settings > API Keys >
        Private > Create key) and include it as a Bearer token in the
        Authorization header.

````

---

# Source: https://docs.pipecat.ai/server/utilities/filters/stt-mute.md

> ## Documentation Index
> Fetch the complete documentation index at: https://docs.pipecat.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# STTMuteFilter

> Processor for controlling STT muting and interruption handling during bot speech and function calls

<Warning>
  DEPRECATED: STTMuteFilter has been deprecated in favor of [User Mute
  Strategies](/server/utilities/turn-management/user-mute-strategies). Configure
  `user_mute_strategies` on the `LLMUserAggregator` instead.
</Warning>

## Overview

`STTMuteFilter` is a general-purpose processor that combines STT muting and interruption control. When active, it prevents both transcription and interruptions during specified conditions (e.g., bot speech, function calls), providing a cleaner conversation flow.

The processor supports multiple simultaneous strategies for when to mute the STT service, making it flexible for different use cases.

<Tip>
  Want to try it out? Check out the [STTMuteFilter foundational
  demo](https://github.com/pipecat-ai/pipecat/blob/main/examples/foundational/24-stt-mute-filter.py)
</Tip>

## Constructor Parameters

<ParamField path="config" type="STTMuteConfig" required>
  Configuration object that defines the muting strategies and optional custom
  logic
</ParamField>

<ParamField path="stt_service" type="Optional[STTService]" required>
  The STT service to control (deprecated, will be removed in a future version)
</ParamField>

## Configuration

The processor is configured using `STTMuteConfig`, which determines when and how the STT service should be muted:

<ParamField path="strategies" type="set[STTMuteStrategy]">
  Set of muting strategies to apply
</ParamField>

<ParamField path="should_mute_callback" type="Callable[[STTMuteFilter], Awaitable[bool]]" default="None">
  Optional callback for custom muting logic (required when strategy is `CUSTOM`)
</ParamField>

### Muting Strategies

`STTMuteConfig` accepts a set of these `STTMuteStrategy` values:

<ParamField path="FIRST_SPEECH" type="STTMuteStrategy">
  Mute only during the bot's first speech (typically during introduction)
</ParamField>

<ParamField path="MUTE_UNTIL_FIRST_BOT_COMPLETE" type="STTMuteStrategy">
  Start muted and remain muted until first bot speech completes. Useful when bot
  speaks first and you want to ensure its first response cannot be interrupted.
</ParamField>

<ParamField path="FUNCTION_CALL" type="STTMuteStrategy">
  Mute during LLM function calls (e.g., API requests, external service calls)
</ParamField>

<ParamField path="ALWAYS" type="STTMuteStrategy">
  Mute during all bot speech
</ParamField>

<ParamField path="CUSTOM" type="STTMuteStrategy">
  Use custom logic provided via callback to determine when to mute. The callback
  is invoked when the bot is speaking and can use application state to decide
  whether to mute. When the bot stops speaking, unmuting occurs automatically if
  no other strategy requires muting.
</ParamField>

<Note>
  `MUTE_UNTIL_FIRST_BOT_COMPLETE` and `FIRST_SPEECH` strategies should not be
  used together as they handle the first bot speech differently.
</Note>

## Input Frames

<ParamField path="BotStartedSpeakingFrame" type="Frame">
  Indicates bot has started speaking
</ParamField>

<ParamField path="BotStoppedSpeakingFrame" type="Frame">
  Indicates bot has stopped speaking
</ParamField>

<ParamField path="FunctionCallInProgressFrame" type="Frame">
  Indicates a function call has started
</ParamField>

<ParamField path="FunctionCallResultFrame" type="Frame">
  Indicates a function call has completed
</ParamField>

<ParamField path="InterimTranscriptionFrame" type="Frame">
  Indicates an interim transcription result (suppressed when muted)
</ParamField>

<ParamField path="StartInterruptionFrame" type="Frame">
  User interruption start event (suppressed when muted)
</ParamField>

<ParamField path="StopInterruptionFrame" type="Frame">
  User interruption stop event (suppressed when muted)
</ParamField>

<ParamField path="TranscriptionFrame" type="Frame">
  Indicates a transcription result (suppressed when muted)
</ParamField>

<ParamField path="UserStartedSpeakingFrame" type="Frame">
  Indicates user has started speaking (suppressed when muted)
</ParamField>

<ParamField path="UserStoppedSpeakingFrame" type="Frame">
  Indicates user has stopped speaking (suppressed when muted)
</ParamField>

## Output Frames

<ParamField path="STTMuteFrame" type="Frame">
  Control frame to mute/unmute the STT service
</ParamField>

All input frames are passed through except VAD-related frames (interruptions and user speaking events) when muted.

## Usage Examples

### Basic Usage (Mute During Bot's First Speech)

```python  theme={null}
stt = DeepgramSTTService(api_key=os.getenv("DEEPGRAM_API_KEY"))
stt_mute_filter = STTMuteFilter(
    config=STTMuteConfig(strategies={
        STTMuteStrategy.FIRST_SPEECH
    })
)

pipeline = Pipeline([
    transport.input(),
    stt,
    stt_mute_filter,  # Between the STT service and context aggregator
    context_aggregator.user(),
    # ... rest of pipeline
])
```

### Mute Until First Bot Response Completes

```python  theme={null}
stt_mute_filter = STTMuteFilter(
    config=STTMuteConfig(strategies={STTMuteStrategy.MUTE_UNTIL_FIRST_BOT_COMPLETE})
)
```

This ensures no user speech is processed until after the bot's first complete response.

### Always Mute During Bot Speech

```python  theme={null}
stt_mute_filter = STTMuteFilter(
    config=STTMuteConfig(strategies={STTMuteStrategy.ALWAYS})
)
```

### Custom Muting Logic

The `CUSTOM` strategy allows you to control muting based on application state when the bot is speaking. The callback will be invoked whenever the bot is speaking, and your logic decides whether to mute:

```python  theme={null}
# Create a state manager
class SessionState:
    def __init__(self):
        self.session_ending = False

session_state = SessionState()

# Callback function that determines whether to mute
async def session_state_mute_logic(stt_filter: STTMuteFilter) -> bool:
    # Return True to mute, False otherwise
    # This is called when the bot is speaking
    return session_state.session_ending

# Configure filter with CUSTOM strategy
stt_mute_filter = STTMuteFilter(
    config=STTMuteConfig(
        strategies={STTMuteStrategy.CUSTOM},
        should_mute_callback=session_state_mute_logic
    )
)

# Later, when you want to trigger muting (e.g., during session timeout):
async def handle_session_timeout():
    # Update state that will be checked by the callback
    session_state.session_ending = True

    # Send goodbye message
    goodbye_message = "Thank you for using our service. This session is now ending."
    await pipeline.push_frame(TTSSpeakFrame(text=goodbye_message))

    # The system will automatically mute during this message because:
    # 1. Bot starts speaking, triggering the callback
    # 2. Callback returns True (session_ending is True)
    # 3. When bot stops speaking, unmuting happens automatically
```

### Combining Multiple Strategies

```python  theme={null}
async def custom_mute_logic(processor: STTMuteFilter) -> bool:
    # Example: Mute during business hours only
    current_hour = datetime.now().hour
    return 9 <= current_hour < 17

stt_mute_filter = STTMuteFilter(
    config=STTMuteConfig(
        strategies={
            STTMuteStrategy.FUNCTION_CALL,       # Mute during function calls
            STTMuteStrategy.CUSTOM,              # And during business hours
            STTMuteStrategy.MUTE_UNTIL_FIRST_BOT_COMPLETE  # And until first bot speech completes
        },
        should_mute_callback=custom_mute_logic
    )
)
```

### Frame Flow

```mermaid  theme={null}
graph TD
    A[Transport Input] --> B[STTMuteFilter]
    B --> C[STT Service]
    B -- "Suppressed when muted" --> D[VAD-related Frames]
    B -- "STTMuteFrame" --> C
```

## Notes

* Combines STT muting and interruption control into a single concept
* Muting prevents both transcription and interruptions
* Multiple strategies can be active simultaneously
* CUSTOM strategy callback is only invoked when the bot is speaking
* Unmuting happens automatically when bot speech ends (if no other strategy requires muting)
* Placed between the STT service and context aggregator in pipeline
* Maintains conversation flow during bot speech and function calls
* Efficient state tracking for minimal overhead

---

# Source: https://docs.pipecat.ai/server/services/supported-services.md

> ## Documentation Index
> Fetch the complete documentation index at: https://docs.pipecat.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Supported Services

> AI services integrated with Pipecat and their setup requirements

## Transports

Transports exchange audio and video streams between the user and bot.

| Service                                                                   | Setup                                 |
| ------------------------------------------------------------------------- | ------------------------------------- |
| [DailyTransport](/server/services/transport/daily)                        | `pip install "pipecat-ai[daily]"`     |
| [FastAPIWebSocketTransport](/server/services/transport/fastapi-websocket) | `pip install "pipecat-ai[websocket]"` |
| [HeyGenTransport](/server/services/transport/heygen)                      | `pip install "pipecat-ai[heygen]"`    |
| [LiveKitTransport](/server/services/transport/livekit)                    | `pip install "pipecat-ai[livekit]"`   |
| [SmallWebRTCTransport](/server/services/transport/small-webrtc)           | `pip install "pipecat-ai[webrtc]"`    |
| [TavusTransport](/server/services/transport/tavus)                        | `pip install "pipecat-ai[tavus]"`     |
| [WebSocket Transports](/server/services/transport/websocket-server)       | `pip install "pipecat-ai[websocket]"` |
| [WhatsAppTransport](/server/services/transport/whatsapp)                  | `pip install "pipecat-ai[webrtc]"`    |

## Serializers

Serializers convert between frames and media streams, enabling real-time communication over a websocket.

| Service                                       | Setup                    |
| --------------------------------------------- | ------------------------ |
| [Exotel](/server/services/serializers/exotel) | No dependencies required |
| [Plivo](/server/services/serializers/plivo)   | No dependencies required |
| [Telnyx](/server/services/serializers/telnyx) | No dependencies required |
| [Twilio](/server/services/serializers/twilio) | No dependencies required |
| [Vonage](/server/services/serializers/vonage) | No dependencies required |

## Speech-to-Text

Speech-to-Text services receive and audio input and output transcriptions.

| Service                                               | Setup                                    |
| ----------------------------------------------------- | ---------------------------------------- |
| [AssemblyAI](/server/services/stt/assemblyai)         | `pip install "pipecat-ai[assemblyai]"`   |
| [AWS Transcribe](/server/services/stt/aws)            | `pip install "pipecat-ai[aws]"`          |
| [Azure](/server/services/stt/azure)                   | `pip install "pipecat-ai[azure]"`        |
| [Cartesia](/server/services/stt/cartesia)             | `pip install "pipecat-ai[cartesia]"`     |
| [Deepgram](/server/services/stt/deepgram)             | `pip install "pipecat-ai[deepgram]"`     |
| [ElevenLabs](/server/services/stt/elevenlabs)         | `pip install "pipecat-ai[elevenlabs]"`   |
| [Fal Wizper](/server/services/stt/fal)                | `pip install "pipecat-ai[fal]"`          |
| [Gladia](/server/services/stt/gladia)                 | `pip install "pipecat-ai[gladia]"`       |
| [Google](/server/services/stt/google)                 | `pip install "pipecat-ai[google]"`       |
| [Gradium](/server/services/stt/gradium)               | `pip install "pipecat-ai[gradium]"`      |
| [Groq (Whisper)](/server/services/stt/groq)           | `pip install "pipecat-ai[groq]"`         |
| [Hathora](/server/services/stt/hathora)               | `pip install "pipecat-ai[hathora]"`      |
| [NVIDIA Riva](/server/services/stt/riva)              | `pip install "pipecat-ai[nvidia]"`       |
| [OpenAI (Whisper)](/server/services/stt/openai)       | `pip install "pipecat-ai[openai]"`       |
| [SambaNova (Whisper)](/server/services/stt/sambanova) | `pip install "pipecat-ai[sambanova]"`    |
| [Sarvam](/server/services/stt/sarvam)                 | `pip install "pipecat-ai[sarvam]"`       |
| [Soniox](/server/services/stt/soniox)                 | `pip install "pipecat-ai[soniox]"`       |
| [Speechmatics](/server/services/stt/speechmatics)     | `pip install "pipecat-ai[speechmatics]"` |
| [Whisper](/server/services/stt/whisper)               | `pip install "pipecat-ai[whisper]"`      |

## Large Language Models

LLMs receive text or audio based input and output a streaming text response.

| Service                                                | Setup                                  |
| ------------------------------------------------------ | -------------------------------------- |
| [Anthropic](/server/services/llm/anthropic)            | `pip install "pipecat-ai[anthropic]"`  |
| [AWS Bedrock](/server/services/llm/aws)                | `pip install "pipecat-ai[aws]"`        |
| [Azure](/server/services/llm/azure)                    | `pip install "pipecat-ai[azure]"`      |
| [Cerebras](/server/services/llm/cerebras)              | `pip install "pipecat-ai[cerebras]"`   |
| [DeepSeek](/server/services/llm/deepseek)              | `pip install "pipecat-ai[deepseek]"`   |
| [Fireworks AI](/server/services/llm/fireworks)         | `pip install "pipecat-ai[fireworks]"`  |
| [Google Gemini](/server/services/llm/gemini)           | `pip install "pipecat-ai[google]"`     |
| [Google Vertex AI](/server/services/llm/google-vertex) | `pip install "pipecat-ai[google]"`     |
| [Grok](/server/services/llm/grok)                      | `pip install "pipecat-ai[grok]"`       |
| [Groq](/server/services/llm/groq)                      | `pip install "pipecat-ai[groq]"`       |
| [NVIDIA NIM](/server/services/llm/nim)                 | `pip install "pipecat-ai[nvidia]"`     |
| [Ollama](/server/services/llm/ollama)                  | `pip install "pipecat-ai[ollama]"`     |
| [OpenAI](/server/services/llm/openai)                  | `pip install "pipecat-ai[openai]"`     |
| [OpenPipe](/server/services/llm/openpipe)              | `pip install "pipecat-ai[openpipe]"`   |
| [OpenRouter](/server/services/llm/openrouter)          | `pip install "pipecat-ai[openrouter]"` |
| [Perplexity](/server/services/llm/perplexity)          | `pip install "pipecat-ai[perplexity]"` |
| [Qwen](/server/services/llm/qwen)                      | `pip install "pipecat-ai[qwen]"`       |
| [SambaNova](/server/services/llm/sambanova)            | `pip install "pipecat-ai[sambanova]"`  |
| [Together AI](/server/services/llm/together)           | `pip install "pipecat-ai[together]"`   |

## Text-to-Speech

Text-to-Speech services receive text input and output audio streams or chunks.

| Service                                           | Setup                                    |
| ------------------------------------------------- | ---------------------------------------- |
| [Async](/server/services/tts/asyncai)             | `pip install "pipecat-ai[asyncai]"`      |
| [AWS Polly](/server/services/tts/aws)             | `pip install "pipecat-ai[aws]"`          |
| [Azure](/server/services/tts/azure)               | `pip install "pipecat-ai[azure]"`        |
| [Camb AI](/server/services/tts/camb)              | `pip install "pipecat-ai[camb]"`         |
| [Cartesia](/server/services/tts/cartesia)         | `pip install "pipecat-ai[cartesia]"`     |
| [Deepgram](/server/services/tts/deepgram)         | `pip install "pipecat-ai[deepgram]"`     |
| [ElevenLabs](/server/services/tts/elevenlabs)     | `pip install "pipecat-ai[elevenlabs]"`   |
| [Fish](/server/services/tts/fish)                 | `pip install "pipecat-ai[fish]"`         |
| [Google](/server/services/tts/google)             | `pip install "pipecat-ai[google]"`       |
| [Gradium](/server/services/tts/gradium)           | `pip install "pipecat-ai[gradium]"`      |
| [Groq](/server/services/tts/groq)                 | `pip install "pipecat-ai[groq]"`         |
| [Hathora](/server/services/tts/hathora)           | `pip install "pipecat-ai[hathora]"`      |
| [Hume](/server/services/tts/hume)                 | `pip install "pipecat-ai[hume]"`         |
| [Inworld](/server/services/tts/inworld)           | No dependencies required                 |
| [LMNT](/server/services/tts/lmnt)                 | `pip install "pipecat-ai[lmnt]"`         |
| [MiniMax](/server/services/tts/minimax)           | No dependencies required                 |
| [Neuphonic](/server/services/tts/neuphonic)       | `pip install "pipecat-ai[neuphonic]"`    |
| [NVIDIA Riva](/server/services/tts/riva)          | `pip install "pipecat-ai[nvidia]"`       |
| [OpenAI](/server/services/tts/openai)             | `pip install "pipecat-ai[openai]"`       |
| [Piper](/server/services/tts/piper)               | No dependencies required                 |
| [PlayHT](/server/services/tts/playht)             | `pip install "pipecat-ai[playht]"`       |
| [Resemble](/server/services/tts/resemble)         | `pip install "pipecat-ai[resemble]"`     |
| [Rime](/server/services/tts/rime)                 | `pip install "pipecat-ai[rime]"`         |
| [Sarvam](/server/services/tts/sarvam)             | No dependencies required                 |
| [Speechmatics](/server/services/tts/speechmatics) | `pip install "pipecat-ai[speechmatics]"` |
| [XTTS](/server/services/tts/xtts)                 | `pip install "pipecat-ai[xtts]"`         |

## Speech-to-Speech

Speech-to-Speech services are multi-modal LLM services that take in audio, video, or text and output audio or text.

| Service                                                          | Setup                                      |
| ---------------------------------------------------------------- | ------------------------------------------ |
| [AWS Nova Sonic](/server/services/s2s/aws)                       | `pip install "pipecat-ai[aws-nova-sonic]"` |
| [Gemini Multimodal Live](/server/services/s2s/gemini-live)       | `pip install "pipecat-ai[google]"`         |
| [Gemini Live Vertex AI](/server/services/s2s/gemini-live-vertex) | `pip install "pipecat-ai[google]"`         |
| [Grok Voice Agent](/server/services/s2s/grok)                    | `pip install "pipecat-ai[grok]"`           |
| [OpenAI Realtime](/server/services/s2s/openai)                   | `pip install "pipecat-ai[openai]"`         |
| [Ultravox](/server/services/s2s/ultravox)                        | `pip install "pipecat-ai[ultravox]"`       |

## Image Generation

Image generation services receive text inputs and output images.

| Service                                                          | Setup                              |
| ---------------------------------------------------------------- | ---------------------------------- |
| [fal](/server/services/image-generation/fal)                     | `pip install "pipecat-ai[fal]"`    |
| [Google Imagen](/server/services/image-generation/google-imagen) | `pip install "pipecat-ai[google]"` |
| [OpenAI](/server/services/image-generation/openai)               | `pip install "pipecat-ai[openai]"` |

## Video

Video services enable you to build an avatar where audio and video are synchronized.

| Service                                 | Setup                              |
| --------------------------------------- | ---------------------------------- |
| [HeyGen](/server/services/video/heygen) | `pip install "pipecat-ai[heygen]"` |
| [Simli](/server/services/video/simli)   | `pip install "pipecat-ai[simli]"`  |
| [Tavus](/server/services/video/tavus)   | `pip install "pipecat-ai[tavus]"`  |

## Memory

Memory services can be used to store and retrieve conversations.

| Service                              | Setup                            |
| ------------------------------------ | -------------------------------- |
| [mem0](/server/services/memory/mem0) | `pip install "pipecat-ai[mem0]"` |

## Vision

Vision services receive a streaming video input and output text describing the video input.

| Service                                        | Setup                                 |
| ---------------------------------------------- | ------------------------------------- |
| [Moondream](/server/services/vision/moondream) | `pip install "pipecat-ai[moondream]"` |

## Analytics & Monitoring

Analytics services help you better understand how your service operates.

| Service                                     | Setup                              |
| ------------------------------------------- | ---------------------------------- |
| [Sentry](/server/services/analytics/sentry) | `pip install "pipecat-ai[sentry]"` |

---

# Source: https://docs.pipecat.ai/cli/tail.md

> ## Documentation Index
> Fetch the complete documentation index at: https://docs.pipecat.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# tail

> A terminal dashboard for monitoring Pipecat sessions in real-time

**Tail** is a terminal dashboard for monitoring your Pipecat sessions in real-time with logs, conversations, metrics, and audio levels all in one place.

With Tail you can:

* 📜 Follow system logs in real time
* 💬 Track conversations as they happen
* 🔊 Monitor user and agent audio levels
* 📈 Keep an eye on service metrics and usage
* 🖥️ Run locally as a pipeline runner or connect to a remote session

**Usage:**

```shell  theme={null}
pipecat tail [OPTIONS]
```

**Options:**

<ParamField path="--url / -u" type="string">
  WebSocket URL to connect to. Defaults to `ws://localhost:9292`.
</ParamField>

## How to Use Tail

* Add `pipecat-ai-cli` to your project's dependencies.

* Update your Pipecat code to include the `TailObserver`:

  ```python  theme={null}
  from pipecat_cli.tail import TailObserver

  task = PipelineTask(
      pipeline,
      observers=[TailObserver()]
  )
  ```

* Start the Tail app separately:

  ```bash  theme={null}
  # Connect to local session (default)
  pipecat tail

  # Connect to remote session
  pipecat tail --url wss://my-bot.example.com
  ```

---

# Source: https://docs.pipecat.ai/server/services/video/tavus.md

# Source: https://docs.pipecat.ai/server/services/transport/tavus.md

> ## Documentation Index
> Fetch the complete documentation index at: https://docs.pipecat.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# TavusTransport

> Conversational AI applications with Tavus avatars using real-time audio/video streaming

## Overview

TavusTransport enables your Pipecat bot to join the same virtual room as a Tavus avatar and human participants. The transport integrates with the Tavus platform to create conversational AI applications where a Tavus avatar provides synchronized video and audio output while your bot handles the conversation logic.

When used, the Pipecat bot connects to a Daily room alongside the Tavus avatar and user. The bot receives audio input from participants, processes it through your pipeline, and sends TTS audio to the Tavus avatar for synchronized video rendering.

<CardGroup cols={2}>
  <Card title="Tavus Transport API Reference" icon="code" href="https://reference-server.pipecat.ai/en/latest/api/pipecat.transports.tavus.transport.html">
    Pipecat's API methods for Tavus platform integration
  </Card>

  <Card title="Example Implementation" icon="play" href="https://github.com/pipecat-ai/pipecat/blob/main/examples/foundational/21a-tavus-video-service.py">
    Complete Tavus avatar conversation example
  </Card>

  <Card title="Tavus API Documentation" icon="book" href="https://docs.tavus.io/api-reference/phoenix-replica-model/create-replica">
    Tavus official API reference and replica creation
  </Card>

  <Card title="Client SDK Integration" icon="link" href="/client/js/transports/daily">
    Connect using Pipecat Client SDK with Daily transport
  </Card>
</CardGroup>

## Installation

To use TavusTransport, install the required dependencies:

```bash  theme={null}
pip install "pipecat-ai[tavus]"
```

## Prerequisites

### Tavus Platform Setup

Before using TavusTransport, you need:

1. **Tavus Account**: Sign up at [Tavus.io](https://tavus.io)
2. **API Key**: Generate a Tavus API key for authentication
3. **Replica ID**: Create or obtain a Tavus replica model ID
4. **Persona Configuration**: Set up a persona (optional, defaults to Pipecat TTS voice)

<Tip>
  Use `persona_id="pipecat-stream"` to have Tavus use your Pipecat bot's TTS
  voice instead of a Tavus persona voice.
</Tip>

### Required Environment Variables

* `TAVUS_API_KEY`: Your Tavus API key for authentication
* `TAVUS_REPLICA_ID`: ID of the Tavus replica model to use

## Key Features

* **Avatar Integration**: Seamlessly integrates Tavus avatars with Pipecat conversations
* **Synchronized Audio/Video**: Tavus avatar renders video synchronized with bot's TTS output
* **Multi-participant Rooms**: Supports bot, avatar, and human participants in the same session
* **Conversation Management**: Handles Tavus conversation lifecycle through API
* **Real-time Streaming**: Bidirectional audio streaming with video avatar output

## Usage

TavusTransport creates a three-way conversation between your Pipecat bot, a Tavus avatar, and human participants. The transport manages the Tavus API integration and Daily room connectivity automatically.

See the [complete example](https://github.com/pipecat-ai/pipecat/blob/main/examples/foundational/21a-tavus-video-service.py) for a full implementation including:

* Tavus transport configuration
* Avatar and replica setup
* Pipeline integration with TTS
* Event handling for participant management

---

# Source: https://docs.pipecat.ai/deployment/pipecat-cloud/guides/telephony/telnyx-websocket.md

> ## Documentation Index
> Fetch the complete documentation index at: https://docs.pipecat.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Telnyx Websocket Transport

> Using Telnyx's WebSocket Transport for your Pipecat Cloud agents

<Note>
  This guide covers Pipecat Cloud-specific configuration for Telnyx WebSocket
  integration. For a complete guide including dial-in, dial-out, custom
  parameters, and advanced features, see the [Telnyx WebSocket Integration
  guide](/guides/telephony/telnyx-websockets).
</Note>

Native support for Telnyx's WebSocket Transport with Pipecat Cloud allows you to connect your AI agents with Telnyx's voice infrastructure. This integration enables your Pipecat bots to handle real phone calls using Telnyx's WebSocket streaming.

## How It Works

Pipecat Cloud implements Telnyx's bidirectional [Media Streaming protocol](https://developers.telnyx.com/docs/voice/programmable-voice/media-streaming). While audio streams flow through WebSockets, the call session is controlled by updating the Telnyx Extensible Markup Language (TeXML) associated with each call's unique identifier (`call_control_id`).

When Pipecat Cloud receives an incoming WebSocket connection from Telnyx, it processes the `connected` and `start` messages to initialize a new bot instance. All WebSocket messages are forwarded to your bot, including any custom parameters set in your TeXML. This allows your bot to leverage Telnyx's Call Control API for advanced call control - such as recording conversations, transferring to human agents, or implementing complex call flows.

<Frame>
  <img src="https://mintcdn.com/daily/2bYrACcmgvvzC075/deployment/pipecat-cloud/images/telnyx-ws-flow.png?fit=max&auto=format&n=2bYrACcmgvvzC075&q=85&s=4d6b81ee1f466387cb42cfcda54ab599" data-og-width="3238" width="3238" data-og-height="1220" height="1220" data-path="deployment/pipecat-cloud/images/telnyx-ws-flow.png" data-optimize="true" data-opv="3" srcset="https://mintcdn.com/daily/2bYrACcmgvvzC075/deployment/pipecat-cloud/images/telnyx-ws-flow.png?w=280&fit=max&auto=format&n=2bYrACcmgvvzC075&q=85&s=663b41c3c94cd048b7d0d6709547306d 280w, https://mintcdn.com/daily/2bYrACcmgvvzC075/deployment/pipecat-cloud/images/telnyx-ws-flow.png?w=560&fit=max&auto=format&n=2bYrACcmgvvzC075&q=85&s=8ad41fb360f27b7a02e07978b4b10c14 560w, https://mintcdn.com/daily/2bYrACcmgvvzC075/deployment/pipecat-cloud/images/telnyx-ws-flow.png?w=840&fit=max&auto=format&n=2bYrACcmgvvzC075&q=85&s=0b6de82ed6463cbd73594c219390d293 840w, https://mintcdn.com/daily/2bYrACcmgvvzC075/deployment/pipecat-cloud/images/telnyx-ws-flow.png?w=1100&fit=max&auto=format&n=2bYrACcmgvvzC075&q=85&s=4fd54d5b0940a175b8a2f2d49f4300a9 1100w, https://mintcdn.com/daily/2bYrACcmgvvzC075/deployment/pipecat-cloud/images/telnyx-ws-flow.png?w=1650&fit=max&auto=format&n=2bYrACcmgvvzC075&q=85&s=63da73642f10dc539d3351fdfd00ec8c 1650w, https://mintcdn.com/daily/2bYrACcmgvvzC075/deployment/pipecat-cloud/images/telnyx-ws-flow.png?w=2500&fit=max&auto=format&n=2bYrACcmgvvzC075&q=85&s=794d6610dd0a31f7faa8d2aff650e4bb 2500w" />
</Frame>

## Prerequisites

Before setting up this integration, ensure you have:

* A Telnyx account with voice capabilities
* A Pipecat Cloud account with a Telnyx WebSocket-compatible bot

A ready-to-build example of a Telnyx websockets bot with complete source code is available in the [pipecat-examples repo](https://github.com/pipecat-ai/pipecat-examples/tree/main/telnyx-chatbot).

## Pipecat Cloud Configuration

### 1. Get Your Organization Name

Retrieve your Pipecat Cloud organization name using the Pipecat CLI:

```bash  theme={null}
$ pipecat cloud organizations list
```

This command will output a list of organizations associated with your account. For example:

```bash  theme={null}
Organization        Name
──────────────────────────────────────
Default Workspace   three-random-words-randomnumber (active)
```

### 2. Create a TeXML Application

[Purchase a phone number from Telnyx](https://telnyx.com/resources/purchase-a-phone-number-with-telnyx) if you haven't already, then create a [TeXML Application](https://developers.telnyx.com/docs/voice/programmable-voice/texml-setup) with the following configuration:

```xml  theme={null}
<?xml version="1.0" encoding="UTF-8"?>
<Response>
  <Connect>
    <Stream url="wss://api.pipecat.daily.co/ws/telnyx?serviceHost=AGENT_NAME.ORGANIZATION_NAME" bidirectionalMode="rtp"></Stream>
  </Connect>
  <Pause length="40"/>
</Response>
```

Replace the placeholder values:

* `AGENT_NAME` with your deployed bot's name (e.g., `my-first-agent`)
* `ORGANIZATION_NAME` with your organization name from step 1 (e.g., `three-random-words-randomnumber`)

For example, if your agent is named "customer-support" and your organization is "industrious-purple-cat-12345", your serviceHost would be: `customer-support.industrious-purple-cat-12345`

<Note>
  **Using Regional Endpoints**

  If you deployed your agent to a specific region, use the regional WebSocket endpoint to connect:

  `wss://{region}.api.pipecat.daily.co/ws/telnyx?serviceHost=AGENT_NAME.ORGANIZATION_NAME`

  For example, for Europe: `wss://eu-central.api.pipecat.daily.co/ws/telnyx?serviceHost=my-agent.my-org`

  Learn more about [regional endpoints](/deployment/pipecat-cloud/guides/regions#regional-websocket-endpoints).
</Note>

### 3. Assign TeXML Application to Your Phone Number

1. Navigate to Voice → Programmable Voice in your Telnyx dashboard
2. In the TeXML Applications tab, select the pencil icon for the TeXML Application you created in step 2
3. In the Numbers tab, select Assign numbers
4. Select the number you would like to assign the TeXML Application to
5. Click Save to apply your changes

## Testing Your Integration

To test your integration, simply dial your Telnyx phone number from any phone. The call will connect to your Pipecat Cloud bot, which will respond according to your bot's configuration.

## Next Steps

For complete implementation details including dial-out, caller personalization, custom parameters, and advanced call control features, see the [Telnyx WebSocket Integration guide](/guides/telephony/telnyx-websockets).

---

# Source: https://docs.pipecat.ai/guides/telephony/telnyx-websockets.md

> ## Documentation Index
> Fetch the complete documentation index at: https://docs.pipecat.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Telnyx WebSocket Integration

> Complete guide to using Telnyx Media Streaming with Pipecat for dial-in and dial-out functionality

## Things you'll need

* An active [Telnyx](https://telnyx.com) account with API credentials
* One or more Telnyx provisioned phone numbers
* A public-facing server or tunneling service like [ngrok](https://ngrok.com/) (for dial-out only)
* API keys for speech-to-text, text-to-speech, and LLM services

<CardGroup cols={2}>
  <Card title="Telnyx Dial-in Example" icon="phone-arrow-down-left" href="https://github.com/pipecat-ai/pipecat-examples/tree/main/telnyx-chatbot/inbound">
    Complete dial-in implementation using Telnyx Media Streaming over WebSocket
  </Card>

  <Card title="Telnyx Dial-out Example" icon="phone-arrow-up-right" href="https://github.com/pipecat-ai/pipecat-examples/tree/main/telnyx-chatbot/outbound">
    Outbound calling using Telnyx Media Streaming with WebSocket transport
  </Card>
</CardGroup>

## Phone Number Setup

You'll need Telnyx phone numbers for both dial-in and dial-out functionality:

* Visit [telnyx.com](https://telnyx.com) and purchase phone numbers
* Ensure your numbers support Voice capabilities
* Configure TeXML applications for dial-in numbers (covered below)

## Environment Setup

Configure your environment variables for Telnyx and AI services:

```shell .env theme={null}
TELNYX_API_KEY=...
OPENAI_API_KEY=...
DEEPGRAM_API_KEY=...
CARTESIA_API_KEY=...
```

## Dial-in

Dial-in allows users to call your Telnyx number and connect to your Pipecat bot via WebSocket Media Streaming. Unlike Twilio, Telnyx automatically provides caller information (to/from numbers) in the WebSocket messages, so no custom server is needed for basic dial-in functionality.

### How It Works

Here's the sequence of events when someone calls your Telnyx number:

1. **Telnyx receives an incoming call** to your phone number
2. **Telnyx executes your TeXML application** which establishes a WebSocket connection
3. **Telnyx opens a WebSocket** to your server with real-time audio and call metadata
4. **Your bot processes the audio** using the Pipecat pipeline
5. **The bot responds with audio** sent back to Telnyx over WebSocket
6. **Telnyx plays the audio** to the caller in real-time

### Set up your TeXML application

Telnyx uses TeXML (Telnyx Extensible Markup Language) to control call flow. For dial-in, you create a TeXML application that establishes a WebSocket connection directly to your bot:

```xml TeXML for dial-in theme={null}
<?xml version="1.0" encoding="UTF-8"?>
<Response>
  <Connect>
    <Stream url="wss://your-server.com/ws" bidirectionalMode="rtp"></Stream>
  </Connect>
  <Pause length="40"/>
</Response>
```

The `bidirectionalMode="rtp"` parameter enables real-time audio streaming in both directions.

### Personalizing Your Bot with Caller Information

Telnyx automatically includes caller information (to/from numbers) in the WebSocket messages. You can use the caller's phone number to:

* Look up customer information in your database
* Personalize greetings and responses based on caller identity
* Route calls to different bot behaviors (e.g., VIP handling, support vs sales)
* Implement custom business logic based on the caller

The example bot demonstrates how to extract caller information from Telnyx's WebSocket messages and use it to personalize the conversation.

### Configure your Pipecat bot for dial-in

Your bot receives the WebSocket connection and automatically gets call information from Telnyx's WebSocket messages. The key components are:

**WebSocket Parsing**: Pipecat's built-in parser extracts call data from Telnyx's WebSocket messages:

```python bot.py theme={null}
from pipecat.runner.utils import parse_telephony_websocket

async def run_bot(websocket: WebSocket):
    # Parse Telnyx WebSocket data
    transport_type, call_data = await parse_telephony_websocket(websocket)

    # Extract call information (automatically provided by Telnyx)
    # Data format: {
    #     "stream_id": str,
    #     "call_control_id": str,
    #     "outbound_encoding": str,
    #     "from": str,
    #     "to": str,
    # }
    stream_id = call_data["stream_id"]
    call_control_id = call_data["call_control_id"]
    outbound_encoding = call_data["outbound_encoding"]
    from_number = call_data["from"]  # Caller's number
    to_number = call_data["to"]      # Your Telnyx number
```

**Transport Creation**: Configure the WebSocket transport with Telnyx serialization:

```python bot.py theme={null}
# Create Telnyx serializer with call details
serializer = TelnyxFrameSerializer(
    stream_id=stream_id,
    call_control_id=call_control_id,
    api_key=os.getenv("TELNYX_API_KEY"),
)

# Configure WebSocket transport
transport = FastAPIWebsocketTransport(
    websocket=websocket,
    params=FastAPIWebsocketParams(
        audio_in_enabled=True,
        audio_out_enabled=True,
        add_wav_header=False,
        serializer=serializer,
    ),
)

# Your bot pipeline setup here...
```

<Card title="Complete Bot Implementation" icon="code" href="https://github.com/pipecat-ai/pipecat-examples/blob/main/telnyx-chatbot/inbound/bot.py">
  See the full bot.py with WebSocket parsing, transport setup, and pipeline
  configuration
</Card>

### Set up Telnyx TeXML application

Configure your Telnyx phone number to use your TeXML application:

1. Go to the [Telnyx Portal](https://portal.telnyx.com)
2. Navigate to Voice → Programmable Voice → TeXML Applications
3. Create a new TeXML Application with your WebSocket URL
4. Assign the TeXML Application to your phone number

<Card title="Complete Setup Instructions" icon="book-open" href="https://github.com/pipecat-ai/pipecat-examples/tree/main/telnyx-chatbot/inbound">
  See the full README with step-by-step setup, TeXML configuration, and testing
</Card>

## Dial-out

Dial-out allows your bot to initiate calls to phone numbers using Telnyx's outbound calling capabilities with WebSocket Media Streaming.

### How It Works

Here's the sequence of events for dial-out calls:

1. **Your application triggers a dial-out** (via API call or user action)
2. **Server initiates a Telnyx call** using the Call Control API
3. **Telnyx establishes the call** and opens a WebSocket connection
4. **Your bot joins the WebSocket** and sets up the pipeline
5. **The recipient answers** and is connected to your bot
6. **The bot handles the conversation** with real-time audio streaming

### Set up your server for dial-out

The dial-out server creates outbound calls and manages WebSocket connections. When you trigger a dial-out call, your server:

1. **Receives the dial-out request** with target phone number and parameters
2. **Creates a Telnyx call** using the Call Control API with a webhook URL
3. **Accepts the WebSocket** connection from Telnyx
4. **Parses call data** including call control information
5. **Runs the Pipecat bot** with the call information

<Card title="Complete Server Implementation" icon="code" href="https://github.com/pipecat-ai/pipecat-examples/blob/main/telnyx-chatbot/outbound/server.py">
  See the full FastAPI server code with outbound call creation and WebSocket
  handling
</Card>

### Passing Custom Data to Your Bot

For dial-out calls, you often need to pass custom data to personalize the conversation—such as user information, campaign details, or context from your application. The dial-out example supports passing a `body` object that becomes available to your bot via `runner_args.body`.

**Triggering a call with custom data:**

```bash  theme={null}
curl -X POST https://your-server.com/start \
  -H "Content-Type: application/json" \
  -d '{
    "phone_number": "+1234567890",
    "body": {
      "user": {
        "id": "user123",
        "firstName": "John",
        "lastName": "Doe",
        "accountType": "premium"
      },
      "campaign": "renewal_reminder",
      "context": "subscription expiring in 3 days"
    }
  }'
```

**How it works:**

1. The `/start` endpoint receives the request with `phone_number` and optional `body`
2. The `body` is base64-encoded and passed through the TeXML URL chain
3. When the WebSocket connects, the server decodes the body and passes it to your bot via `runner_args.body`
4. Your bot accesses the custom data to personalize the conversation

**Accessing custom data in your bot:**

```python bot.py theme={null}
async def bot(runner_args: RunnerArguments):
    body_data = runner_args.body or {}
    first_name = body_data.get("user", {}).get("firstName", "there")
    # Use first_name to personalize greetings, prompts, etc.
```

This approach works consistently in both local development and Pipecat Cloud production deployments.

### Configure your Pipecat bot for dial-out

The dial-out bot configuration is similar to dial-in, with Telnyx automatically providing call information.

**Call Information**: Telnyx provides call details in the WebSocket messages:

```python bot.py theme={null}
# Parse WebSocket data (same as dial-in)
transport_type, call_data = await parse_telephony_websocket(websocket)

# Extract call information
stream_id = call_data["stream_id"]
call_control_id = call_data["call_control_id"]
from_number = call_data["from"]  # Your Telnyx number
to_number = call_data["to"]      # Target number you're calling

# Customize bot behavior for outbound calls
greeting = f"Hi! This is an automated call from {from_number}. How are you today?"
```

**Custom Parameters for Dial-out**: You can pass custom data to your bot by adding query parameters to the WebSocket URL in your TeXML:

```python server.py theme={null}
# When creating the TeXML for outbound calls, include custom parameters
texml = f"""<?xml version="1.0" encoding="UTF-8"?>
<Response>
  <Connect>
    <Stream url="wss://your-server.com/ws?campaign_id=summer_sale&user_id={user_id}&priority=high" bidirectionalMode="rtp"></Stream>
  </Connect>
  <Pause length="40"/>
</Response>"""
```

These parameters are accessible in your server code and can be used to customize your bot's behavior based on the specific call context.

<Card title="Complete Bot Implementation" icon="code" href="https://github.com/pipecat-ai/pipecat-examples/blob/main/telnyx-chatbot/outbound/bot.py">
  See the full bot.py with outbound call handling and call information usage
</Card>

### Run the Example

To test dial-out functionality:

1. **Start your server**: Run your FastAPI server
2. **Trigger a call**: Make an API request to start an outbound call
3. **Answer your phone**: The bot will call the specified number
4. **Talk to your bot**: Have a conversation with your AI agent

<Card title="Complete Setup Instructions" icon="book-open" href="https://github.com/pipecat-ai/pipecat-examples/tree/main/telnyx-chatbot/outbound">
  See the full README with step-by-step setup, API usage, and outbound call
  configuration
</Card>

## Key Features

### Audio Format and Sample Rate

Telnyx Media Streaming uses 8kHz mono audio with 16-bit PCM encoding. Configure your pipeline accordingly:

```python  theme={null}
task = PipelineTask(
    pipeline,
    params=PipelineParams(
        audio_in_sample_rate=8000,
        audio_out_sample_rate=8000,
    ),
)
```

### Automatic Call Termination

When you provide Telnyx API credentials to the `TelnyxFrameSerializer`, it automatically ends calls when your pipeline ends:

```python  theme={null}
serializer = TelnyxFrameSerializer(
    stream_id=stream_id,
    call_control_id=call_control_id,
    api_key=os.getenv("TELNYX_API_KEY"),  # Enables auto-termination
)
```

### Built-in Call Information

Unlike other providers, Telnyx automatically includes caller information (to/from numbers) in the WebSocket messages, eliminating the need for custom webhook servers in basic dial-in scenarios.

## Deployment

### Pipecat Cloud Deployment

For production deployment without managing your own infrastructure, use Pipecat Cloud:

<Card title="Telnyx WebSocket on Pipecat Cloud" icon="cloud" href="/deployment/pipecat-cloud/guides/telephony/telnyx-websocket">
  Deploy Telnyx WebSocket bots with automatic scaling and managed infrastructure
</Card>

### Self-Hosted Production Deployment

For fully self-hosted production deployment, ensure your servers are:

* Publicly accessible with HTTPS
* Able to handle concurrent WebSocket connections
* Properly configured with Telnyx API credentials
* Implementing proper error handling and logging

## Next Steps

* Explore the [complete examples](https://github.com/pipecat-ai/pipecat-examples/tree/main/telnyx-chatbot) for full implementations
* Learn about [Daily + Twilio SIP integration](./twilio-daily-sip) for advanced telephony scenarios
* Check out [Daily PSTN integration](./daily-pstn) for direct phone number provisioning

---

# Source: https://docs.pipecat.ai/server/services/serializers/telnyx.md

> ## Documentation Index
> Fetch the complete documentation index at: https://docs.pipecat.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# TelnyxFrameSerializer

> Serializer for Telnyx WebSocket media streaming protocol

## Overview

`TelnyxFrameSerializer` enables integration with Telnyx's WebSocket media streaming protocol, allowing your Pipecat application to handle phone calls via Telnyx's voice services with bidirectional audio conversion, DTMF event handling, and support for multiple audio encodings.

<CardGroup cols={2}>
  <Card title="Telnyx Serializer API Reference" icon="code" href="https://reference-server.pipecat.ai/en/latest/api/pipecat.serializers.telnyx.html">
    Pipecat's API methods for Telnyx WebSocket integration
  </Card>

  <Card title="Example Implementation" icon="play" href="https://github.com/pipecat-ai/pipecat-examples/tree/main/telnyx-chatbot">
    Complete telephony examples with Telnyx
  </Card>

  <Card title="Telnyx Documentation" icon="book" href="https://developers.telnyx.com/docs/voice/media-streaming">
    Official Telnyx media streaming documentation
  </Card>

  <Card title="Telnyx Portal" icon="microphone" href="https://portal.telnyx.com/">
    Manage phone numbers and streaming configuration
  </Card>
</CardGroup>

## Installation

The `TelnyxFrameSerializer` does not require any additional dependencies beyond the core Pipecat library:

```bash  theme={null}
pip install "pipecat-ai"
```

## Prerequisites

### Telnyx Account Setup

Before using TelnyxFrameSerializer, you need:

1. **Telnyx Account**: Sign up at [Telnyx Portal](https://portal.telnyx.com/)
2. **Phone Number**: Purchase a Telnyx phone number with voice capabilities
3. **Media Streaming**: Configure your phone number for WebSocket streaming
4. **Webhook Configuration**: Set up webhook endpoints for call handling

### Required Environment Variables

* `TELNYX_API_KEY`: Your Telnyx API key for authentication and call control

### Required Configuration

* **Stream ID**: Provided by Telnyx during WebSocket connection
* **Audio Encodings**: Configure inbound/outbound encodings (PCMU, PCMA)
* **Call Control ID**: Required for automatic call termination (optional)

### Key Features

* **Bidirectional Audio**: Convert between Pipecat and Telnyx audio formats
* **DTMF Handling**: Process touch-tone events from callers
* **Auto Hang-up**: Terminate calls via Telnyx's REST API
* **Multiple Encodings**: Support for PCMU and PCMA audio formats

---

# Source: https://docs.pipecat.ai/guides/learn/text-to-speech.md

> ## Documentation Index
> Fetch the complete documentation index at: https://docs.pipecat.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Text to Speech

> Learn how to configure speech synthesis to convert text into natural-sounding audio in your voice AI pipeline

**Text to Speech (TTS)** services are responsible for converting text into natural-sounding speech audio. They receive text input from LLMs and other sources, then generate audio output that users can hear through their connected devices.

## Pipeline Placement

TTS processors must be positioned correctly in your pipeline to receive text and generate audio frames:

```python  theme={null}
pipeline = Pipeline([
    transport.input(),
    stt,
    context_aggregator.user(),
    llm,                           # Generates LLMTextFrames
    tts,                           # Processes text → creates TTSAudioRawFrames
    transport.output(),            # Sends audio to user
    context_aggregator.assistant(), # Processes TTSTextFrames for context
])
```

**Placement requirements:**

* **After LLM processing**: TTS needs `LLMTextFrame`s from language model responses
* **Before transport output**: Audio must be generated before sending to user
* **Before assistant context aggregator**: Ensures spoken text is captured in conversation history

### Frame Processing Flow

**TTS generates speech through two primary mechanisms:**

1. **Streamed LLM tokens** via `LLMTextFrame`s:

   * TTS aggregates streaming tokens into complete sentences
   * Sentences are sent to TTS service for audio generation
   * Audio bytes stream back and play immediately
   * End-to-end latency often under 200ms

2. **Direct speech requests** via `TTSSpeakFrame`s:
   * Bypasses LLM and context aggregators
   * Immediate audio generation for specific text
   * Useful for system messages or prompts

**Frame output:**

* `TTSAudioRawFrame`s: Raw audio data for playback
* `TTSTextFrame`s: Text that was actually spoken (for context updates)
* `TTSStartedFrame`/`TTSStoppedFrame`: Speech boundary markers

## Supported TTS Services

Pipecat supports a wide range of TTS providers with different capabilities and performance characteristics:

<Card title="Supported TTS Services" icon="list" href="/server/services/supported-services#text-to-speech">
  View the complete list of supported text-to-speech providers
</Card>

### Service Categories

**WebSocket-Based Services (Recommended):**

* **Cartesia**: Ultra-low latency with word timestamps
* **ElevenLabs**: High-quality voices with emotion control
* **Rime**: Ultra-realistic voices with advanced features

**HTTP-Based Services:**

* **OpenAI TTS**: High-quality synthesis with multiple voices
* **Azure Speech**: Enterprise-grade with extensive language support
* **Google Text-to-Speech**: Reliable with WaveNet voices

**Advanced Features:**

* **Word timestamps**: Enable word-level accuracy for context and subtitles
* **Voice cloning**: Custom voice creation from samples
* **Emotion control**: Dynamic emotional expression
* **SSML support**: Fine-grained pronunciation control

<Note>
  WebSocket services typically provide the lowest latency, while HTTP services
  may have intermittent higher latency due to their request/response nature.
</Note>

## TTS Configuration

### Service-Specific Configuration

Each TTS service has its own configuration options. Here's an example with Cartesia:

```python  theme={null}
from pipecat.services.cartesia.tts import CartesiaTTSService
from pipecat.transcriptions.language import Language

tts = CartesiaTTSService(
    api_key=os.getenv("CARTESIA_API_KEY"),
    voice_id="voice-id-here",
    model="sonic-2",              # TTS model to use
    params=CartesiaTTSService.InputParams(
        language=Language.EN,     # Speech language
        speed="normal",           # Speech rate control
    ),
    # Word timestamps automatically enabled for precise context updates
)
```

**Word timestamps:** Services like Cartesia, ElevenLabs, and Rime provide word-level timestamps that enable precise context updates during interruptions and better synchronization with other pipeline components. For example, if an interruption occurs while the bot is speaking, the word timestamps allow you to accurately capture which words were spoken up to that point, enabling better context management and user experience. Additionally, transcription events streamed from server to client can be done in sync with the audio output, allowing for real-time subtitles or captions.

<Card title="Individual TTS Services" icon="settings" href="/server/services/supported-services#text-to-speech">
  Explore configuration options for each supported TTS provider
</Card>

### Pipeline-Level Audio Configuration

Set consistent audio settings across your entire pipeline:

```python  theme={null}
task = PipelineTask(
    pipeline,
    params=PipelineParams(
        audio_in_sample_rate=16000,   # Input audio quality
        audio_out_sample_rate=24000,  # Output audio quality (TTS)
    ),
)
```

<Tip>
  Set the `audio_out_sample_rate` to match your TTS service's requirements for
  optimal quality. This is preferred to setting the sample\_rate directly in the
  TTS service as the PipelineParam ensures that all output sample\_rates match.
</Tip>

## Text Processing and Filtering

### Custom Text Aggregation

By default, TTS services have a built-in text aggregator that collects streaming text into sentences before passing them to the underlying service. However, you can customize this behavior by inserting an [`LLMTextProcessor`](/server/utilities/frame/llm-text-processor) with a different text aggregator before the TTS in your pipeline. This allows the ability to categorize and structure text into logical units beyond simple sentences, such as code blocks, URLs, or custom tags. You can then configure the TTS to handle these different text types appropriately, such as skipping code blocks or transforming them in a just-in-time manner before speaking.

### Skipping Text Aggregations

To skip certain text aggregations (e.g., code snippets or URLs) and keep them from being spoken, use a custom text aggregator like [`PatternPairAggregator`](/server/utilities/text/pattern-pair-aggregator) within an [`LLMTextProcessor`](/server/utilities/frame/llm-text-processor), and configure it to identify and handle specific patterns in the text stream. With this, you can then pass any aggregated types you want to skip (like "code") to the TTS service's `skip_aggregator_types` parameter.

```python  theme={null}
# Create pattern aggregator
pattern_aggregator = PatternPairAggregator()

# Add pattern for JSON data
pattern_aggregator.add_pattern(
    type="code",
    start_pattern="<code>",
    end_pattern="</code>",
    action=MatchAction.AGGREGATE
)

# Set the aggregator on an LLMTextProcessor
llm_text_processor = LLMTextProcessor(text_aggregator=pattern_aggregator)

# Initialize TTS service, and don't speak JSON data
tts = CartesiaTTSService(
    api_key=os.getenv("CARTESIA_API_KEY"),
    skip_aggregator_types=["code"], # The strings here should match the types defined in the PatternPairAggregator
)

# add the llm_text_processor to your pipeline after the llm and before the tts
# llm -> llm_text_processor -> tts
```

### Text Transforms

For TTS-specific text preprocessing, you can provide custom text transforms that modify text in a just-in-time manner before sending the text off to the TTS service. This is useful for handling special text segments that need to be altered for better pronunciation or clarity, such as spelling out phone numbers, removing URLs, or expanding abbreviations. These text transforms can be mapped to a specific text aggregation type, like with `skip_aggregator_types`, or applied globally to all text using `'*'` as the type.

Text transforms are registered directly on the TTS service instance via the `add_text_transformer()` method or during initialization using the `text_transforms` parameter.

<Note>
  The intentions of text transforms are meant to be TTS-specific modifications
  that do not affect the underlying LLM text or context. That said, since the
  context aggregator attempts to base its context on what was actually spoken,
  for services that support word timestamps, like Cartesia, ElevenLabs, and
  Rime,these transforms will modify the context as they modify what is spoken.
</Note>

```python  theme={null}
# Create pattern aggregator
pattern_aggregator = PatternPairAggregator()

# Add patterns for different parts of an explanation
pattern_aggregator.add_pattern(
    type="phone_number",
    start_pattern="<pnum>",
    end_pattern="</pnum>",
    action=MatchAction.AGGREGATE
)

# Set the aggregator on an LLMTextProcessor
llm_text_processor = LLMTextProcessor(text_aggregator=pattern_aggregator)

# Text-to-Speech service
tts = CartesiaTTSService(
    api_key=os.getenv("CARTESIA_API_KEY"),
)

# Text transformers for TTS
# This will insert Cartesia's spell tags around the provided text.
async def spell_out_text(text: str, type: str) -> str:
    # CartesiaTTSService provides a helper for this along with other common transforms
    return CartesiaTTSService.SPELL(text)

async def replace_acronyms(text: str, type: str) -> str:
    # Replace "SEC" with "Southeastern Conference"
    return text.replace(" SEC ", " Southeastern Conference ")

# Setup the text transformers in TTS to spell out phone numbers and replace
# acronyms. The string below matches the type defined in the PatternPairAggregator
# above so that whenever those segments are encountered, this transform
# is applied
tts.add_text_transformer(spell_out_text, "phone_number")
tts.add_text_transformer(replace_acronyms, "*")  # Apply to all text

# add the llm_text_processor to your pipeline after the llm and before the tts
#   llm -> llm_text_processor -> tts
```

### Text Filters

<Warning>
  Text filters are no longer the preferred method for text preprocessing and
  will be deprecated in future releases. Instead, you should use one of the
  methods described above.
</Warning>

Apply preprocessing to text before synthesis:

```python  theme={null}
from pipecat.utils.text.markdown_text_filter import MarkdownTextFilter

tts = YourTTSService(
    # ... other options
    text_filters=[
        MarkdownTextFilter(),     # Remove markdown formatting
        CustomTextFilter(),       # Your custom processing
    ],
)
```

**Common filters:**

* **MarkdownTextFilter**: Strips markdown formatting from LLM responses
* **Custom filters**: Implement your own text preprocessing logic

## Advanced TTS Features

### Direct Speech Commands

Use `TTSSpeakFrame` for immediate speech synthesis:

```python  theme={null}
from pipecat.frames.frames import TTSSpeakFrame

# Make bot speak directly
await tts.queue_frame(TTSSpeakFrame("Hello, how can I help you?"))
```

### Dynamic Settings Updates

Update TTS settings during conversation:

```python  theme={null}
from pipecat.frames.frames import TTSUpdateSettingsFrame

# Change voice speed during conversation
await task.queue_frames([
    TTSUpdateSettingsFrame({"speed": "fast"}),
    TTSSpeakFrame("I'm speaking faster now!")
])
```

## Key Takeaways

* **Pipeline placement matters** - TTS must come after LLM, before transport output
* **Service types differ** - WebSocket services provide lower latency than HTTP
* **Text processing affects quality** - use aggregation and filters for better results
* **Word timestamps enable precision** - better interruption handling and context accuracy
* **Configuration impacts performance** - balance quality, latency, and bandwidth needs
* **Services are modular** - easily swap providers without changing pipeline code

## What's Next

You've now learned how to build a complete voice AI pipeline! Let's explore some additional topics to enhance your implementation.

<Card title="Pipeline Termination" icon="arrow-right" href="/guides/learn/pipeline-termination">
  Learn how to terminate your voice AI pipeline at the end of a conversation
</Card>

---

# Source: https://docs.pipecat.ai/server/services/llm/together.md

> ## Documentation Index
> Fetch the complete documentation index at: https://docs.pipecat.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Together AI

> LLM service implementation using Together AI's API with OpenAI-compatible interface

## Overview

`TogetherLLMService` provides access to Together AI's language models, including Meta's Llama 3.1 and 3.2 models, through an OpenAI-compatible interface. It inherits from `OpenAILLMService` and supports streaming responses, function calling, and context management with optimized open-source model hosting.

<CardGroup cols={2}>
  <Card title="Together AI LLM API Reference" icon="code" href="https://reference-server.pipecat.ai/en/latest/api/pipecat.services.together.llm.html">
    Pipecat's API methods for Together AI integration
  </Card>

  <Card title="Example Implementation" icon="play" href="https://github.com/pipecat-ai/pipecat/blob/main/examples/foundational/14c-function-calling-together.py">
    Complete example with function calling
  </Card>

  <Card title="Together AI Documentation" icon="book" href="https://docs.together.ai/reference/chat-completions-1">
    Official Together AI API documentation and features
  </Card>

  <Card title="Together AI Platform" icon="microphone" href="https://together.ai/">
    Access open-source models and manage API keys
  </Card>
</CardGroup>

## Installation

To use Together AI services, install the required dependencies:

```bash  theme={null}
pip install "pipecat-ai[together]"
```

## Prerequisites

### Together AI Account Setup

Before using Together AI LLM services, you need:

1. **Together AI Account**: Sign up at [Together AI](https://together.ai/)
2. **API Key**: Generate an API key from your account dashboard
3. **Model Selection**: Choose from available open-source models (Llama, Mistral, etc.)

### Required Environment Variables

* `TOGETHER_API_KEY`: Your Together AI API key for authentication

---

# Source: https://docs.pipecat.ai/server/utilities/transcript-processor.md

> ## Documentation Index
> Fetch the complete documentation index at: https://docs.pipecat.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# TranscriptProcessor

> Factory for creating and managing conversation transcript processors with shared event handling

<Warning>
  DEPRECATED: TranscriptProcessor has been deprecated. Use
  `on_user_turn_stopped` and `on_assistant_turn_stopped` events on the context
  aggregators to collect transcriptions, see
  [Transcriptions](/server/utilities/turn-management/transcriptions) for
  details.
</Warning>

## Overview

The `TranscriptProcessor` is a factory class that creates and manages processors for handling conversation transcripts from both users and assistants. It provides unified access to transcript processors with shared event handling, making it easy to track and respond to conversation updates in real-time.

The processor normalizes messages from various sources into a consistent `TranscriptionMessage` format and emits events when new messages are added to the conversation.

## Constructor

```python  theme={null}
TranscriptProcessor()
```

Creates a new transcript processor factory with no parameters.

## Methods

### user()

```python  theme={null}
def user(**kwargs) -> UserTranscriptProcessor
```

Get or create the user transcript processor instance. This processor handles `TranscriptionFrame`s from STT services.

**Parameters:**

* `**kwargs`: Arguments passed to the `UserTranscriptProcessor` constructor

**Returns:** `UserTranscriptProcessor` instance for processing user messages.

### assistant()

```python  theme={null}
def assistant(**kwargs) -> AssistantTranscriptProcessor
```

Get or create the assistant transcript processor instance. This processor handles `TTSTextFrame`s from TTS services and aggregates them into complete utterances.

**Parameters:**

* `**kwargs`: Arguments passed to the `AssistantTranscriptProcessor` constructor

**Returns:** `AssistantTranscriptProcessor` instance for processing assistant messages.

### event\_handler()

```python  theme={null}
def event_handler(event_name: str)
```

Decorator that registers event handlers for both user and assistant processors.

**Parameters:**

* `event_name`: Name of the event to handle

**Returns:** Decorator function that registers the handler with both processors.

## Event Handlers

### on\_transcript\_update

Triggered when new messages are added to the conversation transcript.

```python  theme={null}
@transcript.event_handler("on_transcript_update")
async def handle_transcript_update(processor, frame):
    # Handle transcript updates
    pass
```

**Parameters:**

* `processor`: The specific processor instance that emitted the event (UserTranscriptProcessor or AssistantTranscriptProcessor)
* `frame`: `TranscriptionUpdateFrame` containing the new messages

## Data Structures

### TranscriptionMessage

```python  theme={null}
@dataclass
class TranscriptionMessage:
    role: Literal["user", "assistant"]
    content: str
    timestamp: str | None = None
    user_id: str | None = None
```

**Fields:**

* `role`: The message sender type ("user" or "assistant")
* `content`: The transcribed text content
* `timestamp`: ISO 8601 timestamp when the message was created
* `user_id`: Optional user identifier (for user messages only)

### TranscriptionUpdateFrame

Frame containing new transcript messages, emitted by the `on_transcript_update` event.

**Properties:**

* `messages`: List of `TranscriptionMessage` objects containing the new transcript content

## Frames

### UserTranscriptProcessor

* **Input:** `TranscriptionFrame` from STT services
* **Output:** `TranscriptionMessage` with role "user"

### AssistantTranscriptProcessor

* **Input:** `TTSTextFrame` from TTS services
* **Output:** `TranscriptionMessage` with role "assistant"

## Integration Notes

### Pipeline Placement

Place the processors at specific positions in your pipeline for accurate transcript collection:

```python  theme={null}
pipeline = Pipeline([
    transport.input(),
    stt,                        # Speech-to-text service
    transcript.user(),          # Place after STT
    context_aggregator.user(),
    llm,
    tts,                        # Text-to-speech service
    transport.output(),
    transcript.assistant(),     # Place after transport.output()
    context_aggregator.assistant(),
])
```

### Event Handler Registration

Event handlers are automatically applied to both user and assistant processors:

```python  theme={null}
transcript = TranscriptProcessor()

# This handler will receive events from both processors
@transcript.event_handler("on_transcript_update")
async def handle_update(processor, frame):
    for message in frame.messages:
        print(f"{message.role}: {message.content}")
```

---

# Source: https://docs.pipecat.ai/server/utilities/observers/transcription-observer.md

> ## Documentation Index
> Fetch the complete documentation index at: https://docs.pipecat.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Transcription Log Observer

> Logging speech-to-text transcription activity in Pipecat

The `TranscriptionLogObserver` logs all speech-to-text transcription activity in your Pipecat pipeline, providing visibility into both final and interim transcription results.

## Frame Types Monitored

The observer tracks the following frame types (only from STT service):

* **TranscriptionFrame**: Final transcription results
* **InterimTranscriptionFrame**: In-progress transcription results

## Usage

```python  theme={null}
from pipecat.observers.loggers.transcription_log_observer import TranscriptionLogObserver

task = PipelineTask(
    pipeline,
    params=PipelineParams(
        observers=[TranscriptionLogObserver()],
    ),
)
```

## Log Output Format

The observer uses consistent formatting with emoji indicators:

* 💬 \[Source] → TRANSCRIPTION: \[text] from \[user\_id]
* 💬 \[Source] → INTERIM TRANSCRIPTION: \[text] from \[user\_id]

All log entries include timestamps for precise timing analysis.

---

# Source: https://docs.pipecat.ai/server/utilities/turn-management/transcriptions.md

> ## Documentation Index
> Fetch the complete documentation index at: https://docs.pipecat.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Transcriptions

> Collect user and assistant conversation transcripts using turn events

## Overview

Pipecat provides a straightforward way to collect conversation transcriptions using [turn events](/server/utilities/turn-management/turn-events). When a user or assistant turn ends, the corresponding event includes the complete transcript for that turn.

The key events for transcription collection are:

* **`on_user_turn_stopped`** - Provides the user's complete transcript via `UserTurnStoppedMessage`
* **`on_assistant_turn_stopped`** - Provides the assistant's complete transcript via `AssistantTurnStoppedMessage`

## Basic Example

```python  theme={null}
from pipecat.processors.aggregators.llm_response_universal import (
    LLMContextAggregatorPair,
    UserTurnStoppedMessage,
    AssistantTurnStoppedMessage,
)

# Create context aggregator
user_aggregator, assistant_aggregator = LLMContextAggregatorPair(context)

# Handle user transcriptions
@user_aggregator.event_handler("on_user_turn_stopped")
async def on_user_turn_stopped(aggregator, strategy, message: UserTurnStoppedMessage):
    print(f"[USER] {message.content}")

# Handle assistant transcriptions
@assistant_aggregator.event_handler("on_assistant_turn_stopped")
async def on_assistant_turn_stopped(aggregator, message: AssistantTurnStoppedMessage):
    print(f"[ASSISTANT] {message.content}")
```

## Saving Transcripts to a File

```python  theme={null}
import json
from datetime import datetime

transcript_log = []

@user_aggregator.event_handler("on_user_turn_stopped")
async def on_user_turn_stopped(aggregator, strategy, message: UserTurnStoppedMessage):
    transcript_log.append({
        "role": "user",
        "content": message.content,
        "timestamp": message.timestamp,
        "user_id": message.user_id,
    })

@assistant_aggregator.event_handler("on_assistant_turn_stopped")
async def on_assistant_turn_stopped(aggregator, message: AssistantTurnStoppedMessage):
    transcript_log.append({
        "role": "assistant",
        "content": message.content,
        "timestamp": message.timestamp,
    })

# Save transcript when session ends
async def save_transcript():
    with open(f"transcript_{datetime.now().isoformat()}.json", "w") as f:
        json.dump(transcript_log, f, indent=2)
```

## Sending Transcripts to an External Service

```python  theme={null}
import aiohttp

async def send_to_service(role: str, content: str, timestamp: str):
    async with aiohttp.ClientSession() as session:
        await session.post(
            "https://api.example.com/transcripts",
            json={"role": role, "content": content, "timestamp": timestamp}
        )

@user_aggregator.event_handler("on_user_turn_stopped")
async def on_user_turn_stopped(aggregator, strategy, message: UserTurnStoppedMessage):
    await send_to_service("user", message.content, message.timestamp)

@assistant_aggregator.event_handler("on_assistant_turn_stopped")
async def on_assistant_turn_stopped(aggregator, message: AssistantTurnStoppedMessage):
    await send_to_service("assistant", message.content, message.timestamp)
```

## Message Types

For details on `UserTurnStoppedMessage` and `AssistantTurnStoppedMessage` fields, see [Turn Events - Message Types](/server/utilities/turn-management/turn-events#message-types).

## Related

* [Turn Events](/server/utilities/turn-management/turn-events) - Complete reference for turn lifecycle events
* [Saving Transcripts Guide](/guides/fundamentals/saving-transcripts) - Guide for saving conversation transcripts

---

# Source: https://docs.pipecat.ai/server/utilities/runner/transport-utils.md

> ## Documentation Index
> Fetch the complete documentation index at: https://docs.pipecat.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Transport Utilities

> Configuration and helper utilities for Daily, LiveKit, telephony, and WebRTC transports

## Overview

Pipecat provides several utility modules for configuring transports and handling transport-specific operations. While the [development runner](/server/utilities/runner/guide) handles most of these automatically, these utilities are useful for custom setups, advanced configurations, or when building your own deployment infrastructure.

## Daily Configuration

The Daily utilities handle room creation, token generation, and authentication setup for Daily integration. They support both standard video/audio rooms and SIP-enabled rooms for telephony integration.

### Basic Configuration

Use `configure()` for simple room and token setup:

```python  theme={null}
import aiohttp
from pipecat.runner.daily import configure

async with aiohttp.ClientSession() as session:
    # Returns a DailyRoomConfig object
    room_config = await configure(session)

    room_url = room_config.room_url
    token = room_config.token

    # Use with DailyTransport
    transport = DailyTransport(room_url, token, "Bot Name", params=DailyParams())
```

The `configure()` function returns a `DailyRoomConfig` object with:

* `room_url`: The Daily room URL for joining
* `token`: Authentication token for the bot
* `sip_endpoint`: SIP endpoint URI (None for standard rooms)

This function:

* Uses `DAILY_SAMPLE_ROOM_URL` environment variable if set, otherwise creates a new room
* Creates rooms with 2-hour expiration and automatic ejection by default
* Generates an authentication token with 2-hour expiration using `DAILY_API_KEY`

### Environment Variables

**Required**:

* `DAILY_API_KEY`: Daily API key for creating rooms and tokens

**Optional**:

* `DAILY_SAMPLE_ROOM_URL`: Use an existing room instead of creating one
* `DAILY_API_URL`: Override Daily API endpoint (defaults to [https://api.daily.co/v1](https://api.daily.co/v1))

### Room and Token Management

When no `DAILY_SAMPLE_ROOM_URL` is provided, rooms are created automatically with:

* 2-hour expiration from creation time
* Automatic participant ejection when the room expires
* Unique names using UUID prefixes (e.g. `pipecat-uuid`)

<Note>
  Expired rooms are automatically cleaned up by Daily, so you don't need to
  manage them manually.
</Note>

Tokens are generated with 2-hour expiration and include necessary permissions for bot participation. The utilities handle all the Daily REST API interactions automatically.

### SIP Configuration

Daily utilities support SIP-enabled rooms for telephony integration. When SIP parameters are provided, the function creates rooms with telephony capabilities:

```python  theme={null}
import aiohttp
from pipecat.runner.daily import configure

async with aiohttp.ClientSession() as session:
    # Create SIP-enabled room
    sip_config = await configure(
        session,
        sip_caller_phone="+15551234567",
        sip_enable_video=False,  # Voice-only by default
        sip_num_endpoints=1,
        sip_codecs={"audio": ["OPUS"], "video": ["H264"]}
    )

    print(f"Room URL: {sip_config.room_url}")
    print(f"SIP Endpoint: {sip_config.sip_endpoint}")
    print(f"Token: {sip_config.token}")
```

**SIP Parameters**:

* `sip_caller_phone`: Phone number or identifier for SIP display name (enables SIP mode)
* `sip_enable_video`: Whether video is enabled for SIP calls (default: False)
* `sip_num_endpoints`: Number of allowed SIP endpoints (default: 1)
* `sip_codecs`: Audio/video codecs to support (optional, uses Daily defaults if not specified)

<Note>
  SIP-enabled rooms cannot use existing room URLs from `DAILY_SAMPLE_ROOM_URL`.
  They always create new temporary rooms with SIP configuration.
</Note>

### Backward Compatibility

The `configure()` function supports tuple unpacking:

```python  theme={null}
# Tuple unpacking (legacy style)
room_url, token = await configure(session)

# Object access (recommended)
config = await configure(session)
room_url = config.room_url
token = config.token
```

## LiveKit Configuration

LiveKit utilities manage authentication tokens, room setup, and agent permissions for LiveKit server integration.

### Basic Configuration

Use `configure()` for standard setup:

```python  theme={null}
from pipecat.runner.livekit import configure

url, token, room_name = await configure()

# Use with LiveKitTransport
transport = LiveKitTransport(url=url, token=token, room_name=room_name, params=LiveKitParams())
```

### Configuration with Arguments

For command-line integration:

```python  theme={null}
import argparse
from pipecat.runner.livekit import configure_with_args

parser = argparse.ArgumentParser()
url, token, room_name, args = await configure_with_args(parser)
```

Supports these command-line options:

* `-r, --room`: Specify LiveKit room name
* `-u, --url`: Specify LiveKit server URL

### Token Generation

LiveKit provides two token generation functions:

**`generate_token(room_name, participant_name, api_key, api_secret)`**
Creates a standard participant token for users or testing.

**`generate_token_with_agent(room_name, participant_name, api_key, api_secret)`**
Creates an agent token with special permissions. Use this for your bots.

```python  theme={null}
from pipecat.runner.livekit import generate_token_with_agent

# Generate agent token for your bot
agent_token = generate_token_with_agent("my-room", "Pipecat Bot", api_key, api_secret)

# Generate user token for testing
user_token = generate_token("my-room", "Test User", api_key, api_secret)
```

### Environment Variables

**Required**:

* `LIVEKIT_API_KEY`: LiveKit API key
* `LIVEKIT_API_SECRET`: LiveKit API secret
* `LIVEKIT_URL`: LiveKit server URL
* `LIVEKIT_ROOM_NAME`: Default room name

All environment variables are required for LiveKit to function properly.

## WebSocket and Transport Utilities

The transport utilities provide helper functions for WebSocket parsing, SDP manipulation, and transport management.

### Telephony WebSocket Parsing

Use `parse_telephony_websocket()` to auto-detect telephony providers and extract call data:

```python  theme={null}
from pipecat.runner.utils import parse_telephony_websocket

transport_type, call_data = await parse_telephony_websocket(websocket)

if transport_type == "twilio":
    stream_id = call_data["stream_id"]
    call_id = call_data["call_id"]
elif transport_type == "telnyx":
    stream_id = call_data["stream_id"]
    call_control_id = call_data["call_control_id"]
    outbound_encoding = call_data["outbound_encoding"]
elif transport_type == "plivo":
    stream_id = call_data["stream_id"]
    call_id = call_data["call_id"]
elif transport_type == "exotel":
    stream_sid=call_data["stream_id"],
    call_sid=call_data["call_id"],
```

The function automatically:

* Reads and parses initial WebSocket messages
* Detects provider based on message structure
* Extracts provider-specific call information
* Returns structured data for transport configuration

### Transport Helper Functions

**Client ID Detection**:

```python  theme={null}
from pipecat.runner.utils import get_transport_client_id

client_id = get_transport_client_id(transport, client)
# Returns pc_id for WebRTC or participant ID for Daily
```

**Video Capture** (Daily only):

```python  theme={null}
from pipecat.runner.utils import maybe_capture_participant_camera, maybe_capture_participant_screen

# Capture participant's camera
await maybe_capture_participant_camera(transport, client, framerate=30)

# Capture participant's screen
await maybe_capture_participant_screen(transport, client, framerate=15)
```

These functions safely handle transport detection and only execute if the transport supports the operation.

## When to Use These Utilities

### Automatic Usage (Most Common)

The development runner and `create_transport` utility handle these automatically. Most users won't need to call these functions directly.

### Manual Usage (Advanced)

Use these utilities directly when:

**Custom deployment infrastructure**: Building your own bot runner or deployment system

**Advanced transport configuration**: Need specific room settings, token permissions, or custom authentication

**Non-runner scenarios**: Integrating Pipecat transports into existing applications

**Testing and debugging**: Need to create rooms/tokens independently for testing

### Integration Example

Here's how you might use these utilities in a custom deployment:

```python  theme={null}
import aiohttp
from pipecat.runner.daily import configure
from pipecat.runner.utils import parse_telephony_websocket
from pipecat.transports.daily.transport import DailyTransport, DailyParams

async def create_custom_bot_session(transport_type: str):
    if transport_type == "daily":
        async with aiohttp.ClientSession() as session:
            room_config = await configure(session)

            room_url = room_config.room_url
            token = room_config.token

            return DailyTransport(room_url, token, "Custom Bot", DailyParams())

    elif transport_type == "telephony":
        # Handle custom telephony setup
        transport_type, call_data = await parse_telephony_websocket(websocket)
        # Configure based on detected provider...
```

These utilities provide the building blocks for any transport configuration scenario while maintaining the same reliability and functionality as the development runner.

---

# Source: https://docs.pipecat.ai/client/js/transports/transport.md

# Source: https://docs.pipecat.ai/client/c++/transport.md

> ## Documentation Index
> Fetch the complete documentation index at: https://docs.pipecat.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Daily WebRTC Transport

> WebRTC implementation for C++ using Daily

The Daily transport implementation enables real-time audio and video communication in your Pipecat C++ applications using [Daily's](https://www.daily.co/) WebRTC infrastructure.

## Dependencies

### Daily Core C++ SDK

Download the [Daily Core C++ SDK](https://github.com/daily-co/daily-core-sdk) from the [available releases](https://github.com/daily-co/daily-core-sdk/releases) for your platform and set:

```bash  theme={null}
export DAILY_CORE_PATH=/path/to/daily-core-sdk
```

### Pipecat C++ SDK

Build the base [Pipecat C++ SDK](https://github.com/pipecat-ai/pipecat-client-cxx-daily) first and set:

```bash  theme={null}
export PIPECAT_SDK_PATH=/path/to/pipecat-client-cxx
```

## Building

First, set a few environment variables:

```bash  theme={null}
PIPECAT_SDK_PATH=/path/to/pipecat-client-cxx
DAILY_CORE_PATH=/path/to/daily-core-sdk
```

Then, build the project:

<Tabs>
  <Tab title="Linux/macOS">
    ```bash  theme={null}
    cmake . -G Ninja -Bbuild -DCMAKE_BUILD_TYPE=Release
    ninja -C build
    ```
  </Tab>

  <Tab title="Windows">
    ```bash  theme={null}
    # Initialize Visual Studio environment
    "C:\Program Files (x86)\Microsoft Visual Studio\2019\Professional\VC\Auxiliary\Build\vcvarsall.bat" amd64

    # Configure and build
    cmake . -Bbuild --preset vcpkg
    cmake --build build --config Release
    ```
  </Tab>
</Tabs>

## Examples

<CardGroup cols={3}>
  <Card title="Basic Client" icon="code" href="https://github.com/pipecat-ai/pipecat-client-cxx-daily/blob/main/examples/c++">
    Simple C++ implementation example
  </Card>

  {" "}

  <Card title="Audio Client" icon="waveform" href="https://github.com/pipecat-ai/pipecat-client-cxx-daily/blob/main/examples/c++-portaudio">
    C++ client with PortAudio support
  </Card>

  <Card title="Node.js Server" icon="server" href="https://github.com/pipecat-ai/pipecat-client-cxx-daily/blob/main/examples/server">
    Example Node.js proxy implementation
  </Card>
</CardGroup>

---

# Source: https://docs.pipecat.ai/guides/learn/transports.md

> ## Documentation Index
> Fetch the complete documentation index at: https://docs.pipecat.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Transports

> Learn about the different ways users can connect to your Pipecat voice AI bot

**Transports** are the communication layer between users and your Pipecat bot. They handle receiving and sending audio, video, and data, serving as the media interface that enables real-time interaction.

## Available Transport Types

Pipecat supports multiple transport types to fit different use cases and deployment scenarios:

<CardGroup cols={2}>
  <Card title="DailyTransport" icon="microphone" href="/server/services/transport/daily">
    WebRTC-based transport using Daily's infrastructure for video calls and
    conferencing
  </Card>

  <Card title="FastAPIWebsocketTransport" icon="plug" href="/server/services/transport/fastapi-websocket">
    WebSocket transport for telephony providers and custom WebSocket connections
  </Card>

  <Card title="HeyGenTransport" icon="camera" href="/server/services/transport/heygen">
    Specialized transport for HeyGen LiveAvatar video generation and streaming
  </Card>

  <Card title="LiveKitTransport" icon="microphone" href="/server/services/transport/livekit">
    WebRTC transport using LiveKit's real-time communication platform
  </Card>

  <Card title="SmallWebRTCTransport" icon="link" href="/server/services/transport/small-webrtc">
    Direct peer-to-peer WebRTC connections without cloud infrastructure
  </Card>

  <Card title="TavusTransport" icon="camera" href="/server/services/transport/tavus">
    Specialized transport for Tavus video generation and streaming
  </Card>

  <Card title="WebsocketTransport" icon="globe" href="/server/services/transport/websocket-server">
    General-purpose WebSocket transport for custom implementations
  </Card>
</CardGroup>

## Pipeline Integration

Transports provide two key components for your pipeline: `input()` and `output()` methods. These methods define how the transport interacts with the pipeline:

### Transport Input and Output

```python  theme={null}
pipeline = Pipeline([
    transport.input(),              # Receives user audio/video
    stt,
    context_aggregator.user(),
    llm,
    tts,
    transport.output(),             # Sends bot audio/video
    context_aggregator.assistant(), # Processes after output
])
```

**Key points about transport placement:**

* **`transport.input()`** typically goes first in the pipeline to receive user input
* **`transport.output()`** doesn't always go last - you may want processors after it
* **Post-output processing** enables synchronized actions like:
  * Recording with word-level accuracy
  * Displaying subtitles synchronized to audio
  * Capturing context information precisely timed to output

## Transport Modularity

Transports are modular components in your Pipeline, allowing you to flexibly change how users connect to your bot depending on the context. This modularity enables you to:

* **Switch environments easily**: Use P2P WebRTC for development, Daily for production
* **Support multiple connection types**: Same bot logic works across different transports
* **Optimize for use case**: Choose the best transport for your specific requirements

## Transport Configuration

All transports are configured using `TransportParams`, which provides common settings across transport types:

```python  theme={null}
from pipecat.transports.base_transport import TransportParams

params = TransportParams(
    # Audio settings
    audio_in_enabled=True,
    audio_out_enabled=True,

    # Video settings
    video_in_enabled=False,
    video_out_enabled=False,

    # Video stream configuration
    video_out_width=1024,
    video_out_height=576,
    video_out_bitrate=800000,
    video_out_framerate=30,
)
```

<Note>
  Each transport may have its own specialized parameters class that extends
  TransportParams with transport-specific options. Check the individual
  transport documentation for details.
</Note>

<Tip>
  For advanced turn detection (like Smart Turn), configure [User Turn
  Strategies](/server/utilities/turn-management/user-turn-strategies) on the
  context aggregator instead of using the transport's turn\_analyzer parameter.
</Tip>

<Card title="TransportParams Reference" icon="code" href="https://reference-server.pipecat.ai/en/latest/api/pipecat.transports.base_transport.html#pipecat.transports.base_transport.TransportParams">
  Complete reference for all transport configuration options
</Card>

## Telephony Integration

Telephony services (phone calls) use WebSocket connections with specialized serialization:

### Supported Telephony Providers

<CardGroup cols={2}>
  <Card title="Twilio" href="/server/services/serializers/twilio">
    Media Streams over WebSocket with TwilioFrameSerializer
  </Card>

  <Card title="Telnyx" href="/server/services/serializers/telnyx">
    Real-time media streaming with TelnyxFrameSerializer
  </Card>

  <Card title="Plivo" href="/server/services/serializers/plivo">
    Voice streaming API with PlivoFrameSerializer
  </Card>

  <Card title="Exotel" href="/server/services/serializers/exotel">
    Voice streaming integration with ExotelFrameSerializer
  </Card>
</CardGroup>

### Telephony Transport Setup

Telephony requires a `FrameSerializer` to handle provider-specific message formats:

```python  theme={null}
# Create provider-specific serializer
serializer = TwilioFrameSerializer(
    stream_sid=stream_sid,
    call_sid=call_sid,
    account_sid=os.getenv("TWILIO_ACCOUNT_SID", ""),
    auth_token=os.getenv("TWILIO_AUTH_TOKEN", ""),
)

# Configure transport with serializer
transport = FastAPIWebsocketTransport(
    websocket=websocket_client,
    params=FastAPIWebsocketParams(
        audio_in_enabled=True,
        audio_out_enabled=True,
        add_wav_header=False,
        serializer=serializer,  # Provider-specific serialization
    ),
)
```

<Note>
  The development runner automatically detects and configures the appropriate
  serializer when using `parse_telephony_websocket()`.
</Note>

## Conditional Transport Selection

The development runner provides a pattern for conditionally selecting transports based on the environment:

```python  theme={null}
async def bot(runner_args: RunnerArguments):
    """Main bot entry point compatible with Pipecat Cloud."""

    transport = None

    if isinstance(runner_args, DailyRunnerArguments):
        from pipecat.transports.daily.transport import DailyParams, DailyTransport

        transport = DailyTransport(
            runner_args.room_url,
            runner_args.token,
            "Pipecat Bot",
            params=DailyParams(
                audio_in_enabled=True,
                audio_out_enabled=True,
            ),
        )

    elif isinstance(runner_args, SmallWebRTCRunnerArguments):
        from pipecat.transports.base_transport import TransportParams
        from pipecat.transports.network.small_webrtc import SmallWebRTCTransport

        transport = SmallWebRTCTransport(
            params=TransportParams(
                audio_in_enabled=True,
                audio_out_enabled=True,
            ),
            webrtc_connection=runner_args.webrtc_connection,
        )
    else:
        logger.error(f"Unsupported runner arguments type: {type(runner_args)}")
        return

    if transport is None:
        logger.error("Failed to create transport")
        return

    await run_bot(transport)
```

This pattern allows you to run the same bot code across different environments with different connection types.

## WebRTC vs WebSocket Considerations

Understanding when to use each connection type is crucial for building effective voice AI applications:

### WebRTC (Recommended for Client Applications)

**Best for:** Browser apps, mobile apps, real-time conversations

**Advantages:**

* **Low latency**: Optimized for real-time media with minimal delay
* **Built-in resilience**: Handles packet loss and network variations
* **Advanced audio processing**: Echo cancellation, noise reduction, automatic gain control
* **Quality monitoring**: Detailed performance and media quality statistics
* **Automatic timestamping**: Simplifies interruption and playout logic
* **Robust reconnection**: Built-in connection management

**Use WebRTC when:**

* Building client-facing applications (web, mobile)
* Conversational latency is critical
* Users are on potentially unreliable networks
* You need built-in audio processing features

### WebSocket (Good for Server-to-Server)

**Best for:** Telephony integration, server-to-server communication, prototyping

**Limitations for real-time media:**

* **TCP-based**: Subject to head-of-line blocking
* **Network sensitivity**: Less resilient to packet loss and jitter
* **Manual implementation**: Requires custom logic for reconnection, timestamping
* **Limited observability**: Harder to monitor connection quality

**Use WebSocket when:**

* Integrating with telephony providers (Twilio, Telnyx, etc.)
* Building server-to-server connections
* Prototyping or latency isn't critical
* Working within existing WebSocket infrastructure

## Key Takeaways

* **Transports are modular** - swap them without changing bot logic
* **Choose based on use case** - WebRTC for clients, WebSocket for telephony
* **Configuration is standardized** - TransportParams work across transport types
* **Pipeline placement matters** - consider what processing happens after output
* **Development runner helps** - provides patterns for multi-transport bots

## What's Next

Now that you understand how transports connect users to your bot, let's explore how to configure speech recognition to convert user audio into text.

<Card title="Speech Input & Turn Detection" icon="arrow-right" href="/guides/learn/speech-input">
  Learn how to configure speech recognition in your voice AI pipeline
</Card>

---

# Source: https://docs.pipecat.ai/server/utilities/turn-management/turn-events.md

> ## Documentation Index
> Fetch the complete documentation index at: https://docs.pipecat.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Turn Events

> Handle user and assistant turn lifecycle events for transcriptions and turn tracking

## Overview

Turn events provide hooks into the conversation turn lifecycle, allowing you to know when users and assistants start or stop speaking. These events are emitted by the context aggregators (`LLMUserAggregator` and `LLMAssistantAggregator`) and are particularly useful for:

* **Collecting transcriptions** - Get complete user and assistant transcripts when turns end
* **Turn tracking** - Monitor conversation flow and timing
* **Analytics** - Measure turn durations, detect timeouts, and track conversation patterns

## Events Summary

| Event                       | Emitter                | Description                                         |
| --------------------------- | ---------------------- | --------------------------------------------------- |
| `on_user_turn_started`      | `user_aggregator`      | User begins speaking                                |
| `on_user_turn_stopped`      | `user_aggregator`      | User finishes speaking (includes transcript)        |
| `on_user_turn_stop_timeout` | `user_aggregator`      | User turn ended due to timeout                      |
| `on_user_turn_idle`         | `user_aggregator`      | User has been idle (not speaking) for timeout       |
| `on_assistant_turn_started` | `assistant_aggregator` | Assistant begins responding                         |
| `on_assistant_turn_stopped` | `assistant_aggregator` | Assistant finishes responding (includes transcript) |

## User Turn Events

User turn events are registered on the `user_aggregator` from an `LLMContextAggregatorPair`.

### on\_user\_turn\_started

Fired when a user turn is detected to have started, based on the configured [start strategies](/server/utilities/turn-management/user-turn-strategies#start-strategies).

```python  theme={null}
@user_aggregator.event_handler("on_user_turn_started")
async def on_user_turn_started(aggregator, strategy):
    print(f"User started speaking (detected by {strategy})")
```

**Parameters:**

| Parameter    | Type                        | Description                                |
| ------------ | --------------------------- | ------------------------------------------ |
| `aggregator` | `LLMUserAggregator`         | The user aggregator instance               |
| `strategy`   | `BaseUserTurnStartStrategy` | The strategy that triggered the turn start |

### on\_user\_turn\_stopped

Fired when a user turn is detected to have ended, based on the configured [stop strategies](/server/utilities/turn-management/user-turn-strategies#stop-strategies). This event includes the complete user transcript for the turn.

```python  theme={null}
@user_aggregator.event_handler("on_user_turn_stopped")
async def on_user_turn_stopped(aggregator, strategy, message: UserTurnStoppedMessage):
    print(f"User said: {message.content}")
    print(f"Turn started at: {message.timestamp}")
    if message.user_id:
        print(f"User ID: {message.user_id}")
```

**Parameters:**

| Parameter    | Type                       | Description                                 |
| ------------ | -------------------------- | ------------------------------------------- |
| `aggregator` | `LLMUserAggregator`        | The user aggregator instance                |
| `strategy`   | `BaseUserTurnStopStrategy` | The strategy that triggered the turn stop   |
| `message`    | `UserTurnStoppedMessage`   | Contains the user's transcript and metadata |

### on\_user\_turn\_stop\_timeout

Fired when a user turn times out without any stop strategy triggering. This is a fallback mechanism that ends the turn after a configurable timeout period (default: 5.0 seconds) when the user has stopped speaking according to VAD but no transcription-based stop has occurred. Commonly, this event is used to retrigger the LLM response after the user has stopped speaking.

```python  theme={null}
@user_aggregator.event_handler("on_user_turn_stop_timeout")
async def on_user_turn_stop_timeout(aggregator):
    message = {
        "role": "system",
        "content": "Continue.",
    }
    await user_aggregator.queue_frame(LLMMessagesAppendFrame([message], run_llm=True))
```

**Parameters:**

| Parameter    | Type                | Description                  |
| ------------ | ------------------- | ---------------------------- |
| `aggregator` | `LLMUserAggregator` | The user aggregator instance |

<Note>
  After `on_user_turn_stop_timeout` fires, `on_user_turn_stopped` will also be
  called with the accumulated transcript.
</Note>

### on\_user\_turn\_idle

Fired when the user has been idle (not speaking) for a configured timeout period. This event is useful for re-engaging users who may have stepped away or need a prompt to continue the conversation. The idle timer only starts after the first conversation activity and does not trigger while the bot is speaking or function calls are in progress.

```python  theme={null}
@user_aggregator.event_handler("on_user_turn_idle")
async def on_user_turn_idle(aggregator):
    # Re-engage the user with a contextual prompt
    message = {
        "role": "system",
        "content": "The user has been quiet. Politely and briefly ask if they're still there.",
    }
    await aggregator.push_frame(LLMMessagesAppendFrame([message], run_llm=True))
```

**Parameters:**

| Parameter    | Type                | Description                  |
| ------------ | ------------------- | ---------------------------- |
| `aggregator` | `LLMUserAggregator` | The user aggregator instance |

<Note>
  The idle timer resets on user or bot speaking activity, and does not trigger
  during function call execution. This event can fire multiple times if the user
  remains idle.
</Note>

## Assistant Turn Events

Assistant turn events are registered on the `assistant_aggregator` from an `LLMContextAggregatorPair`.

### on\_assistant\_turn\_started

Fired when the assistant begins generating a response.

```python  theme={null}
@assistant_aggregator.event_handler("on_assistant_turn_started")
async def on_assistant_turn_started(aggregator):
    print("Assistant started responding")
```

**Parameters:**

| Parameter    | Type                     | Description                       |
| ------------ | ------------------------ | --------------------------------- |
| `aggregator` | `LLMAssistantAggregator` | The assistant aggregator instance |

### on\_assistant\_turn\_stopped

Fired when the assistant finishes responding or is interrupted. This event includes the complete assistant transcript for the turn.

```python  theme={null}
@assistant_aggregator.event_handler("on_assistant_turn_stopped")
async def on_assistant_turn_stopped(aggregator, message: AssistantTurnStoppedMessage):
    print(f"Assistant said: {message.content}")
    print(f"Turn started at: {message.timestamp}")
```

**Parameters:**

| Parameter    | Type                          | Description                                      |
| ------------ | ----------------------------- | ------------------------------------------------ |
| `aggregator` | `LLMAssistantAggregator`      | The assistant aggregator instance                |
| `message`    | `AssistantTurnStoppedMessage` | Contains the assistant's transcript and metadata |

<Note>
  This event fires when the LLM response completes, when the user interrupts, or
  when a user image is appended to context.
</Note>

## Message Types

### UserTurnStoppedMessage

Contains the user's complete transcript when their turn ends.

```python  theme={null}
from pipecat.processors.aggregators.llm_response_universal import UserTurnStoppedMessage
```

<ParamField path="content" type="str">
  The complete transcribed text from the user's turn.
</ParamField>

<ParamField path="timestamp" type="str">
  ISO 8601 timestamp indicating when the user turn started.
</ParamField>

<ParamField path="user_id" type="Optional[str]">
  Optional identifier for the user, if available from the transport.
</ParamField>

### AssistantTurnStoppedMessage

Contains the assistant's complete transcript when their turn ends.

```python  theme={null}
from pipecat.processors.aggregators.llm_response_universal import AssistantTurnStoppedMessage
```

<ParamField path="content" type="str">
  The complete text content from the assistant's turn.
</ParamField>

<ParamField path="timestamp" type="str">
  ISO 8601 timestamp indicating when the assistant turn started.
</ParamField>

## Related

* [Transcriptions](/server/utilities/turn-management/transcriptions) - Collect conversation transcripts using turn events
* [User Turn Strategies](/server/utilities/turn-management/user-turn-strategies) - Configure turn detection behavior
* [Turn Tracking Observer](/server/utilities/observers/turn-tracking-observer) - Track complete turn cycles with timing

---

# Source: https://docs.pipecat.ai/server/utilities/observers/turn-tracking-observer.md

> ## Documentation Index
> Fetch the complete documentation index at: https://docs.pipecat.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Turn Tracking Observer

> Track conversation turns and events in your Pipecat pipeline

The `TurnTrackingObserver` monitors and tracks conversational turns in your Pipecat pipeline, providing events when turns start and end. It intelligently identifies when a user-bot interaction cycle begins and completes.

## Turn Lifecycle

A turn represents a complete user-bot interaction cycle:

1. **Start**: When the user starts speaking (or pipeline starts for first turn)
2. **Processing**: User speaks, bot processes and responds
3. **End**: After the bot finishes speaking and either:
   * The user starts speaking again
   * A timeout period elapses with no further activity

## Events

The observer emits two main events:

* **`on_turn_started`**: When a new turn begins
  * Parameters: `turn_number` (int)
* **`on_turn_ended`**: When a turn completes
  * Parameters: `turn_number` (int), `duration` (float, in seconds), `was_interrupted` (bool)

## Usage

The observer is automatically created when you initialize a `PipelineTask` with `enable_turn_tracking=True` (which is the default):

```python  theme={null}
task = PipelineTask(
    pipeline,
    # Turn tracking is enabled by default
)

# Access the observer
turn_observer = task.turn_tracking_observer

# Register event handlers
@turn_observer.event_handler("on_turn_started")
async def on_turn_started(observer, turn_number):
    logger.info(f"Turn {turn_number} started")

@turn_observer.event_handler("on_turn_ended")
async def on_turn_ended(observer, turn_number, duration, was_interrupted):
    status = "interrupted" if was_interrupted else "completed"
    logger.info(f"Turn {turn_number} {status} in {duration:.2f}s")
```

## Configuration

You can configure the observer's behavior when creating a `PipelineTask`:

```python  theme={null}
from pipecat.observers.turn_tracking_observer import TurnTrackingObserver

# Create a custom observer instance
custom_turn_tracker = TurnTrackingObserver(
    turn_end_timeout_secs=3.5,     # Turn end timeout (default: 2.5)
)

# Add it as a regular observer
task = PipelineTask(
    pipeline,
    observers=[custom_turn_tracker],
    # Disable the default one if adding your own
    enable_turn_tracking=False,
)
```

## Interruptions

The observer automatically detects interruptions when the user starts speaking while the bot is still speaking. In this case:

* The current turn is marked as interrupted (`was_interrupted=True`)
* A new turn begins immediately

## How It Works

The observer monitors specific frame types to track conversation flow:

* **StartFrame**: Initiates the first turn
* **UserStartedSpeakingFrame**: Starts user speech or triggers a new turn
* **BotStartedSpeakingFrame**: Marks bot speech beginning
* **BotStoppedSpeakingFrame**: Starts the turn end timeout

After a bot stops speaking, the observer waits for the configured timeout period. If no further bot speech occurs, the turn ends; otherwise, it continues as part of the same turn.

## Use Cases

* **Analytics**: Measure turn durations, interruption rates, and conversation flow
* **Logging**: Record turn-based logs for diagnostics and analysis
* **Visualization**: Show turn-based conversation timelines in UIs
* **Tracing**: Group spans and metrics by conversation turns

---

# Source: https://docs.pipecat.ai/guides/telephony/twilio-daily-sip.md

> ## Documentation Index
> Fetch the complete documentation index at: https://docs.pipecat.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Daily + Twilio SIP

> Complete guide to using Daily's WebRTC transport with Twilio's SIP services for dial-in and dial-out

## Things you'll need

* An active [Twilio](https://www.twilio.com) developer account with API credentials
* One or more Twilio provisioned phone numbers
* Daily API key for WebRTC transport
* The Twilio Python client library (`pip install twilio`)

<CardGroup cols={2}>
  <Card title="SIP Dial-in Example" icon="phone-arrow-down-left" href="https://github.com/pipecat-ai/pipecat-examples/tree/main/phone-chatbot/daily-twilio-sip-dial-in">
    Complete dial-in implementation using Twilio SIP with Daily WebRTC transport
  </Card>

  <Card title="SIP Dial-out Example" icon="phone-arrow-up-right" href="https://github.com/pipecat-ai/pipecat-examples/tree/main/phone-chatbot/daily-twilio-sip-dial-out">
    Outbound calling using Daily WebRTC transport with Twilio SIP routing
  </Card>
</CardGroup>

## Phone Number Setup

You'll need Twilio phone numbers for both dial-in and dial-out functionality:

* Visit [console.twilio.com](https://console.twilio.com) and purchase phone numbers
* Ensure your numbers support Voice capabilities
* Configure webhook URLs for dial-in numbers (covered below)

## Environment Setup

Configure your environment variables for both Twilio and Daily:

```shell .env theme={null}
DAILY_API_KEY=...
DAILY_API_URL=https://api.daily.co/v1
TWILIO_ACCOUNT_SID=...
TWILIO_AUTH_TOKEN=...
OPENAI_API_KEY=...
CARTESIA_API_KEY=...
```

## Dial-in

Dial-in allows users to call your Twilio number and connect to your Pipecat bot via Daily's WebRTC transport.

### How It Works

Here's the sequence of events when someone calls your Twilio number:

1. **Twilio receives an incoming call** to your phone number
2. **Twilio calls your webhook server** (`/call` endpoint)
3. **Your server creates a Daily room** with SIP capabilities
4. **Your server starts the bot process** with the room details
5. **Your server responds to Twilio** with TwiML that puts the caller on hold with music
6. **Upon receiving the `on_dialin_ready` event**, the bot forwards the call to the Daily SIP endpoint
7. **The caller and bot are connected**, and the bot handles the conversation

### Set up your webhook server

Your server acts as the orchestrator between Twilio's telephony infrastructure and Daily's WebRTC transport. When Twilio receives a call, it sends a webhook to your server. Your server then:

1. **Extracts call information** from Twilio's webhook (CallSid, caller details)
2. **Creates a Daily room** with SIP capabilities enabled
3. **Spawns a bot process** with the room details and call information
4. **Responds to Twilio** with TwiML that puts the caller on hold with music
5. **Coordinates the connection** between Twilio SIP and Daily WebRTC

The server handles the complex orchestration between two different telephony systems, ensuring seamless audio quality through Daily's WebRTC transport while leveraging Twilio's robust SIP infrastructure.

<Card title="Complete Server Implementation" icon="code" href="https://github.com/pipecat-ai/pipecat-examples/blob/main/phone-chatbot/daily-twilio-sip-dial-in/server.py">
  See the full FastAPI server code with Twilio webhook handling and Daily room
  creation
</Card>

### Configure your Pipecat bot for dial-in

The bot receives arguments from the server process and uses them to coordinate between Daily's WebRTC transport and Twilio's SIP infrastructure. The key components are:

**Bot Entry Point**: The bot receives arguments containing:

* `room_url`: Daily room URL for the WebRTC connection
* `token`: Daily room token for authentication
* `call_sid`: Twilio call identifier for SIP forwarding
* `sip_endpoint`: Daily's SIP endpoint URL for call routing

**Transport Creation**: These arguments are used to configure the `DailyTransport` for WebRTC:

```python bot.py theme={null}
from pipecat.transports.daily import DailyTransport, DailyParams

async def run_bot(room_url: str, token: str, call_sid: str, sip_endpoint: str):
    # Create Daily WebRTC transport
    transport = DailyTransport(
        room_url,
        token,
        "Voice Bot",
        DailyParams(
            audio_in_enabled=True,
            audio_out_enabled=True,
            video_out_enabled=False,
            transcription_enabled=True,
        )
    )

    # Handle when Daily SIP endpoint is ready for call forwarding
    @transport.event_handler("on_dialin_ready")
    async def on_dialin_ready(transport, cdata):
        # Forward the Twilio call to Daily's SIP endpoint
        twilio_client.calls(call_sid).update(
            twiml=f"<Response><Dial><Sip>{sip_endpoint}</Sip></Dial></Response>"
        )
        logger.info("Call forwarded from Twilio to Daily SIP endpoint")

    # Your bot pipeline setup here...
```

**Call Forwarding Flow**: The `on_dialin_ready` event is crucial. It signals when Daily's SIP infrastructure is ready to receive the call. At this point, the bot uses Twilio's API to update the call with new TwiML that forwards the audio from Twilio's SIP to Daily's SIP endpoint, completing the connection.

<Card title="Complete Bot Implementation" icon="code" href="https://github.com/pipecat-ai/pipecat-examples/blob/main/phone-chatbot/daily-twilio-sip-dial-in/bot.py">
  See the full bot.py with Daily transport setup and Twilio call forwarding
</Card>

### Set up Twilio webhook

Configure your Twilio phone number to use your server's webhook URL:

1. Go to the [Twilio Console](https://console.twilio.com)
2. Navigate to Phone Numbers → Manage → Active Numbers
3. Click on your phone number
4. Under "Configure", set "A Call Comes In" to:
   * Webhook: `https://your-server.com/call` (your server's URL)
   * HTTP Method: POST

### Run the Example

For local development, you can use [ngrok](https://ngrok.com/) to expose your local server:

```shell  theme={null}
# Start your server
python server.py

# In another terminal, start ngrok
ngrok http 8000

# Use the ngrok URL (e.g., https://a1b2c3.ngrok.io/call) as your webhook
```

<Card title="Complete Setup Instructions" icon="book-open" href="https://github.com/pipecat-ai/pipecat-examples/tree/main/phone-chatbot/daily-twilio-sip-dial-in">
  See the full README with step-by-step setup, webhook configuration, and
  troubleshooting
</Card>

## Dial-out

Dial-out allows your bot to initiate calls through Twilio's SIP infrastructure using Daily's WebRTC transport.

### How It Works

Here's the sequence of events for dial-out calls:

1. **Your application triggers a dial-out** (via API call or user action)
2. **Server creates a Daily room** with SIP capabilities
3. **Bot joins the Daily room** and sets up the WebRTC transport
4. **Bot initiates SIP call** through Twilio to the target number
5. **Twilio routes the call** to the destination phone number
6. **The recipient answers** and is connected to your bot via Daily WebRTC
7. **The bot handles the conversation** with high-quality WebRTC audio

### Set up your server for dial-out

The dial-out server orchestrates outbound calls by coordinating between your application, Daily's WebRTC transport, and Twilio's SIP infrastructure. When you trigger a dial-out call, your server:

1. **Receives the dial-out request** with target phone number and call parameters
2. **Creates a Daily room** with SIP capabilities and dial-out enabled
3. **Spawns a bot process** with the room details and target phone number
4. **Initiates the SIP call** through Twilio to the destination number
5. **Manages the connection** between Daily WebRTC and Twilio SIP routing
6. **Handles call events** like answered, busy, or failed connections

The server simplifies outbound calling by abstracting the complexity of coordinating between WebRTC transport and SIP telephony, providing high-quality audio through Daily while leveraging Twilio's reliable call routing.

<Card title="Complete Server Implementation" icon="code" href="https://github.com/pipecat-ai/pipecat-examples/blob/main/phone-chatbot/daily-twilio-sip-dial-out/server.py">
  See the full FastAPI server code with Daily room creation and SIP dial-out
  management
</Card>

### Configure your Pipecat bot for dial-out

The dial-out bot receives arguments from the server process and uses them to initiate outbound calls through Twilio's SIP infrastructure via Daily's WebRTC transport. The key components are:

**Bot Entry Point**: The bot receives arguments containing:

* `room_url`: Daily room URL for the WebRTC connection
* `token`: Daily room token for authentication
* `target_number`: Phone number to call via Twilio SIP
* `sip_uri`: Twilio SIP URI for routing the outbound call

**Transport Creation**: These arguments are used to configure the `DailyTransport` for dial-out:

```python bot.py theme={null}
from pipecat.transports.daily import DailyTransport, DailyParams

async def run_bot(room_url: str, token: str, target_number: str, sip_uri: str):
    # Create Daily WebRTC transport
    transport = DailyTransport(
        room_url,
        token,
        "Voice Bot",
        DailyParams(
            audio_in_enabled=True,
            audio_out_enabled=True,
            video_out_enabled=False,
            transcription_enabled=True,
        )
    )

    # Initiate the outbound call through Twilio SIP
    await transport.start_dialout(sip_uri)

    # Your bot pipeline setup here...
```

**Dial-out Flow**: The `start_dialout()` method initiates the call by connecting Daily's WebRTC transport to Twilio's SIP infrastructure. The `sip_uri` parameter contains the Twilio SIP endpoint configured with the target phone number, allowing Daily to route the call through Twilio's telephony network to reach the destination.

<Card title="Complete Bot Implementation" icon="code" href="https://github.com/pipecat-ai/pipecat-examples/blob/main/phone-chatbot/daily-twilio-sip-dial-out/bot.py">
  See the full bot.py with WebRTC transport setup and SIP dial-out configuration
</Card>

### Run the Example

To test dial-out functionality:

1. **Start your server**: Run your FastAPI server
2. **Trigger a call**: Make an API request to start a dial-out call
3. **Answer your phone**: The bot will call the specified number via Twilio
4. **Talk to your bot**: Have a conversation with high-quality WebRTC audio

<Card title="Complete Setup Instructions" icon="book-open" href="https://github.com/pipecat-ai/pipecat-examples/tree/main/phone-chatbot/daily-twilio-sip-dial-out">
  See the full README with step-by-step setup, API usage, and SIP configuration
</Card>

## Best Practices

### Hold Music for Initialization

Always respond to Twilio's initial webhook with hold music to give your bot time to initialize:

```python  theme={null}
resp = VoiceResponse()
resp.play(url="https://your-hold-music.mp3", loop=10)
return str(resp)
```

**Avoid using `<Pause>`** - Twilio's pause duration is limited and may not provide enough time for Daily SIP setup.

### Handle Multiple Events

Use a flag to ensure you only forward calls once when handling multiple `on_dialin_ready` events:

```python  theme={null}
call_already_forwarded = False

@transport.event_handler("on_dialin_ready")
async def on_dialin_ready(transport, cdata):
    nonlocal call_already_forwarded
    if call_already_forwarded:
        return
    # Forward the call...
    call_already_forwarded = True
```

## Deployment

### Local Development

Use [ngrok](https://ngrok.com/) to expose your local server for testing:

```shell  theme={null}
python server.py
ngrok http 8000
# Use the ngrok URL as your Twilio webhook
```

### Pipecat Cloud Deployment

For production deployment on Pipecat Cloud, your webhook server calls the `/{agent}/start` endpoint instead of spawning local bot processes:

```python  theme={null}
# Instead of subprocess.Popen(bot_cmd)
response = await httpx.post(
    f"https://api.pipecat.daily.co/v1/public/{agent_id}/start",
    headers={"Authorization": f"Bearer {pipecat_api_key}"},
    json={
        "createDailyRoom": True,
        "dailyRoomProperties": {
            "sip": {"sip_mode": "dial-in", "num_endpoints": 1}
        },
        "body": {
            "room_url": room_url,
            "token": token,
            "call_sid": call_sid,  # For dial-in
            "sip_endpoint": sip_endpoint,
            # Or for dial-out:
            # "target_number": target_number,
            # "sip_uri": sip_uri
        }
    }
)
```

The room information and call parameters are passed in the `body` field, which your Pipecat Cloud bot receives as arguments.

### Self-Hosted Production Deployment

For self-hosted production deployment, ensure your servers are:

* Publicly accessible with HTTPS
* Able to handle concurrent webhook requests
* Properly configured with both Twilio and Daily API credentials

## Next Steps

* Explore the [complete examples](https://github.com/pipecat-ai/pipecat-examples/tree/main/phone-chatbot) for full implementations
* Learn about [Daily PSTN integration](./daily-pstn) for direct phone number provisioning
* Check out [Twilio WebSocket integration](./twilio-websockets) for simpler telephony workflows

---

# Source: https://docs.pipecat.ai/deployment/pipecat-cloud/guides/telephony/twilio-websocket.md

> ## Documentation Index
> Fetch the complete documentation index at: https://docs.pipecat.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Twilio Websocket Transport

> Using Twilio's WebSocket Transport for your Pipecat Cloud agents

<Note>
  This guide covers Pipecat Cloud-specific configuration for Twilio WebSocket
  integration. For a complete guide including dial-in, dial-out, custom
  parameters, and advanced features, see the [Twilio WebSocket Integration
  guide](/guides/telephony/twilio-websockets).
</Note>

Native support for Twilio's WebSocket Transport with Pipecat Cloud allows you to connect your AI agents with Twilio's voice infrastructure. This integration enables your Pipecat bots to handle real phone calls using Twilio's Websockets.

## How It Works

Pipecat Cloud implements Twilio's bidirectional [Media Streams protocol](https://www.twilio.com/docs/voice/media-streams/websocket-messages). While audio streams flow through WebSockets, the call session is controlled by updating the Twilio Markup Language (TwiML) associated with each call's unique identifier (`CallSid`).

When Pipecat Cloud receives an incoming WebSocket connection from Twilio, it processes the `Connected` and `Start` messages to initialize a new bot instance. All WebSocket messages are forwarded to your bot, including any custom parameters set in your TwiML. This allows your bot to leverage Twilio's REST API for advanced call control - such as recording conversations, transferring to human agents, or implementing complex call flows.

<Frame>
  <img src="https://mintcdn.com/daily/2bYrACcmgvvzC075/deployment/pipecat-cloud/images/twilio-ws-flow.png?fit=max&auto=format&n=2bYrACcmgvvzC075&q=85&s=6555c5049696069d3c1711606e84db1c" data-og-width="3238" width="3238" data-og-height="1220" height="1220" data-path="deployment/pipecat-cloud/images/twilio-ws-flow.png" data-optimize="true" data-opv="3" srcset="https://mintcdn.com/daily/2bYrACcmgvvzC075/deployment/pipecat-cloud/images/twilio-ws-flow.png?w=280&fit=max&auto=format&n=2bYrACcmgvvzC075&q=85&s=9f3205f69a37c029600c597b2103a4dd 280w, https://mintcdn.com/daily/2bYrACcmgvvzC075/deployment/pipecat-cloud/images/twilio-ws-flow.png?w=560&fit=max&auto=format&n=2bYrACcmgvvzC075&q=85&s=b118152c93d946697e49897e1cc1667d 560w, https://mintcdn.com/daily/2bYrACcmgvvzC075/deployment/pipecat-cloud/images/twilio-ws-flow.png?w=840&fit=max&auto=format&n=2bYrACcmgvvzC075&q=85&s=fb9498d8ceea59d077bcb36a02efeafe 840w, https://mintcdn.com/daily/2bYrACcmgvvzC075/deployment/pipecat-cloud/images/twilio-ws-flow.png?w=1100&fit=max&auto=format&n=2bYrACcmgvvzC075&q=85&s=5f96344c0fbd948691d20914f2e0d681 1100w, https://mintcdn.com/daily/2bYrACcmgvvzC075/deployment/pipecat-cloud/images/twilio-ws-flow.png?w=1650&fit=max&auto=format&n=2bYrACcmgvvzC075&q=85&s=b779c87cbc390f72fb09e3069821b6d1 1650w, https://mintcdn.com/daily/2bYrACcmgvvzC075/deployment/pipecat-cloud/images/twilio-ws-flow.png?w=2500&fit=max&auto=format&n=2bYrACcmgvvzC075&q=85&s=0c94974bca2e510f964c6a13d56d1ac7 2500w" />
</Frame>

## Prerequisites

Before setting up this integration, ensure you have:

* A Twilio account with voice capabilities
* A Pipecat Cloud account with a Twilio Websockets-compatible bot

A ready-to-build example of a Twilio websockets bot with complete source code is available in the [pipecat-examples repo](https://github.com/pipecat-ai/pipecat-examples/tree/main/twilio-chatbot).

## Pipecat Cloud Configuration

### 1. Get Your Organization Name

Retrieve your Pipecat Cloud organization name using the Pipecat CLI:

```bash  theme={null}
$ pipecat cloud organizations list
```

This command will output a list of organizations associated with your account. For example:

```bash  theme={null}
Organization        Name
──────────────────────────────────────
Default Workspace   three-random-words-randomnumber (active)
```

### 2. Create a TwiML Bin

[Purchase a phone number from Twilio](https://help.twilio.com/articles/223135247-How-to-Search-for-and-Buy-a-Twilio-Phone-Number-from-Console) if you haven't already, then create a [TwiML Bin](https://help.twilio.com/articles/360043489573-Getting-started-with-TwiML-Bins) with the following configuration:

```xml  theme={null}
<?xml version="1.0" encoding="UTF-8"?>
<Response>
  <Connect>
    <Stream url="wss://api.pipecat.daily.co/ws/twilio">
      <Parameter name="_pipecatCloudServiceHost"
         value="AGENT_NAME.ORGANIZATION_NAME"/>
    </Stream>
  </Connect>
</Response>
```

Replace the placeholder values:

* `AGENT_NAME` with your deployed bot's name (e.g., `my-first-agent`)
* `ORGANIZATION_NAME` with your organization name from step 1 (e.g., `three-random-words-randomnumber`)

For example, if your agent is named "customer-support" and your organization is "industrious-purple-cat-12345", your value would be: `customer-support.industrious-purple-cat-12345`

<Note>
  **Using Regional Endpoints**

  If you deployed your agent to a specific region, use the regional WebSocket endpoint to connect:

  `wss://{region}.api.pipecat.daily.co/ws/twilio`

  For example, for Europe: `wss://eu-central.api.pipecat.daily.co/ws/twilio`

  Learn more about [regional endpoints](/deployment/pipecat-cloud/guides/regions#regional-websocket-endpoints).
</Note>

### 3. Assign TwiML Bin to Your Phone Number

1. Navigate to the Phone Numbers section in your Twilio dashboard
2. Select your phone number from the list
3. In the Configure tab, under "Voice Configuration" section, find "A call comes in"
4. Set this dropdown to "TwiML Bin"
5. Select the TwiML Bin you created in step 2
6. Click Save to apply your changes

## Testing Your Integration

To test your integration, simply dial your Twilio phone number from any phone. The call will connect to your Pipecat Cloud bot, which will respond according to your bot's configuration.

## Next Steps

For complete implementation details including dial-out, custom parameters, caller personalization, and advanced call control features, see the [Twilio WebSocket Integration guide](/guides/telephony/twilio-websockets).

---

# Source: https://docs.pipecat.ai/guides/telephony/twilio-websockets.md

> ## Documentation Index
> Fetch the complete documentation index at: https://docs.pipecat.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Twilio WebSocket Integration

> Complete guide to using Twilio Media Streams with Pipecat for dial-in and dial-out functionality

## Things you'll need

* An active [Twilio](https://www.twilio.com) account with API credentials
* One or more Twilio provisioned phone numbers
* A tunneling service like [ngrok](https://ngrok.com/) for local development
* API keys for speech-to-text, text-to-speech, and LLM services

<CardGroup cols={2}>
  <Card title="Twilio Dial-in Example" icon="phone-arrow-down-left" href="https://github.com/pipecat-ai/pipecat-examples/tree/main/twilio-chatbot/inbound">
    Complete dial-in implementation using Twilio Media Streams over WebSocket
  </Card>

  <Card title="Twilio Dial-out Example" icon="phone-arrow-up-right" href="https://github.com/pipecat-ai/pipecat-examples/tree/main/twilio-chatbot/outbound">
    Outbound calling using Twilio Media Streams with WebSocket transport
  </Card>
</CardGroup>

## Phone Number Setup

You'll need Twilio phone numbers for both dial-in and dial-out functionality:

* Visit [console.twilio.com](https://console.twilio.com) and purchase phone numbers
* Ensure your numbers support Voice capabilities
* Configure webhook URLs for dial-in numbers (covered below)

## Environment Setup

Configure your environment variables for Twilio and AI services:

```shell .env theme={null}
TWILIO_ACCOUNT_SID=...
TWILIO_AUTH_TOKEN=...
OPENAI_API_KEY=...
DEEPGRAM_API_KEY=...
CARTESIA_API_KEY=...
```

## Dial-in

Dial-in allows users to call your Twilio number and connect to your Pipecat bot via WebSocket Media Streams.

### How It Works

Here's the sequence of events when someone calls your Twilio number:

1. **Twilio sends WebSocket messages**: Twilio processes the associated TwiML Bin and starts a WebSocket stream to your bot (local or Pipecat Cloud)
2. **Parse the WebSocket messages**: Your bot parses the WebSocket connection messages to set up the corresponding Pipecat transport
3. **(Optional) Look up the caller**: Optionally, look up the caller using Twilio's REST API to retrieve custom information about the call and personalize your bot's behavior
4. **Bot starts responding**: Once the pipeline is started, your bot will initiate the conversation

### Setting Up Twilio

#### 1. Create a TwiML Bin

A TwiML Bin tells Twilio how to handle incoming calls. You'll create one that establishes a WebSocket connection to your bot.

1. Go to the [Twilio Console](https://console.twilio.com)
2. Navigate to **TwiML Bins** → **My TwiML Bins**
3. Click the **+** to create a new TwiML Bin
4. Name your bin and add the TwiML:

<Tabs>
  <Tab title="Local Development">
    ```xml  theme={null}
    <?xml version="1.0" encoding="UTF-8"?>
    <Response>
      <Connect>
        <Stream url="wss://your-url.ngrok.io/ws" />
      </Connect>
    </Response>
    ```

    Replace `your-url.ngrok.io` with your ngrok URL.
  </Tab>

  <Tab title="Pipecat Cloud">
    ```xml  theme={null}
    <?xml version="1.0" encoding="UTF-8"?>
    <Response>
      <Connect>
        <Stream url="wss://api.pipecat.daily.co/ws/twilio">
          <Parameter name="_pipecatCloudServiceHost"
            value="AGENT_NAME.ORGANIZATION_NAME"/>
        </Stream>
      </Connect>
    </Response>
    ```

    Replace:

    * `AGENT_NAME` with the name of the agent you deployed to Pipecat Cloud
    * `ORGANIZATION_NAME` with the name of your Pipecat Cloud organization
  </Tab>
</Tabs>

5. Click **Save**

#### 2. Assign TwiML Bin to Your Phone Number

1. Navigate to **Phone Numbers** → **Manage** → **Active Numbers**
2. Click on your Twilio phone number
3. In the "Voice Configuration" section:
   * Set "A call comes in" to **TwiML Bin**
   * Select the TwiML Bin you created
4. Click **Save configuration**

### Running the Example

You'll need two terminal windows to run the example.

1. In your first terminal, start ngrok to expose your local server:

   ```shell  theme={null}
   ngrok http 7860
   ```

   > Tip: Use the `--subdomain` flag for a reusable ngrok URL.

2. In your second terminal, install dependencies:

   ```shell  theme={null}
   uv sync
   ```

3. Then run the bot:

   ```shell  theme={null}
   uv run bot.py -t twilio
   ```

4. Call your bot by placing a call to the number associated with your bot. The bot will answer and start the conversation.

### Personalizing Your Bot with Caller Information

When a call comes in, your bot receives a Call SID from Twilio. You can use this Call SID to fetch caller information from Twilio's REST API, including the caller's phone number. With this information, you can:

* Look up customer information in your database
* Personalize greetings and responses based on caller identity
* Route calls to different bot behaviors (e.g., VIP handling, support vs sales)
* Implement custom business logic based on the caller

The example bot demonstrates how to fetch caller information asynchronously using `aiohttp` to avoid blocking the event loop, ensuring your bot remains responsive even with multiple concurrent calls.

<Card title="Complete Dial-in Example" icon="code" href="https://github.com/pipecat-ai/pipecat-examples/tree/main/twilio-chatbot/inbound">
  See the full implementation with WebSocket parsing, REST API integration, and
  caller personalization
</Card>

## Dial-out

Dial-out allows your bot to initiate calls to phone numbers using Twilio's outbound calling capabilities with WebSocket Media Streams.

### How It Works

Here's the sequence of events for dial-out calls:

1. **Your application triggers a dial-out** (via API call or user action)
2. **Server initiates a Twilio call** to the target phone number
3. **Twilio establishes the call** and opens a WebSocket connection
4. **Your bot joins the WebSocket** and sets up the pipeline
5. **The recipient answers** and is connected to your bot
6. **The bot handles the conversation** with real-time audio streaming

### Set up your server for dial-out

The dial-out server creates outbound calls and manages WebSocket connections. When you trigger a dial-out call, your server:

1. **Receives the dial-out request** with target phone number and parameters
2. **Creates a Twilio call** using the REST API with TwiML that establishes WebSocket
3. **Accepts the WebSocket** connection from Twilio
4. **Parses call data** including any custom parameters
5. **Runs the Pipecat bot** with the call information

<Card title="Complete Server Implementation" icon="code" href="https://github.com/pipecat-ai/pipecat-examples/blob/main/twilio-chatbot/outbound/server.py">
  See the full FastAPI server code with outbound call creation and WebSocket
  handling
</Card>

### Configure your Pipecat bot for dial-out

The dial-out bot configuration is similar to dial-in, but you can pass custom parameters when creating the outbound call:

**Custom Parameters for Dial-out**: Pass information to your bot through TwiML parameters:

```python server.py theme={null}
# When creating the outbound call, include custom parameters
twiml = f"""
<Response>
  <Connect>
    <Stream url="wss://your-server.com/ws">
      <Parameter name="campaign_id" value="summer_sale" />
      <Parameter name="customer_id" value="{customer_id}" />
      <Parameter name="call_type" value="outbound" />
    </Stream>
  </Connect>
</Response>
"""

# Create the outbound call
call = twilio_client.calls.create(
    to=target_number,
    from_=your_twilio_number,
    twiml=twiml
)
```

**Bot Configuration**: The bot receives and uses these parameters:

```python bot.py theme={null}
# Parse WebSocket data (same as dial-in)
transport_type, call_data = await parse_telephony_websocket(websocket)

# Access custom parameters
custom_params = call_data["body"]
campaign_id = custom_params.get("campaign_id")
customer_id = custom_params.get("customer_id")

# Customize bot behavior for outbound calls
if campaign_id == "summer_sale":
    greeting = f"Hi! I'm calling about our summer sale promotion..."
```

<Card title="Complete Bot Implementation" icon="code" href="https://github.com/pipecat-ai/pipecat-examples/blob/main/twilio-chatbot/outbound/bot.py">
  See the full bot.py with outbound call handling and parameter usage
</Card>

### Run the Example

To test dial-out functionality:

1. **Start your server**: Run your FastAPI server
2. **Trigger a call**: Make an API request to start an outbound call
3. **Answer your phone**: The bot will call the specified number
4. **Talk to your bot**: Have a conversation with your AI agent

<Card title="Complete Setup Instructions" icon="book-open" href="https://github.com/pipecat-ai/pipecat-examples/tree/main/twilio-chatbot/outbound">
  See the full README with step-by-step setup, API usage, and outbound call
  configuration
</Card>

## Key Features

### Audio Format and Sample Rate

Twilio Media Streams uses 8kHz mono audio with 16-bit PCM encoding. To avoid resampling, set the audio input and output sample rate to 8000 Hz:

```python  theme={null}
task = PipelineTask(
    pipeline,
    params=PipelineParams(
        audio_in_sample_rate=8000,
        audio_out_sample_rate=8000,
    ),
)
```

### Automatic Call Termination

When you provide Twilio credentials to the `TwilioFrameSerializer`, it automatically ends calls when your pipeline ends:

```python  theme={null}
serializer = TwilioFrameSerializer(
    stream_sid=stream_id,
    call_sid=call_id,
    account_sid=os.getenv("TWILIO_ACCOUNT_SID"),
    auth_token=os.getenv("TWILIO_AUTH_TOKEN"),
)
```

## Deployment

### Pipecat Cloud Deployment

For production deployment without managing your own infrastructure, use Pipecat Cloud:

<Card title="Twilio WebSocket on Pipecat Cloud" icon="cloud" href="/deployment/pipecat-cloud/guides/telephony/twilio-websocket">
  Deploy Twilio WebSocket bots with automatic scaling and managed infrastructure
</Card>

### Self-Hosted Production Deployment

For fully self-hosted production deployment, ensure your servers are:

* Publicly accessible with HTTPS
* Able to handle concurrent WebSocket connections
* Properly configured with Twilio API credentials
* Implementing proper error handling and logging

## Next Steps

* Explore the [complete examples](https://github.com/pipecat-ai/pipecat-examples/tree/main/twilio-chatbot) for full implementations
* Learn about [Daily + Twilio SIP integration](./twilio-daily-sip) for advanced telephony scenarios
* Check out [Daily PSTN integration](./daily-pstn) for direct phone number provisioning

---

# Source: https://docs.pipecat.ai/server/services/serializers/twilio.md

> ## Documentation Index
> Fetch the complete documentation index at: https://docs.pipecat.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# TwilioFrameSerializer

> Serializer for Twilio Media Streams WebSocket protocol

## Overview

`TwilioFrameSerializer` enables integration with Twilio's Media Streams WebSocket protocol, allowing your Pipecat application to handle phone calls via Twilio's voice services with bidirectional audio conversion, DTMF event handling, and automatic call termination.

<CardGroup cols={2}>
  <Card title="Twilio Serializer API Reference" icon="code" href="https://reference-server.pipecat.ai/en/latest/api/pipecat.serializers.twilio.html">
    Pipecat's API methods for Twilio Media Streams integration
  </Card>

  <Card title="Example Implementation" icon="play" href="https://github.com/pipecat-ai/pipecat-examples/tree/main/twilio-chatbot">
    Complete telephony examples with Twilio
  </Card>

  <Card title="Twilio Documentation" icon="book" href="https://www.twilio.com/docs/voice/media-streams">
    Official Twilio Media Streams documentation
  </Card>

  <Card title="Twilio Console" icon="microphone" href="https://console.twilio.com/">
    Manage phone numbers and Media Stream configuration
  </Card>
</CardGroup>

## Installation

The `TwilioFrameSerializer` does not require any additional dependencies beyond the core Pipecat library:

```bash  theme={null}
pip install "pipecat-ai"
```

## Prerequisites

### Twilio Account Setup

Before using TwilioFrameSerializer, you need:

1. **Twilio Account**: Sign up at [Twilio Console](https://console.twilio.com/)
2. **Phone Number**: Purchase a Twilio phone number with voice capabilities
3. **Media Streams**: Configure your phone number to use Media Streams
4. **Webhook Configuration**: Set up webhook endpoints for call handling

### Required Environment Variables

* `TWILIO_ACCOUNT_SID`: Your Twilio Account SID for authentication
* `TWILIO_AUTH_TOKEN`: Your Twilio Auth Token for API operations

### Required Configuration

* **Stream SID**: Provided by Twilio during Media Stream connection
* **Call SID**: Required for automatic call termination (optional)

### Key Features

* **Bidirectional Audio**: Convert between Pipecat and Twilio audio formats
* **DTMF Handling**: Process touch-tone events from callers
* **Auto Hang-up**: Terminate calls via Twilio's REST API
* **μ-law Encoding**: Handle Twilio's standard audio encoding format

---

# Source: https://docs.pipecat.ai/server/services/s2s/ultravox.md

> ## Documentation Index
> Fetch the complete documentation index at: https://docs.pipecat.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Ultravox Realtime

> Real-time speech-to-speech service implementation using Ultravox's Realtime API

## Overview

`UltravoxRealtimeLLMService` provides real-time conversational AI capabilities using Ultravox's Realtime API. It supports both text and audio modalities with voice transcription, streaming responses, and tool usage for creating interactive AI experiences.

<CardGroup cols={2}>
  <Card title="Ultravox Realtime API Reference" icon="code" href="https://reference-server.pipecat.ai/en/latest/api/pipecat.services.ultravox.llm.html">
    Pipecat's API methods for Ultravox Realtime integration
  </Card>

  <Card title="Example Implementation" icon="play" href="https://github.com/pipecat-ai/pipecat/blob/main/examples/foundational/50-ultravox-realtime.py">
    Complete Ultravox Realtime conversation example
  </Card>

  <Card title="Ultravox Documentation" icon="book" href="https://docs.ultravox.ai/overview">
    Official Ultravox API documentation
  </Card>

  <Card title="Ultravox Console" icon="external-link" href="https://app.ultravox.ai/">
    Access Ultravox models and manage API keys
  </Card>
</CardGroup>

## Installation

To use Ultravox Realtime services, install the required dependencies:

```bash  theme={null}
pip install "pipecat-ai[ultravox]"
```

## Prerequisites

### Ultravox Account Setup

Before using Ultravox Realtime services, you need:

1. **Ultravox Account**: Sign up at [Ultravox Console](https://app.ultravox.ai/)
2. **API Key**: Generate an Ultravox API key from your account dashboard
3. **Model Access**: Ensure access to Ultravox Realtime models
4. **Usage Limits**: Configure appropriate usage limits and billing

### Required Environment Variables

* `ULTRAVOX_API_KEY`: Your Ultravox API key for authentication

### Key Features

* **Audio-Native Model**: Ultravox is an audio-native model for natural voice interactions
* **Real-time Streaming**: Low-latency audio processing and streaming responses
* **Multiple Input Modes**: Support for Agent, One-Shot, and Join URL input parameters
* **Voice Transcription**: Built-in transcription with streaming output
* **Function Calling**: Support for tool integration and API calling
* **Configurable Duration**: Set maximum call duration limits

---

# Source: https://docs.pipecat.ai/server/utilities/observers/user-bot-latency-observer.md

> ## Documentation Index
> Fetch the complete documentation index at: https://docs.pipecat.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# User-Bot Latency Observer

> Measure response time between user speech and bot responses in Pipecat

The `UserBotLatencyLogObserver` measures the time between when a user stops speaking and when the bot starts responding, providing metrics for conversational AI performance optimization.

## Features

* Tracks user speech start/stop timing
* Measures bot response latency
* Calculates statistics: average, minimum, maximum
* Provides real-time latency logging
* Automatically resets between conversation turns

## Usage

### Basic Latency Monitoring

Add latency monitoring to your pipeline:

```python  theme={null}
from pipecat.observers.loggers.user_bot_latency_log_observer import UserBotLatencyLogObserver

task = PipelineTask(
    pipeline,
    params=PipelineParams(observers=[UserBotLatencyLogObserver()]),
)
```

## How It Works

The observer tracks conversation flow through these key events:

1. **User starts speaking** → Resets latency tracking
2. **User stops speaking** → Records timestamp
3. **Bot starts speaking** → Calculates and logs latency
4. **Pipeline ends** → Reports session latency statistics

## Log Output

### Real-time Latency Logs

During conversation, each response latency is logged:

```
⏱️ LATENCY FROM USER STOPPED SPEAKING TO BOT STARTED SPEAKING: 1.234s
```

### Final Statistics

When the pipeline ends, comprehensive statistics are reported:

```
⏱️ LATENCY FROM USER STOPPED SPEAKING TO BOT STARTED SPEAKING - Avg: 1.456s, Min: 0.892s, Max: 2.103s
```

## Limitations

* Only measures speech-to-speech latency (not text processing time)
* Requires proper frame sequencing to work accurately

---

# Source: https://docs.pipecat.ai/server/utilities/user-idle-processor.md

> ## Documentation Index
> Fetch the complete documentation index at: https://docs.pipecat.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# UserIdleProcessor

> A processor that monitors user inactivity and triggers callbacks after specified timeout periods

<Warning>
  DEPRECATED: UserIdleProcessor has been deprecated. Use `user_idle_timeout`
  parameter when creating your aggregator, see [Detecting Idle
  Users](/guides/fundamentals/detecting-user-idle) for details.
</Warning>

The `UserIdleProcessor` is a specialized frame processor that monitors user activity in a conversation and executes callbacks when the user becomes idle. It's particularly useful for maintaining engagement by detecting periods of user inactivity and providing escalating responses to inactivity.

## Constructor Parameters

<ParamField path="callback" type="Union[BasicCallback, RetryCallback]" required>
  An async function that will be called when user inactivity is detected. Can be
  either:

  * Basic callback: `async def(processor: UserIdleProcessor) -> None`

  * Retry callback: `async def(processor: UserIdleProcessor, retry_count: int) ->
    bool` where returning `False` stops idle monitoring
</ParamField>

<ParamField path="timeout" type="float" required>
  The number of seconds to wait before considering the user idle.
</ParamField>

## Behavior

The processor starts monitoring for inactivity only after the first conversation activity (either `UserStartedSpeakingFrame` or `BotSpeakingFrame`). It manages idle state based on the following rules:

* Resets idle timer when user starts or stops speaking
* Pauses idle monitoring while user is speaking
* Resets idle timer when bot is speaking
* Stops monitoring on conversation end or cancellation
* Manages a retry count for the retry callback
* Stops monitoring when retry callback returns `False`

## Properties

<ParamField path="retry_count" type="int">
  The current number of retry attempts made to engage the user.
</ParamField>

## Example Implementations

Here are two example showing how to use the `UserIdleProcessor`: one with the basic callback and one with the retry callback:

<Tabs>
  <Tab title="Basic Callback">
    ```python  theme={null}
    from pipecat.frames.frames import LLMMessagesAppendFrame
    from pipecat.pipeline.pipeline import Pipeline
    from pipecat.processors.user_idle_processor import UserIdleProcessor

    async def handle_idle(user_idle: UserIdleProcessor) -> None:
        await user_idle.push_frame(
            LLMMessagesAppendFrame(
                [
                    {
                        "role": "system",
                        "content": "Ask the user if they are still there and try to prompt for some input.",
                    }
                ],
                run_llm=True,
            )
        )


    # Create the processor
    user_idle = UserIdleProcessor(callback=handle_idle, timeout=5.0)

    # Add to pipeline
    pipeline = Pipeline(
        [
            transport.input(),
            user_idle,  # Add the processor to monitor user activity
            context_aggregator.user(),
            # ... rest of pipeline
        ]
    )
    ```
  </Tab>

  <Tab title="Retry Callback">
    ```python  theme={null}
    from pipecat.frames.frames import EndFrame, LLMMessagesAppendFrame, TTSSpeakFrame
    from pipecat.pipeline.pipeline import Pipeline
    from pipecat.processors.user_idle_processor import UserIdleProcessor

    async def handle_user_idle(user_idle: UserIdleProcessor, retry_count: int) -> bool:
        if retry_count == 1:
            # First attempt: Gentle reminder
            await user_idle.push_frame(
                LLMMessagesAppendFrame(
                    [
                        {
                            "role": "system",
                            "content": "The user has been quiet. Politely and briefly ask if they're still there.",
                        }
                    ],
                    run_llm=True,
                )
            )
            return True
        elif retry_count == 2:
            # Second attempt: Direct prompt
            await user_idle.push_frame(
                LLMMessagesAppendFrame(
                    [
                        {
                            "role": "system",
                            "content": "The user is still inactive. Ask if they'd like to continue our conversation.",
                        }
                    ],
                    run_llm=True,
                )
            )
            return True
        else:
            # Third attempt: End conversation
            await user_idle.push_frame(
                TTSSpeakFrame("It seems like you're busy right now. Have a nice day!")
            )
            await task.queue_frame(EndFrame())
            return False  # Stop monitoring


    # Create the processor
    user_idle = UserIdleProcessor(callback=handle_user_idle, timeout=5.0)

    # Add to pipeline
    pipeline = Pipeline(
        [
            transport.input(),
            user_idle,  # Add the processor to monitor user activity
            context_aggregator.user(),
            # ... rest of pipeline
        ]
    )
    ```
  </Tab>
</Tabs>

## Frame Handling

The processor handles the following frame types:

* `UserStartedSpeakingFrame`: Marks user as active, resets idle timer and retry count
* `UserStoppedSpeakingFrame`: Starts idle monitoring
* `BotSpeakingFrame`: Resets idle timer
* `EndFrame` / `CancelFrame`: Stops idle monitoring

## Notes

* The idle callback won't be triggered while the user or bot is actively speaking
* The processor automatically cleans up its resources when the pipeline ends
* Basic callbacks are supported for backward compatibility

---

# Source: https://docs.pipecat.ai/guides/fundamentals/user-input-muting.md

> ## Documentation Index
> Fetch the complete documentation index at: https://docs.pipecat.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# User Input Muting

> Learn how to control when user speech is processed in your conversational bot

## Overview

In conversational applications, there are moments when you don't want to process user speech, such as during bot introductions or while executing function calls. Pipecat's user mute strategies let you selectively "mute" user input based on different conversation states.

## When to Use Mute Strategies

Common scenarios for muting user input include:

* **During introductions**: Prevent the bot from being interrupted during its initial greeting
* **While processing functions**: Block input while the bot is retrieving external data
* **During bot speech**: Reduce false transcriptions while the bot is speaking
* **For guided conversations**: Create more structured interactions with clear turn-taking

## How It Works

User mute strategies work by blocking specific user-related frames from flowing through your pipeline. When muted, the following frames are filtered:

* Voice activity detection (VAD) events
* Interruption signals
* Raw audio input frames
* Transcription frames (both interim and final)

This prevents user speech from being processed during muted periods.

<Note>
  Mute strategies are configured on the `LLMUserAggregator` via the
  `user_mute_strategies` parameter.
</Note>

## Mute Strategies

Pipecat provides several built-in strategies for determining when to mute user input:

<CardGroup cols={2}>
  <Card title="FirstSpeechUserMuteStrategy" icon="microphone-slash" iconType="duotone">
    Mute only during the bot's first speech utterance. Useful for introductions
    when you want the bot to complete its greeting before the user can speak.
  </Card>

  <Card title="MuteUntilFirstBotCompleteUserMuteStrategy" icon="hourglass-half" iconType="duotone">
    Start muted and remain muted until the first bot utterance completes.
    Ensures the bot's initial instructions are fully delivered.
  </Card>

  <Card title="FunctionCallUserMuteStrategy" icon="gear" iconType="duotone">
    Mute during function calls. Prevents users from speaking while the bot is
    processing external data requests.
  </Card>

  <Card title="AlwaysUserMuteStrategy" icon="volume-xmark" iconType="duotone">
    Mute whenever the bot is speaking. Creates a strict turn-taking conversation
    pattern.
  </Card>
</CardGroup>

<Warning>
  The `FirstSpeechUserMuteStrategy` and
  `MuteUntilFirstBotCompleteUserMuteStrategy` strategies should not be used
  together as they handle the first bot speech differently.
</Warning>

## Basic Implementation

Import and configure the mute strategies you need:

```python  theme={null}
from pipecat.processors.aggregators.llm_response_universal import (
    LLMContextAggregatorPair,
    LLMUserAggregatorParams,
)
from pipecat.turns.user_mute import AlwaysUserMuteStrategy

# Configure with one or more strategies
user_aggregator, assistant_aggregator = LLMContextAggregatorPair(
    context,
    user_params=LLMUserAggregatorParams(
        user_mute_strategies=[AlwaysUserMuteStrategy()],
    ),
)
```

## Combining Multiple Strategies

Multiple strategies can be combined. They use OR logic—if **any** strategy indicates the user should be muted, input is suppressed:

```python  theme={null}
from pipecat.processors.aggregators.llm_response_universal import (
    LLMContextAggregatorPair,
    LLMUserAggregatorParams,
)

from pipecat.turns.user_mute import (
    MuteUntilFirstBotCompleteUserMuteStrategy,
    FunctionCallUserMuteStrategy,
)

user_aggregator, assistant_aggregator = LLMContextAggregatorPair(
    context,
    user_params=LLMUserAggregatorParams(
        user_mute_strategies=[
            MuteUntilFirstBotCompleteUserMuteStrategy(),  # Mute until first response
            FunctionCallUserMuteStrategy(),               # Mute during function calls
        ],
    ),
)
```

## Responding to Mute Events

You can register event handlers to be notified when muting starts or stops. This is particularly useful for providing visual feedback to users:

```python  theme={null}
@user_aggregator.event_handler("on_user_mute_started")
async def on_user_mute_started(aggregator):
    logger.info("User mute started")
    # Send a visual indicator to your client
    # e.g., show a "Bot is speaking" indicator

@user_aggregator.event_handler("on_user_mute_stopped")
async def on_user_mute_stopped(aggregator):
    logger.info("User mute stopped")
    # Update your client UI
    # e.g., show a "You can speak now" indicator
```

These events fire whenever the mute state changes, allowing you to keep your UI synchronized with the bot's state.

## Best Practices

* **Choose strategies wisely**: Select the minimal set of strategies needed for your use case
* **Test user experience**: Excessive muting can frustrate users; balance control with usability
* **Provide feedback**: Use the mute event handlers to show visual cues when the user is muted to improve the experience

## Next Steps

<CardGroup cols={2}>
  <Card title="User Mute Strategies Reference" icon="book" iconType="duotone" href="/server/utilities/turn-management/user-mute-strategies">
    Read the complete API reference documentation for all available mute
    strategies and their behavior.
  </Card>

  <Card title="User Turn Strategies" icon="arrows-turn-to-dots" iconType="duotone" href="/server/utilities/turn-management/user-turn-strategies">
    Learn how to configure turn detection behavior for more control over
    conversation flow.
  </Card>
</CardGroup>

Experiment with different muting strategies to find the right balance for your application.

---

# Source: https://docs.pipecat.ai/server/utilities/turn-management/user-mute-strategies.md

> ## Documentation Index
> Fetch the complete documentation index at: https://docs.pipecat.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# User Mute Strategies

> Control when user input is suppressed during bot operations

## Overview

User mute strategies control whether incoming user input should be suppressed based on the current system state. They determine when user audio and transcriptions should be muted to prevent interruptions during critical bot operations like initial responses or function calls.

By default, user input is never muted. You can configure mute strategies to automatically suppress user input in specific scenarios, such as while the bot is speaking or during function execution. Custom strategies can also be implemented for specific use cases.

## Configuration

User mute strategies are configured via `LLMUserAggregatorParams` when creating an `LLMContextAggregatorPair`:

```python  theme={null}
from pipecat.processors.aggregators.llm_response_universal import (
    LLMContextAggregatorPair,
    LLMUserAggregatorParams,
)
from pipecat.turns.user_mute import (
    MuteUntilFirstBotCompleteUserMuteStrategy,
    FunctionCallUserMuteStrategy,
)

user_aggregator, assistant_aggregator = LLMContextAggregatorPair(
    context,
    user_params=LLMUserAggregatorParams(
        user_mute_strategies=[
            MuteUntilFirstBotCompleteUserMuteStrategy(),
            FunctionCallUserMuteStrategy(),
        ],
    ),
)
```

## Available Strategies

### AlwaysUserMuteStrategy

Mutes user input whenever the bot is speaking. This prevents any interruptions during bot speech.

```python  theme={null}
from pipecat.turns.user_mute import AlwaysUserMuteStrategy

strategy = AlwaysUserMuteStrategy()
```

**Behavior:**

* Mutes when `BotStartedSpeakingFrame` is received
* Unmutes when `BotStoppedSpeakingFrame` is received

### FirstSpeechUserMuteStrategy

Mutes user input only during the bot's first speech. After the initial response completes, user input is allowed even while the bot is speaking.

```python  theme={null}
from pipecat.turns.user_mute import FirstSpeechUserMuteStrategy

strategy = FirstSpeechUserMuteStrategy()
```

**Behavior:**

* Allows user input before bot speaks
* Mutes during the first bot speech only
* Unmutes permanently after first speech completes

<Note>
  Use this strategy when you want to ensure the bot's greeting or initial
  response isn't interrupted, but allow normal interruptions afterward.
</Note>

### MuteUntilFirstBotCompleteUserMuteStrategy

Mutes user input from the start of the interaction until the bot completes its first speech. This ensures the bot maintains full control at the beginning of a conversation.

```python  theme={null}
from pipecat.turns.user_mute import MuteUntilFirstBotCompleteUserMuteStrategy

strategy = MuteUntilFirstBotCompleteUserMuteStrategy()
```

**Behavior:**

* Mutes immediately when the pipeline starts (before bot speaks)
* Remains muted until first `BotStoppedSpeakingFrame` is received
* Unmutes permanently after first speech completes

<Note>
  Unlike `FirstSpeechUserMuteStrategy`, this strategy mutes user input even
  before the bot starts speaking. Use this when you don't want to process any
  user input until the bot has delivered its initial message.
</Note>

### FunctionCallUserMuteStrategy

Mutes user input while function calls are executing. This prevents user interruptions during potentially long-running tool operations.

```python  theme={null}
from pipecat.turns.user_mute import FunctionCallUserMuteStrategy

strategy = FunctionCallUserMuteStrategy()
```

**Behavior:**

* Mutes when `FunctionCallsStartedFrame` is received
* Tracks multiple concurrent function calls
* Unmutes when all function calls complete (via `FunctionCallResultFrame` or `FunctionCallCancelFrame`)

<Tip>
  This strategy is particularly useful when function calls trigger external API
  requests or database operations that may take several seconds to complete and
  you don't want to the user to interrupt the output.
</Tip>

## Combining Multiple Strategies

Multiple strategies can be combined in a list. The strategies are combined with OR logic—if **any** strategy indicates the user should be muted, user input is suppressed.

```python  theme={null}
user_aggregator, assistant_aggregator = LLMContextAggregatorPair(
    context,
    user_params=LLMUserAggregatorParams(
        user_mute_strategies=[
            MuteUntilFirstBotCompleteUserMuteStrategy(),  # Mute until first response
            FunctionCallUserMuteStrategy(),               # Mute during function calls
        ],
    ),
)
```

In this example, user input is muted:

* From pipeline start until the bot completes its first speech
* Whenever function calls are executing (even after first speech)

## Usage Examples

### Prevent Interruptions During Greeting

Ensure the bot's greeting plays completely before accepting user input:

```python  theme={null}
from pipecat.turns.user_mute import MuteUntilFirstBotCompleteUserMuteStrategy

user_aggregator, assistant_aggregator = LLMContextAggregatorPair(
    context,
    user_params=LLMUserAggregatorParams(
        user_mute_strategies=[
            MuteUntilFirstBotCompleteUserMuteStrategy(),
        ],
    ),
)
```

### Mute During Function Calls Only

Allow normal interruptions but prevent them during tool execution:

```python  theme={null}
from pipecat.turns.user_mute import FunctionCallUserMuteStrategy

user_aggregator, assistant_aggregator = LLMContextAggregatorPair(
    context,
    user_params=LLMUserAggregatorParams(
        user_mute_strategies=[
            FunctionCallUserMuteStrategy(),
        ],
    ),
)
```

### Never Allow Interruptions

Always mute user input while the bot is speaking:

```python  theme={null}
from pipecat.turns.user_mute import AlwaysUserMuteStrategy

user_aggregator, assistant_aggregator = LLMContextAggregatorPair(
    context,
    user_params=LLMUserAggregatorParams(
        user_mute_strategies=[
            AlwaysUserMuteStrategy(),
        ],
    ),
)
```

## Event Handlers

You can register event handlers to be notified when user muting starts or stops. This is useful for observability or providing feedback to users.

### Available Events

#### on\_user\_mute\_started

Called when user input becomes muted due to any active mute strategy.

```python  theme={null}
@user_aggregator.event_handler("on_user_mute_started")
async def on_user_mute_started(aggregator):
    logger.info("User mute started")
```

#### on\_user\_mute\_stopped

Called when user input is unmuted (no active mute strategies).

```python  theme={null}
@user_aggregator.event_handler("on_user_mute_stopped")
async def on_user_mute_stopped(aggregator):
    logger.info("User mute stopped")
```

<Note>
  These events fire whenever the mute state changes, regardless of which
  strategy triggered the change. Use them to provide consistent feedback across
  all mute scenarios.
</Note>

## Related

* [User Turn Strategies](/server/utilities/turn-management/user-turn-strategies) - Configure turn detection behavior
* [User Input Muting Guide](/guides/fundamentals/user-input-muting) - Guide for controlling user input

---

# Source: https://docs.pipecat.ai/server/utilities/turn-management/user-turn-strategies.md

> ## Documentation Index
> Fetch the complete documentation index at: https://docs.pipecat.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# User Turn Strategies

> Configure how user turns are detected and managed in conversations

## Overview

User turn strategies provide fine-grained control over how user speaking turns are detected in conversations. They determine when a user's turn starts (user begins speaking) and when it stops (user finishes speaking and expects a response).

By default, Pipecat uses a combination of VAD (Voice Activity Detection) and transcription-based detection:

* **Start**: VAD detection or transcription received
* **Stop**: Transcription received after VAD indicates silence

You can customize this behavior by providing your own strategies for more sophisticated turn detection, such as requiring a minimum number of words before triggering a turn, or using AI-powered turn detection models.

## How It Works

1. **Turn Start Detection**: When any start strategy triggers, the user aggregator:

   * Marks the start of a user turn
   * Optionally emits `UserStartedSpeakingFrame`
   * Optionally emits an interruption frame (if the bot is speaking)

2. **During User Turn**: The aggregator collects transcriptions and audio frames.

3. **Turn Stop Detection**: When a stop strategy triggers, the user aggregator:

   * Marks the end of the user turn
   * Emits `UserStoppedSpeakingFrame`
   * Pushes the aggregated user message to the LLM context

4. **Timeout Handling**: If no stop strategy triggers within `user_turn_stop_timeout` seconds (default: 5.0), the turn is automatically ended.

## Configuration

User turn strategies are configured via `LLMUserAggregatorParams` when creating an `LLMContextAggregatorPair`:

```python  theme={null}
from pipecat.processors.aggregators.llm_context import LLMContext
from pipecat.processors.aggregators.llm_response_universal import (
    LLMContextAggregatorPair,
    LLMUserAggregatorParams,
)
from pipecat.turns.user_turn_strategies import UserTurnStrategies

context = LLMContext(messages)
user_aggregator, assistant_aggregator = LLMContextAggregatorPair(
    context,
    user_params=LLMUserAggregatorParams(
        user_turn_strategies=UserTurnStrategies(
            start=[...],  # List of start strategies
            stop=[...],   # List of stop strategies
        ),
    ),
)
```

## Start Strategies

Start strategies determine when a user's turn begins. Multiple strategies can be provided, and the first one to trigger will signal the start of a user turn.

### Base Parameters

All start strategies inherit these parameters:

<ParamField path="enable_interruptions" type="bool" default="True">
  If True, the user aggregator will emit an interruption frame when the user
  turn starts, allowing the user to interrupt the bot.
</ParamField>

<ParamField path="enable_user_speaking_frames" type="bool" default="True">
  If True, the user aggregator will emit frames indicating when the user starts
  speaking. Disable this if another component (e.g., an STT service) already
  generates these frames.
</ParamField>

### VADUserTurnStartStrategy

Triggers a user turn start based on Voice Activity Detection. This is the most responsive strategy, detecting speech as soon as the VAD indicates the user has started speaking.

```python  theme={null}
from pipecat.turns.user_start import VADUserTurnStartStrategy

strategy = VADUserTurnStartStrategy()
```

### TranscriptionUserTurnStartStrategy

Triggers a user turn start when a transcription is received. This serves as a fallback for scenarios where VAD-based detection fails (e.g., when the user speaks very softly) but the STT service still produces transcriptions.

<ParamField path="use_interim" type="bool" default="True">
  Whether to trigger on interim (partial) transcription frames for earlier
  detection.
</ParamField>

```python  theme={null}
from pipecat.turns.user_start import TranscriptionUserTurnStartStrategy

strategy = TranscriptionUserTurnStartStrategy(use_interim=True)
```

### MinWordsUserTurnStartStrategy

Requires the user to speak a minimum number of words before triggering a turn start. This is useful for preventing brief utterances like "okay" or "yeah" from triggering responses.

<ParamField path="min_words" type="int" required>
  Minimum number of spoken words required to trigger the start of a user turn.
</ParamField>

<ParamField path="use_interim" type="bool" default="True">
  Whether to consider interim transcription frames for earlier detection.
</ParamField>

```python  theme={null}
from pipecat.turns.user_start import MinWordsUserTurnStartStrategy

# Require at least 3 words to start a turn
strategy = MinWordsUserTurnStartStrategy(min_words=3)
```

<Note>
  When the bot is not speaking, this strategy will trigger after just 1 word.
  The `min_words` threshold only applies when the bot is actively speaking,
  preventing short affirmations from interrupting the bot.
</Note>

### ExternalUserTurnStartStrategy

Delegates turn start detection to an external processor. This strategy listens for `UserStartedSpeakingFrame` frames emitted by other components in the pipeline (such as speech-to-speech services).

```python  theme={null}
from pipecat.turns.user_start import ExternalUserTurnStartStrategy

strategy = ExternalUserTurnStartStrategy()
```

<Note>
  This strategy automatically sets `enable_interruptions=False` and
  `enable_user_speaking_frames=False` since these are expected to be handled by
  the external processor.
</Note>

## Stop Strategies

Stop strategies determine when a user's turn ends and the bot should respond.

### Base Parameters

All stop strategies inherit these parameters:

<ParamField path="enable_user_speaking_frames" type="bool" default="True">
  If True, the aggregator will emit frames indicating when the user stops
  speaking. Disable this if another component already generates these frames.
</ParamField>

### TranscriptionUserTurnStopStrategy

The default stop strategy that signals the end of a user turn when transcription is received and VAD indicates silence.

<ParamField path="timeout" type="float" default="0.5">
  A short delay in seconds used to handle consecutive or slightly delayed
  transcriptions gracefully.
</ParamField>

```python  theme={null}
from pipecat.turns.user_stop import TranscriptionUserTurnStopStrategy

strategy = TranscriptionUserTurnStopStrategy(timeout=0.5)
```

### TurnAnalyzerUserTurnStopStrategy

Uses an AI-powered turn detection model to determine when the user has finished speaking. This provides more intelligent end-of-turn detection that can understand conversational context.

<ParamField path="turn_analyzer" type="BaseTurnAnalyzer" required>
  The turn detection analyzer instance to use for end-of-turn detection.
</ParamField>

<ParamField path="timeout" type="float" default="0.5">
  A short delay in seconds used to handle consecutive or slightly delayed
  transcriptions.
</ParamField>

```python  theme={null}
from pipecat.audio.turn.smart_turn.local_smart_turn_v3 import LocalSmartTurnAnalyzerV3
from pipecat.turns.user_stop import TurnAnalyzerUserTurnStopStrategy

strategy = TurnAnalyzerUserTurnStopStrategy(
    turn_analyzer=LocalSmartTurnAnalyzerV3()
)
```

<Tip>
  See the [Smart Turn
  Detection](/server/utilities/smart-turn/smart-turn-overview) documentation for
  more information on available turn analyzers.
</Tip>

### ExternalUserTurnStopStrategy

Delegates turn stop detection to an external processor. This strategy listens for `UserStoppedSpeakingFrame` frames emitted by other components in the pipeline.

<ParamField path="timeout" type="float" default="0.5">
  A short delay in seconds used to handle consecutive or slightly delayed
  transcriptions.
</ParamField>

```python  theme={null}
from pipecat.turns.user_stop import ExternalUserTurnStopStrategy

strategy = ExternalUserTurnStopStrategy()
```

## UserTurnStrategies

Container for configuring user turn start and stop strategies.

<ParamField path="start" type="List[BaseUserTurnStartStrategy]" default="[VADUser...(), TranscriptionUser...()]">
  List of strategies used to detect when the user starts speaking. The first
  strategy to trigger will signal the start of the user's turn.
</ParamField>

<ParamField path="stop" type="List[BaseUserTurnStopStrategy]" default="[TranscriptionUserTurnStopStrategy()]">
  List of strategies used to detect when the user stops speaking and expects a
  response.
</ParamField>

## ExternalUserTurnStrategies

A convenience class that preconfigures `UserTurnStrategies` with external strategies for both start and stop detection. Use this when an external processor (such as a speech-to-speech service) controls turn management.

```python  theme={null}
from pipecat.turns.user_turn_strategies import ExternalUserTurnStrategies

user_aggregator, assistant_aggregator = LLMContextAggregatorPair(
    context,
    user_params=LLMUserAggregatorParams(
        user_turn_strategies=ExternalUserTurnStrategies(),
    ),
)
```

## Usage Examples

### Default Behavior

The default configuration uses VAD and transcription for turn detection:

```python  theme={null}
from pipecat.turns.user_turn_strategies import UserTurnStrategies

# This is equivalent to the default behavior
strategies = UserTurnStrategies(
    start=[VADUserTurnStartStrategy(), TranscriptionUserTurnStartStrategy()],
    stop=[TranscriptionUserTurnStopStrategy()],
)
```

### Minimum Words for Interruption

Require users to speak at least 3 words before they can interrupt the bot:

```python  theme={null}
from pipecat.turns.user_start import MinWordsUserTurnStartStrategy
from pipecat.turns.user_stop import TranscriptionUserTurnStopStrategy
from pipecat.turns.user_turn_strategies import UserTurnStrategies

user_aggregator, assistant_aggregator = LLMContextAggregatorPair(
    context,
    user_params=LLMUserAggregatorParams(
        user_turn_strategies=UserTurnStrategies(
            start=[MinWordsUserTurnStartStrategy(min_words=3)],
            stop=[TranscriptionUserTurnStopStrategy()],
        ),
    ),
)
```

### Local Smart Turn Detection

Use a local turn detection model instead of a cloud service:

```python  theme={null}
from pipecat.audio.turn.smart_turn.local_smart_turn_v3 import LocalSmartTurnAnalyzerV3
from pipecat.turns.user_stop import TurnAnalyzerUserTurnStopStrategy
from pipecat.turns.user_turn_strategies import UserTurnStrategies

user_aggregator, assistant_aggregator = LLMContextAggregatorPair(
    context,
    user_params=LLMUserAggregatorParams(
        user_turn_strategies=UserTurnStrategies(
            stop=[
                TurnAnalyzerUserTurnStopStrategy(
                    turn_analyzer=LocalSmartTurnAnalyzerV3()
                )
            ]
        ),
    ),
)
```

## Related

* [User Input Muting](/guides/fundamentals/user-input-muting) - Control when user input is ignored
* [Smart Turn Detection](/server/utilities/smart-turn/smart-turn-overview) - AI-powered turn detection

---

# Source: https://docs.pipecat.ai/deployment/pipecat-cloud/guides/using-datadog.md

> ## Documentation Index
> Fetch the complete documentation index at: https://docs.pipecat.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Using Datadog

> How to add Datadog logging, traces, and custom metrics to your Pipecat Cloud agents

This guide explains how to use [Datadog](https://www.datadoghq.com/) with your Pipecat Cloud agents for logging, traces, and custom metrics.

<Warning>
  This guide is a work in progress and may have some sharp edges. However, it
  has been vetted for basic usage.
</Warning>

## Prerequisites

* Have a Datadog account and API key.
* Use at least version `0.0.7` of the Pipecat Cloud base image:

```dockerfile  theme={null}
# dailyco/pipecat-base:latest should also work
FROM dailyco/pipecat-base:0.0.7
```

## Add Datadog API key as a secret

If you're using keeping your secrets in an `.env` file [as recommended](../fundamentals/secrets), add your Datadog API key to that file:

```bash  theme={null}
DD_API_KEY=<your-datadog-api-key>
```

Then update your secret set from that file:

```bash  theme={null}
pipecat cloud secrets set <your-agent-secrets> --file .env
```

See [Secrets](../fundamentals/secrets) for more details.

## Create `datadog.yaml`

This file contains the Datadog Agent configuration.

The configuration shown below assumes you want all Datadog features covered in this guide (logging, traces, and custom metrics).

```yaml  theme={null}
site: <your-datadog-site> # e.g. us5.datadoghq.com
hostname: pipecat.daily.co # (or whatever would be useful for you to see in the Datadog dashboards)

# Include the following line only if you need logging (see "Enabling Logging")
logs_enabled: true

# Uncomment the below lines if you DON'T need traces (see "Enabling Traces")
# apm_config:
#   enabled: false

# Uncomment the below lines if you DON'T need custom metrics (see "Enabling Custom Metrics")
# use_dogstatsd: false

kubelet_tls_verify: false
autoconfig_exclude_features:
  - kubernetes
  - orchestratorexplorer

```

## Create `pre-app.sh`

This script runs before the agent launcher runs.

The file shown below assumes you want all Datadog features covered in this guide (logging, traces, and custom metrics).

```bash  theme={null}
#!/bin/bash

# Write the Datadog API key to the config file
echo "api_key: $DD_API_KEY" >> /etc/datadog-agent/datadog.yaml

# Start the Datadog Agent service
service datadog-agent start || true

# Include the below line only if you need traces (see "Enabling Traces")
# HACK: here we restart the Datadog Trace Agent specifically, since we know it failed to start above
service datadog-agent-trace restart || true

## DEBUGGING

# Check the status of the Datadog Agent subsystems for logging, custom metrics, and traces
# service datadog-agent status
# service datadog-agent-trace status
# datadog-agent status

# Display the Datadog agent log
# cat /var/log/datadog/agent.log
```

<Warning>
  You may see `Starting Datadog Agent: datadog-agent failed!` in your logs,
  triggered by `service datadog-agent start`. You can safely ignore this! As
  long as `service datadog-agent status` and `service datadog-agent-trace
    status` indicate that both of those services end up running, then we should
  have no trouble sending logs, traces, and custom metrics to Datadog.
</Warning>

## Update `Dockerfile` with the Datadog Agent

The snippet shown below assumes you want all Datadog features covered in this guide (logging, traces, and custom metrics).

```dockerfile  theme={null}
# Install the Datadog Agent
# (note that you can literally use "dummy"—or anything else—as the API key)
RUN apt-get -y update; apt-get -y install curl
RUN DD_API_KEY=dummy \
    DD_INSTALL_ONLY=true \
    bash -c "$(curl -L https://install.datadoghq.com/scripts/install_script_agent7.sh)"

# Configure Datadog Agent
COPY ./datadog.yaml /etc/datadog-agent
# Include the below line if you need logging (see "Enabling Logging")
COPY ./python.d /etc/datadog-agent/conf.d/python.d

# Include the below lines only if you need traces or custom metrics (see "Enabling Traces" or "Enabling Custom Metrics")
# Prepare file system for the Datadog Agent to use for Unix Sockets
RUN mkdir -p /var/run/datadog/
RUN chown dd-agent:dd-agent /var/run/datadog/

# Start Datadog Agent (move pre-app script to expected place where it'll be invoked by Pipecat Cloud base image)
COPY --chmod=755 ./pre-app.sh /app/
```

## Enabling logging

### Enable logging in `datadog.yaml`

Ensure the following is present in your [`datadog.yaml`](#create-datadog-yaml):

```yaml  theme={null}
logs_enabled: true
```

### Create `python.d/conf.yaml`

Configure log collection for your agent:

```yaml  theme={null}
# Use whatever agent name would be useful for you to see in the Datadog dashboard
logs:
  - type: file
    path: /var/log/<my-agent-name>/datadog.log
    service: <my-agent-name>
    source: python
    sourcecategory: sourcecode
```

### Update `Dockerfile` for logging

Ensure the following is present in your [`Dockerfile`](#update-dockerfile-with-the-datadog-agent):

```dockerfile  theme={null}
COPY ./python.d /etc/datadog-agent/conf.d/python.d
```

This configures the Datadog Agent to collect logs from your agent.

### Configure `logger` in `bot.py`

Add a Datadog sink to your logger.

This sink will write log lines into a file that your Datadog Agent will read from, in a Datadog-friendly format.

```python  theme={null}
import json

# Format log lines so that:
# - Datadog can parse and display log levels, messages, and timestamps
# - You can search Datadog logs by session_id
def datadog_format(record):
    format_string_safe_message = (
        json.dumps(record["message"]).replace("{", "{{").replace("}", "}}")
    )
    session_id = record["extra"].get("session_id", "")
    return (
        "{{"
        '"timestamp":"{time:YYYY-MM-DD HH:mm:ss.SSS}",'
        '"levelname":"{level}",'
        '"name":"{name}",'
        '"function":"{function}",'
        '"line":{line},'
        f'"message":{format_string_safe_message},'
        f'"session_id": "{session_id}"'
        "}}\n"
    )

async def bot(args: DailySessionArguments):
    # Add Datadog-specific sink
    logger.add(
        "/var/log/<my-agent-name>/datadog.log", # Use same path as in python.d/conf.yaml
        rotation="10 MB",
        retention="7 days",
        format=datadog_format,
    )
    # ...
```

## Enabling traces

### Enable traces in `datadog.yaml`

Ensure `apm_config.enabled` is NOT set to `false` in your [`datadog.yaml`](#create-datadog-yaml):

```yaml  theme={null}
# DELETE or COMMENT OUT the following lines, if they're present
# apm_config:
#  enabled: false
```

### Start the Datadog Trace Agent in `pre-app.sh`

Ensure the following is present in your [`pre-app.sh`](#create-pre-app-sh):

```bash  theme={null}
service datadog-agent-trace restart || true
```

### Prepare file system in `Dockerfile`

Ensure the following is present in your [`Dockerfile`](#update-dockerfile-with-the-datadog-agent):

```dockerfile  theme={null}
RUN mkdir -p /var/run/datadog/
RUN chown dd-agent:dd-agent /var/run/datadog/
```

### Add `ddtrace` Python library

In your `requirements.txt`:

```
ddtrace
```

### Record traces in `bot.py`

```python  theme={null}
from ddtrace import tracer

@tracer.wrap()
def my_function_call():
    # ...
```

## Enabling custom metrics

### Enable custom metrics in `datadog.yaml`

Ensure `use_dogstatsd` is NOT set to `false` in your [`datadog.yaml`](#create-datadog-yaml):

```yaml  theme={null}
# DELETE or COMMENT OUT the following line, if it's present
# use_dogstatsd: false
```

### Prepare file system in `Dockerfile`

Ensure the following is present in your [`Dockerfile`](#update-dockerfile-with-the-datadog-agent):

```dockerfile  theme={null}
RUN mkdir -p /var/run/datadog/
RUN chown dd-agent:dd-agent /var/run/datadog/
```

### Add `datadog` Python library

In your `requirements.txt`:

```
datadog
```

### Record custom metrics in `bot.py`

```python  theme={null}
from datadog import statsd

def my_function_call():
    statsd.increment("myagent.my_function_call")
```

---

# Source: https://docs.pipecat.ai/server/utilities/extensions/voicemail.md

# Source: https://docs.pipecat.ai/guides/fundamentals/voicemail.md

> ## Documentation Index
> Fetch the complete documentation index at: https://docs.pipecat.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Voicemail Detection

> Automatically classify outbound calls as conversation or voicemail and respond appropriately

## Overview

The VoicemailDetector classifies incoming communication as either live conversation or voicemail systems. This module is built primarily for voice AI bots that perform outbound calling, enabling them to respond appropriately based on whether a human answered or the call went to voicemail.

The detector is optimized for fast conversation response times, where TTS output is generated immediately but held in a gate until the classification decision is made. This ensures minimal latency for live conversations while preventing inappropriate responses to voicemail systems.

## How It Works

The VoicemailDetector uses a parallel pipeline architecture to perform real-time classification without interrupting conversation flow. It analyzes the initial response from the called party and determines whether it's a human greeting or an automated voicemail system.

Key features:

* **Real-time classification** - Determines conversation vs voicemail as soon as audio is received
* **TTS gating** - Holds generated audio until classification is complete
* **Event-driven** - Triggers custom handlers when voicemail is detected
* **Configurable timing** - Adjustable delay for voicemail response timing

## Basic Setup

### 1. Initialize the Detector

```python  theme={null}
from pipecat.extensions.voicemail.voicemail_detector import VoicemailDetector
from pipecat.services.openai.llm import OpenAILLMService

# Create an LLM for classification
classifier_llm = OpenAILLMService(api_key=os.getenv("OPENAI_API_KEY"))

# Initialize the detector
voicemail_detector = VoicemailDetector(
    llm=classifier_llm,
    voicemail_response_delay=2.0  # Default: 2 seconds
)
```

<Note>
  The VoicemailDetector works with two LLMs: your main conversation LLM (which
  must be text-based) and the classifier LLM (which can be either text-based or
  realtime with a `text` output modality). Realtime LLMs are not compatible for
  the main conversation LLM due to output control requirements.
</Note>

### 2. Configure the Pipeline

The VoicemailDetector requires two components in your pipeline:

* `detector()`: between STT and user context aggregator
* `gate()`: immediately after TTS service

```python  theme={null}
pipeline = Pipeline([
    transport.input(),
    stt,
    voicemail_detector.detector(),    # Between STT and user context aggregator
    context_aggregator.user(),
    llm,
    tts,
    voicemail_detector.gate(),        # Immediately after TTS service
    transport.output(),
    context_aggregator.assistant(),
])
```

### 3. Handle Voicemail Events

When a voicemail is detected, the `on_voicemail_detected` event is triggered. In your event handler, you have access to `processor`, which is a FrameProcessor instance, allowing you to push frames to the pipeline. For example, you may want to have the bot output a pre-canned message and then end the call.

```python  theme={null}
@voicemail_detector.event_handler("on_voicemail_detected")
async def handle_voicemail(processor):
    logger.info("Voicemail detected! Leaving a message...")

    # Leave a voicemail message
    await processor.push_frame(
        TTSSpeakFrame("Hello, this is Jamie calling about your appointment. Please call me back at 555-0123.")
    )

    # Optionally end the call after leaving the message
    await processor.push_frame(EndTaskFrame(), FrameDirection.UPSTREAM)
```

## Detecting a Conversation

When a conversation is detected, no additional processing is required by the VoicemailDetector. Once the VoicemailDetector classifies the call as a conversation, the TTSGate will be flushed, allowing the conversation to continue.

## Configuration Options

### Custom System Prompt

For specialized use cases, you can provide a custom classification prompt. The prompt must instruct the LLM to respond with exactly "CONVERSATION" or "VOICEMAIL":

```python  theme={null}
custom_prompt = """
Your custom classification logic here.
Consider factors like business hours, call patterns, etc.
""" + VoicemailDetector.CLASSIFIER_RESPONSE_INSTRUCTION

voicemail_detector = VoicemailDetector(
    llm=classifier_llm,
    custom_system_prompt=custom_prompt
)
```

<Note>
  The `CLASSIFIER_RESPONSE_INSTRUCTION` constant contains the required response
  format: `'Respond with ONLY "CONVERSATION" if a person answered, or
    "VOICEMAIL" if it\'s voicemail/recording.'`
</Note>

For most use cases, the default classifier prompt will work effectively. If you need to customize the behavior, reference the built-in prompt as a starting point and modify from there.

### Response Timing

The `voicemail_response_delay` parameter controls how long to wait after the user stops speaking before triggering the voicemail event:

```python  theme={null}
voicemail_detector = VoicemailDetector(
    llm=classifier_llm,
    voicemail_response_delay=3.0  # Wait 3 seconds instead of default 2
)
```

This delay ensures:

* Voicemail greetings have finished playing
* The response occurs during the recording period
* Proper timing for different voicemail system behaviors

---

# Source: https://docs.pipecat.ai/server/services/serializers/vonage.md

> ## Documentation Index
> Fetch the complete documentation index at: https://docs.pipecat.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# VonageFrameSerializer

> Serializer for Vonage Video API Audio Connector WebSocket protocol

## Overview

`VonageFrameSerializer` enables integration with the Vonage Video API Audio Connector WebSocket protocol, allowing Pipecat applications to process real-time audio streams from active Vonage video sessions.

<CardGroup cols={2}>
  <Card title="Vonage Serializer API Reference" icon="code" href="https://reference-server.pipecat.ai/en/latest/api/pipecat.serializers.vonage.html">
    Pipecat's API methods for Vonage Audio Connector Streams integration
  </Card>

  <Card title="Example Implementation" icon="play" href="https://github.com/pipecat-ai/pipecat-examples/tree/main/vonage-audio-bot">
    End-to-end Pipecat example using Vonage Audio Connector
  </Card>

  <Card title="Vonage Audio Connector Documentation" icon="book" href="https://developer.vonage.com/en/video/guides/audio-connector">
    Official Vonage Video API Audio Connector documentation
  </Card>

  <Card title="Vonage Video API Console" icon="microphone" href="https://tokbox.com/account/">
    Manage Vonage Video API projects
  </Card>
</CardGroup>

## Installation

The `VonageFrameSerializer` does not require any additional dependencies beyond the core Pipecat library:

```bash  theme={null}
pip install "pipecat-ai"
```

## Prerequisites

### Vonage Video API Account Setup

Before using VonageFrameSerializer, you need:

1. **Vonage (TokBox) Account**: Sign up at [Vonage Video API Console](https://tokbox.com/account/)
2. **Vonage Video API Project**: Create a project to obtain Project API Key and Project Secret
3. **Existing Vonage Video Session**: A Vonage session must already exist. Sessions can be created using TokBox Playground or Vonage Video API SDKs

### Required Environment Variables

* `VONAGE_API_KEY`: Your Vonage Video API project key
* `VONAGE_API_SECRET`: Your Vonage Video API project secret
* `VONAGE_SESSION_ID`: The existing routed session ID
* `WS_URI`: Public WebSocket endpoint URI of the server application running Pipecat (e.g. via ngrok)

### Required Configuration

* **WebSocket Endpoint (/ws)**: A WebSocket server application (e.g. FastAPI) running Pipecat that accepts raw PCM audio frames.
* **Audio Connector /connect Request**: Triggers Vonage to open a WebSocket connection to your server and begin streaming audio from the active session.

### Key Features

* **Bidirectional Audio**: Convert between Pipecat and Vonage Audio Connector formats
* **Real-Time AI Pipelines**: Stream live audio into Pipecat and process it through any real-time pipeline configuration supported by the framework
* **Session Control Events**: Handle Vonage Audio Connector JSON events
* **Linear PCM Audio**: Handle raw 16-bit linear PCM audio streams used by the Vonage Video API Audio Connector

---

# Source: https://docs.pipecat.ai/server/utilities/filters/wake-check-filter.md

> ## Documentation Index
> Fetch the complete documentation index at: https://docs.pipecat.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# WakeCheckFilter

> Processor that passes frames only after detecting wake phrases in transcriptions

## Overview

`WakeCheckFilter` monitors `TranscriptionFrame`s for specified wake phrases and only allows frames to pass through after a wake phrase has been detected. It includes a keepalive timeout to maintain the awake state for a period after detection, allowing continuous conversation without requiring repeated wake phrases.

## Constructor Parameters

<ParamField path="wake_phrases" type="list[str]" required>
  List of wake phrases to detect in transcriptions
</ParamField>

<ParamField path="keepalive_timeout" type="float" default="3">
  Number of seconds to remain in the awake state after each transcription
</ParamField>

## Functionality

The filter maintains state for each participant and processes frames as follows:

1. `TranscriptionFrame` objects are checked for wake phrases
2. If a wake phrase is detected, the filter enters the "AWAKE" state
3. While in the "AWAKE" state, all transcription frames pass through
4. After no activity for the keepalive timeout period, the filter returns to "IDLE"
5. All non-transcription frames pass through normally

Wake phrases are detected using regular expressions that match whole words with flexible spacing, making detection resilient to minor transcription variations.

## States

<ParamField path="IDLE" type="WakeState">
  Default state - only non-transcription frames pass through
</ParamField>

<ParamField path="AWAKE" type="WakeState">
  Active state after wake phrase detection - all frames pass through
</ParamField>

## Output Frames

* All non-transcription frames pass through unchanged
* After wake phrase detection, transcription frames pass through
* When awake, transcription frames reset the keepalive timer

## Usage Example

```python  theme={null}
from pipecat.processors.filters import WakeCheckFilter

# Create filter with wake phrases
wake_filter = WakeCheckFilter(
    wake_phrases=["hey assistant", "ok computer", "listen up"],
    keepalive_timeout=5.0  # Stay awake for 5 seconds after each transcription
)

# Add to pipeline
pipeline = Pipeline([
    transport.input(),
    stt_service,
    wake_filter,  # Only passes transcriptions after wake phrases
    llm_service,
    tts_service,
    transport.output()
])
```

## Frame Flow

```mermaid  theme={null}
graph TD
    A[Input Frames] --> B[WakeCheckFilter]
    B --> C{Transcription Frame?}
    C -->|No| F[Output Frame]
    C -->|Yes| D{Wake State}
    D -->|AWAKE| E{Keepalive Expired?}
    E -->|No| F
    E -->|Yes| G[Return to IDLE]
    D -->|IDLE| H{Contains Wake Phrase?}
    H -->|Yes| I[Set AWAKE] --> F
    H -->|No| J[Filtered Out]
```

## Notes

* Maintains separate state for each participant ID
* Uses regex pattern matching for resilient wake phrase detection
* Accumulates transcription text to detect phrases across multiple frames
* Trims accumulated text when wake phrase is detected
* Supports multiple wake phrases
* Passes all non-transcription frames through unchanged
* Error handling produces ErrorFrames for robust operation
* Case-insensitive matching for natural language use

---

# Source: https://docs.pipecat.ai/server/utilities/filters/wake-notifier-filter.md

> ## Documentation Index
> Fetch the complete documentation index at: https://docs.pipecat.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# WakeNotifierFilter

> Processor that triggers a notifier when specified frame types pass a custom filter

## Overview

`WakeNotifierFilter` monitors the pipeline for specific frame types and triggers a notification when those frames pass a custom filter condition. It passes all frames through unchanged while performing this notification side-effect.

## Constructor Parameters

<ParamField path="notifier" type="BaseNotifier" required>
  The notifier object to trigger when conditions are met
</ParamField>

<ParamField path="types" type="Tuple[Type[Frame]]" required>
  Tuple of frame types to monitor
</ParamField>

<ParamField path="filter" type="Callable[[Frame], Awaitable[bool]]" required>
  Async function that examines each matching frame and returns True to trigger
  notification
</ParamField>

## Functionality

The processor operates as follows:

1. Checks if the incoming frame matches any of the specified types
2. If it's a matching type, calls the filter function with the frame
3. If the filter returns True, triggers the notifier
4. Passes all frames through unchanged, regardless of the filtering result

This allows for notification side-effects without modifying the pipeline's data flow.

## Output Frames

* All frames pass through unchanged in their original direction
* No frames are modified or filtered out

## Usage Example

```python  theme={null}
from pipecat.frames.frames import TranscriptionFrame, UserStartedSpeakingFrame
from pipecat.processors.filters import WakeNotifierFilter
from pipecat.sync.event_notifier import EventNotifier

# Create an event notifier
wake_event = EventNotifier()

# Create filter that notifies when certain wake phrases are detected
async def wake_phrase_filter(frame):
    if isinstance(frame, TranscriptionFrame):
        return "hey assistant" in frame.text.lower()
    return False

# Add to pipeline
wake_notifier = WakeNotifierFilter(
    notifier=wake_event,
    types=(TranscriptionFrame, UserStartedSpeakingFrame),
    filter=wake_phrase_filter
)

# In another component, wait for the notification
async def handle_wake_event():
    await wake_event.wait()
    print("Wake phrase detected!")
```

## Frame Flow

```mermaid  theme={null}
graph TD
    A[Input Frame] --> B[WakeNotifierFilter]
    B --> C{Matching Type?}
    C -->|Yes| D{Filter Function}
    D -->|Returns True| E[Notify]
    D -->|Returns False| F[Pass Through]
    C -->|No| F
    E --> F
```

## Notes

* Acts as a transparent pass-through for all frames
* Can trigger external events without modifying pipeline flow
* Useful for signaling between pipeline components
* Can monitor for multiple frame types simultaneously
* Uses async filter function for complex conditions
* Functions as a "listener" that doesn't affect the data stream
* Can be used for logging, analytics, or coordinating external systems

---

# Source: https://docs.pipecat.ai/server/services/transport/websocket-server.md

> ## Documentation Index
> Fetch the complete documentation index at: https://docs.pipecat.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# WebSocket Transports

> WebSocket transport implementations for real-time client-server communication

## Overview

WebSocket transports provide both client and server WebSocket implementations for real-time bidirectional communication. These transports support audio streaming, frame serialization, and connection management, making them ideal for prototyping and lightweight client-server applications where WebRTC might be overkill.

<Warning>
  WebSocket transports are best suited for prototyping and controlled network environments.

  For production client-server applications, we recommend WebRTC-based transports for more robust network handling, NAT traversal, and media optimization.
</Warning>

<CardGroup cols={2}>
  <Card title="WebSocket Client API Reference" icon="code" href="https://reference-server.pipecat.ai/en/latest/api/pipecat.transports.websocket.client.html">
    Client transport methods and configuration
  </Card>

  <Card title="WebSocket Server API Reference" icon="code" href="https://reference-server.pipecat.ai/en/latest/api/pipecat.transports.websocket.server.html">
    Server transport methods and configuration
  </Card>

  <Card title="Example Implementation" icon="play" href="https://github.com/pipecat-ai/pipecat-examples/tree/main/websocket">
    Complete client and server WebSocket examples
  </Card>

  <Card title="WebSocket Protocol" icon="book" href="https://developer.mozilla.org/en-US/docs/Web/API/WebSockets_API">
    WebSocket protocol documentation and guides
  </Card>
</CardGroup>

## Installation

To use WebSocket transports, install the required dependencies:

```bash  theme={null}
pip install "pipecat-ai[websocket]"
```

## Prerequisites

### WebSocket Application Setup

Before using WebSocket transports, you need:

1. **Server Implementation**: Set up WebSocket server using your preferred framework
2. **Client Implementation**: Configure WebSocket client for browser or application use
3. **Audio Configuration**: Set up audio streaming parameters and formats
4. **Connection Management**: Handle WebSocket lifecycle and error recovery

### Configuration Options

* **Transport Type**: Choose between client or server WebSocket transport
* **Audio Parameters**: Configure sample rates, channels, and audio formats
* **Frame Serialization**: Set up custom frame serializers if needed
* **Connection Handling**: Configure reconnection and error handling strategies

### Key Features

* **Bidirectional Communication**: Real-time audio and data streaming
* **Simple Protocol**: Lightweight WebSocket-based communication
* **Flexible Serialization**: Support for custom frame formats and audio codecs
* **Cross-Platform**: Works with any WebSocket-compatible client or server

## Usage

WebSocket transports provide simple client-server communication for audio streaming and real-time interaction. They're ideal for prototyping and controlled network environments.

See the [complete examples](https://github.com/pipecat-ai/pipecat-examples/tree/main/websocket) for full implementations including:

* WebSocket server setup and configuration
* Client-side WebSocket integration
* Audio streaming and frame handling
* Connection management and error recovery

---

# Source: https://docs.pipecat.ai/client/js/transports/websocket.md

> ## Documentation Index
> Fetch the complete documentation index at: https://docs.pipecat.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# WebSocketTransport

> A lightweight transport for WebSocket based connections with Pipecat

`WebSocketTransport` enables a purely WebSocket based connection between clients and your Pipecat application. It implements bidirectional audio and video streaming using a WebSocket for real-time communication.

This transport is intended for lightweight implementations, particularly for local development and testing. It expects your Pipecat server to include the corresponding [`WebSocketTransport` server-side](/server/services/transport/websocket-server) implementation.

<Warning>
  The `WebSocketTransport` is best suited for server-server applications and prototyping client/server apps.

  For client/server production applications, we strongly recommend using a WebRTC-based transport for robust network and media handling. For more on WebRTC vs. Websocket communication, check out [this article](https://voiceaiandvoiceagents.com/#websockets-webrtc).
</Warning>

## Usage

### Basic Setup

```javascript  theme={null}
import { PipecatClient } from "@pipecat-ai/client-js";
import {
  WebSocketTransport,
  ProtobufFrameSerializer,
} from "@pipecat-ai/websocket-transport";

const pcClient = new PipecatClient({
  transport: new WebSocketTransport({
    serializer: new ProtobufFrameSerializer(),
    recorderSampleRate: 8000,
    playerSampleRate: 8000,
  }),
  enableCam: false, // Default camera off
  enableMic: true, // Default microphone on
  callbacks: {
    // Event handlers
  },
});

await pcClient.connect({
  wsUrl: "ws://localhost:7860/ws", // Your WebSocket server URL
});
```

## API Reference

### Constructor Options

```typescript  theme={null}
type WebSocketTransportOptions = {
  wsUrl?: string;
  serializer?: WebSocketSerializer;
  recorderSampleRate?: number;
  playerSampleRate?: number;
};

export interface WebSocketTransportConstructorOptions
  extends WebSocketTransportOptions {
  mediaManager?: MediaManager;
}
```

#### Properties

<ParamField name="wsUrl" type="string">
  URL of the WebSocket server. This is the endpoint your client will connect to
  for WebSocket communication.
</ParamField>

<ParamField name="serializer" type="WebSocketSerializer" default="ProtobufFrameSerializer">
  The serializer to use for encoding/decoding messages sent over the WebSocket
  connection. The websocket-transport package provides two serializer options: -
  `ProtobufFrameSerializer`: Uses Protocol Buffers for serialization. -
  `TwilioSerializer`: Uses Twilio's serialization format. The main purpose of
  the TwilioSerializer is to allow testing the bots built to work with Twilio
  without having to make phone calls.
</ParamField>

<ParamField name="recorderSampleRate" type="number">
  Sample rate for which to encode the audio input. Default is `16000`.
</ParamField>

<ParamField name="playerSampleRate" type="number">
  Sample rate for which to decode the incoming audio for output. Default is
  `24000`.
</ParamField>

<ParamField name="mediaManager" type="MediaManager" default="DailyMediaManager">
  The media manager to use for handling local audio and video streams. This
  should not be overridden unless you have a specific reason to use a different
  media manager. The default is `DailyMediaManager`, which is suitable for most
  use cases. Note that the `DailyMediaManager` does not use any of Daily's
  services, it simply takes advantage of vast media support provided by the
  Daily library.
</ParamField>

### TransportConnectionParams

The `WebSocketTransport` takes the same options as the constructor; `WebSocketTransportOptions`. Anything provided here will override the defaults set in the constructor. The `wsUrl` is required to establish a connection.

<CodeGroup>
  ```typescript client theme={null}
  pcClient.connect({
    wsUrl: 'http://localhost:7860/ws'
  });
  // OR...
  pcClient.startBotAndConnect({
    endpoint: '/api/start', // returns { wsUrl }
  });
  ```

  ```python server theme={null}
  # See
  # https://github.com/pipecat-ai/pipecat-examples/blob/main/websocket/server/server.py
  # for a complete example of how to implement the server-side endpoint.
  @app.websocket("/ws")
  async def websocket_endpoint(websocket: WebSocket):
      await websocket.accept()
      print("WebSocket connection accepted")
      try:
          await run_bot(websocket)
      except Exception as e:
          print(f"Exception in run_bot: {e}")


  @app.post("/api/start")
  async def start(request: Request) -> Dict[Any, Any]:
      ws_url = "ws://localhost:7860/ws"
      return {"wsUrl": ws_url}
  ```

  ```python bot theme={null}
  # See
  # https://github.com/pipecat-ai/pipecat-examples/blob/main/websocket/server/bot_websocket_server.py
  # for a complete example of a bot script using the WebSocketTransport.
  from pipecat.serializers.protobuf import ProtobufFrameSerializer
  from pipecat.transports.websocket.fastapi import (
      FastAPIWebsocketParams,
      FastAPIWebsocketTransport,
  )

  async def run_bot(websocket_client):
      ws_transport = FastAPIWebsocketTransport(
          websocket=websocket_client,
          params=FastAPIWebsocketParams(
              audio_in_enabled=True,
              audio_out_enabled=True,
              add_wav_header=False,
              serializer=ProtobufFrameSerializer(),
          ),
      )

      llm = ... # Initialize your LLM here, e.g., OpenAI, HuggingFace, etc.

      messages = [{ role: "system", content: "You are a helpful assistant." }]
      context = LLMContext(messages)
      user_aggregator, assistant_aggregator = LLMContextAggregatorPair(
          context,
          user_params=LLMUserAggregatorParams(
              vad_analyzer=SileroVADAnalyzer(params=VADParams(stop_secs=0.2)),
          ),
      )

      pipeline = Pipeline(
          [
              ws_transport.input(),
              user_aggregator,
              llm,  # LLM
              ws_transport.output(),
              assistant_aggregator,
          ]
      )

      task = PipelineTask(
          pipeline,
          params,
      )
      ...
  ```
</CodeGroup>

### Methods

For most operations, you will not interact with the transport directly. Most methods have an equivalent in the `PipecatClient` and should be called from the `PipecatClient`. However, there is one transport-specific methods that you may need to call directly. When doing so, be sure to access your transport via the `transport` property of the `PipecatClient` instance.

<ResponseField name="handleUserAudioStream" type="method">
  If implementing your own serializer, you will need to pass the user audio
  stream to the transport via this method, which takes an `ArrayBuffer` of audio
  data.

  ```javascript  theme={null}
  transport.handleUserAudioStream(chunk.data);
  ```
</ResponseField>

## Events

The transport implements the various [`PipecatClient` event handlers](/client/js/api-reference/callbacks).

## Reconnection Handling

The WebSocketTransport does provide reconnection handling. If the WebSocket connection is lost, it will attempt to reconnect twice. If all reconnection attempts fail, the transport will gracefully disconnect.

## More Information

<CardGroup cols={2}>
  <Card horizontal title="WebSocket Demo" icon="video" href="https://github.com/pipecat-ai/pipecat-examples/tree/main/websocket">
    Basic Agent example using a WebSocket transport
  </Card>

  <Card horizontal title="Twilio Demo" icon="video" href="https://github.com/pipecat-ai/pipecat-examples/tree/main/twilio-chatbot">
    Example using a WebSocket transport to simulate a Twilio connection to a bot
  </Card>

  <Card horizontal title="Source" icon="github" href="https://github.com/pipecat-ai/pipecat-client-web-transports/tree/main/transports/websocket-transport">
    `WebSocketTransport`
  </Card>

  <Card horizontal title="Package" icon="browser" href="https://www.npmjs.com/package/@pipecat-ai/websocket-transport">
    `@pipecat-ai/websocket-transport`
  </Card>
</CardGroup>

---

# Source: https://docs.pipecat.ai/guides/learn/whats-next.md

> ## Documentation Index
> Fetch the complete documentation index at: https://docs.pipecat.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# What's Next

> Continue your Pipecat journey with advanced features, examples, and production deployment

## You've Mastered Voice AI Pipelines! 🎉

Congratulations! You've learned how to build complete voice AI applications with Pipecat. You're now well on your way to understanding pipelines, processors, transports, and all the components needed to create sophisticated conversational AI.

## Choose Your Path

<CardGroup cols={2}>
  <Card title="Explore Examples" icon="code" href="https://github.com/pipecat-ai/pipecat-examples">
    **Production-ready applications** - Dive into 30+ complete examples including
    multimodal bots, creative applications, and enterprise integrations.
  </Card>

  <Card title="Advanced Guides" icon="book-open" href="/guides/fundamentals/custom-frame-processor">
    **Master advanced features** - Explore specialized topics like telephony,
    deployment, custom processors, and production optimization.
  </Card>
</CardGroup>

## 💬 Join the Community

Get involved with the Pipecat community to share your projects, get support, and contribute to the ecosystem:

* **[Discord Community](https://discord.gg/pipecat)**: Get support, share projects, and connect with other developers
* **[GitHub](https://github.com/pipecat-ai/pipecat)**: Contribute to the project and report issues
* **[Examples Repository](https://github.com/pipecat-ai/pipecat-examples)**: Community-contributed applications

---

# Source: https://docs.pipecat.ai/server/services/transport/whatsapp.md

# Source: https://docs.pipecat.ai/guides/features/whatsapp.md

# Source: https://docs.pipecat.ai/deployment/pipecat-cloud/guides/whatsapp.md

> ## Documentation Index
> Fetch the complete documentation index at: https://docs.pipecat.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# WhatsApp Business Calling API

> Learn how to receive WhatsApp voice calls in your Pipecat Cloud application

## Overview

This guide explains how to integrate WhatsApp Business voice calling into your **Pipecat Cloud** application using the **WhatsApp Cloud API** and **SmallWebRTC**. Once configured, your Pipecat Cloud bot will be able to receive and handle real-time voice calls directly from WhatsApp users.

> For a more detailed explanation of WhatsApp Business setup, voice calling, and webhook configuration, refer to our [full WhatsApp Business Calling guide](/guides/features/whatsapp).

***

## Pipecat Cloud–Specific Setup

In Pipecat Cloud you only need to ensure your bot is deployed and your environment variables are configured correctly.

### Configure Webhook for Pipecat Cloud

Your webhook URL must point to your deployed Pipecat agent:

```
https://api.pipecat.daily.co/v1/public/webhooks/$ORGANIZATION_ID/$AGENT_NAME/whatsapp
```

* **Important:** Use your **public API key** as the `Verify Token` for Meta Developer Console verification.
* Always include the `/whatsapp` path at the end of the webhook URL.
* Make sure your Pipecat agent is **deployed and running**.

### Pipecat Cloud Secrets

In Pipecat Cloud, configure the following environment variables as **secrets**.

```bash  theme={null}
WHATSAPP_TOKEN=<your_access_token>
WHATSAPP_PHONE_NUMBER_ID=<your_phone_number_id>
WHATSAPP_APP_SECRET=<your_app_secret>
```

If you have them in a `.env` file, you can upload to Pipecat Cloud as a secret set like this:

```bash  theme={null}
uv run pipecat cloud secrets set whatsapp-secrets --file .env
```

This creates a secret set named `whatsapp-secrets` and securely stores your API keys for your Pipecat Cloud agent.

> For instructions on generating the access token, phone number ID, and app secret, refer to the [WhatsApp Cloud API Configuration](/guides/features/whatsapp#whatsapp-configuration).

### Example

Your bot must use `SmallWebRTCTransport` with audio enabled:

```python  theme={null}
transport = SmallWebRTCTransport(
    webrtc_connection=webrtc_connection,
    params=TransportParams(
        audio_in_enabled=True,
        audio_out_enabled=True,
    ),
)
```

## Full demo

<Card horizontal title="Demo" icon="play" href="https://github.com/pipecat-ai/pipecat-examples/tree/main/p2p-webrtc/pipecat-cloud">
  Pipecat Cloud WhatsApp Demo
</Card>

### Testing Calls

1. From WhatsApp, call your registered WhatsApp Business number.
2. The call will be routed to your Pipecat Cloud agent.
3. The bot should automatically answer and handle the conversation.

### Troubleshooting

Common Pipecat Cloud issues:

1. **Webhook Verification Failed**

   * Verify the public API key matches the verify token in Meta Developer Console.
   * Ensure the webhook URL is using the right organization and service names.

2. **Bot Not Answering Calls**
   * Ensure the WhatsApp Business number has calling enabled.
   * Ensure that the WhatsApp token is still valid.

> For a complete troubleshooting section, see [WhatsApp Business Calling guide](/guides/features/whatsapp#troubleshooting).

---

# Source: https://docs.pipecat.ai/server/services/stt/whisper.md

> ## Documentation Index
> Fetch the complete documentation index at: https://docs.pipecat.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Whisper

> Speech-to-text service implementation using locally-downloaded Whisper models

## Overview

`WhisperSTTService` provides offline speech recognition using OpenAI's Whisper models running locally. Supports multiple model sizes and hardware acceleration options including CPU, CUDA, and Apple Silicon (MLX) for privacy-focused transcription without external API calls.

<CardGroup cols={2}>
  <Card title="Whisper STT API Reference" icon="code" href="https://reference-server.pipecat.ai/en/latest/api/pipecat.services.whisper.stt.html">
    Pipecat's API methods for Whisper STT integration
  </Card>

  <Card title="Standard Whisper Example" icon="play" href="https://github.com/pipecat-ai/pipecat/blob/main/examples/foundational/13-whisper-transcription.py">
    Complete example with standard Whisper
  </Card>

  <Card title="Whisper Documentation" icon="book" href="https://github.com/openai/whisper">
    OpenAI's Whisper research paper and model details
  </Card>

  <Card title="MLX Whisper Example" icon="microphone" href="https://github.com/pipecat-ai/pipecat/blob/main/examples/foundational/13e-whisper-mlx.py">
    Apple Silicon optimized example
  </Card>
</CardGroup>

## Installation

Choose your installation based on your hardware:

### Standard Whisper (CPU/CUDA)

```bash  theme={null}
pip install "pipecat-ai[whisper]"
```

### MLX Whisper (Apple Silicon)

```bash  theme={null}
pip install "pipecat-ai[mlx-whisper]"
```

## Prerequisites

### Local Model Setup

Before using Whisper STT services, you need:

1. **Model Selection**: Choose appropriate Whisper model size (tiny, base, small, medium, large)
2. **Hardware Configuration**: Set up CPU, CUDA, or Apple Silicon acceleration
3. **Storage Space**: Ensure sufficient disk space for model downloads

### Configuration Options

* **Model Size**: Balance between accuracy and performance based on your hardware
* **Hardware Acceleration**: Configure CUDA for NVIDIA GPUs or MLX for Apple Silicon
* **Language Support**: Whisper supports 99+ languages out of the box

<Tip>
  No API keys required - Whisper runs entirely locally for complete privacy.
</Tip>

---

# Source: https://docs.pipecat.ai/server/services/tts/xtts.md

> ## Documentation Index
> Fetch the complete documentation index at: https://docs.pipecat.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# XTTS

> Text-to-speech service implementation using Coqui's XTTS streaming server

<Warning>
  Coqui, the XTTS maintainer, has shut down. XTTS may not receive future updates
  or support.
</Warning>

## Overview

`XTTSTTSService` provides multilingual voice synthesis with voice cloning capabilities through a locally hosted streaming server. The service supports real-time streaming and custom voice training using Coqui's XTTS-v2 model for cross-lingual text-to-speech.

<CardGroup cols={2}>
  <Card title="XTTS API Reference" icon="code" href="https://reference-server.pipecat.ai/en/latest/api/pipecat.services.xtts.tts.html">
    Pipecat's API methods for XTTS integration
  </Card>

  <Card title="Example Implementation" icon="play" href="https://github.com/pipecat-ai/pipecat/blob/main/examples/foundational/07i-interruptible-xtts.py">
    Complete example with voice cloning
  </Card>

  <Card title="XTTS Repository" icon="book" href="https://github.com/coqui-ai/xtts-streaming-server">
    Official XTTS streaming server repository
  </Card>

  <Card title="Voice Cloning" icon="microphone" href="https://github.com/coqui-ai/xtts-streaming-server#voice-cloning">
    Learn about custom voice training
  </Card>
</CardGroup>

## Installation

XTTS requires a running streaming server. Start the server using Docker:

```bash  theme={null}
docker run --gpus=all -e COQUI_TOS_AGREED=1 --rm -p 8000:80 \
  ghcr.io/coqui-ai/xtts-streaming-server:latest-cuda121
```

## Prerequisites

### XTTS Server Setup

Before using XTTSTTSService, you need:

1. **Docker Environment**: Set up Docker with GPU support for optimal performance
2. **XTTS Server**: Run the XTTS streaming server container
3. **Voice Models**: Configure voice models and cloning samples as needed

### Required Configuration

* **Server URL**: Configure the XTTS server endpoint (default: `http://localhost:8000`)
* **Voice Selection**: Set up voice models or voice cloning samples

<Note>
  GPU acceleration is recommended for optimal performance. The server requires
  CUDA support for best results.
</Note>