# ... create websocket-client named ws ...

def float_to_16bit_pcm(float32_array):
    clipped = [max(-1.0, min(1.0, x)) for x in float32_array]
    pcm16 = b''.join(struct.pack('<h', int(x * 32767)) for x in clipped)
    return pcm16

def base64_encode_audio(float32_array):
    pcm_bytes = float_to_16bit_pcm(float32_array)
    encoded = base64.b64encode(pcm_bytes).decode('ascii')
    return encoded

files = [
    './path/to/sample1.wav',
    './path/to/sample2.wav',
    './path/to/sample3.wav'
]

for filename in files:
    data, samplerate = sf.read(filename, dtype='float32')
    channel_data = data[:, 0] if data.ndim > 1 else data
    base64_chunk = base64_encode_audio(channel_data)

    # Send the client event
    event = {
        "type": "input_audio_buffer.append",
        "audio": base64_chunk
    }
    ws.send(json.dumps(event))
```

### Send full audio messages

It is also possible to create conversation messages that are full audio
recordings. Use the
[conversation.item.create](https://platform.openai.com/docs/api-reference/realtime-client-events/conversation/item/create)
client event to create messages with `input_audio` content.

```javascript
const fullAudio = "<a base64-encoded string of audio bytes>";

const event = {
  type: "conversation.item.create",
  item: {
    type: "message",
    role: "user",
    content: [
      {
        type: "input_audio",
        audio: fullAudio,
      },
    ],
  },
};

// WebRTC data channel and WebSocket both have .send()
dataChannel.send(JSON.stringify(event));
```

```python
fullAudio = "<a base64-encoded string of audio bytes>"

event = {
    "type": "conversation.item.create",
    "item": {
        "type": "message",
        "role": "user",
        "content": [
            {
                "type": "input_audio",
                "audio": fullAudio,
            }
        ],
    },
}

ws.send(json.dumps(event))
```

### Working with audio output from a WebSocket

**To play output audio back on a client device like a web browser, we recommend
using WebRTC rather than WebSockets**. WebRTC will be more robust sending media
to client devices over uncertain network conditions.

But to work with audio output in server-to-server applications using a
WebSocket, you will need to listen for
[response.audio.delta](https://platform.openai.com/docs/api-reference/realtime-server-events/response/audio/delta)
events containing the Base64-encoded chunks of audio data from the model. You
will either need to buffer these chunks and write them out to a file, or maybe
immediately stream them to another source like a phone call with Twilio.

Note that the
[response.audio.done](https://platform.openai.com/docs/api-reference/realtime-server-events/response/audio/done)
and
[response.done](https://platform.openai.com/docs/api-reference/realtime-server-events/response/done)
events won't actually contain audio data in them - just audio content
transcriptions. To get the actual bytes, you'll need to listen for the
[response.audio.delta](https://platform.openai.com/docs/api-reference/realtime-server-events/response/audio/delta)
events.

The format of the output chunks can be configured either for the entire session,
or per response.

- Session: `session.output_audio_format` in
  [session.update](https://platform.openai.com/docs/api-reference/realtime-client-events/session/update)
- Response: `response.output_audio_format` in
  [response.create](https://platform.openai.com/docs/api-reference/realtime-client-events/response/create)

```javascript
function handleEvent(e) {
  const serverEvent = JSON.parse(e.data);
  if (serverEvent.type === "response.audio.delta") {
    // Access Base64-encoded audio chunks
    // console.log(serverEvent.delta);
  }
}

// Listen for server messages (WebSocket)
ws.on("message", handleEvent);
```

```python
def on_message(ws, message):
    server_event = json.loads(message)
    if server_event.type == "response.audio.delta":
        # Access Base64-encoded audio chunks:
        # print(server_event.delta)
```

## Voice activity detection

By default, Realtime sessions have **voice activity detection (VAD)** enabled,
which means the API will determine when the user has started or stopped speaking
and respond automatically.

Read more about how to configure VAD in our
[voice activity detection](https://platform.openai.com/docs/guides/realtime-vad)
guide.

### Disable VAD

VAD can be disabled by setting `turn_detection` to `null` with the
[session.update](https://platform.openai.com/docs/api-reference/realtime-client-events/session/update)
client event. This can be useful for interfaces where you would like to take
granular control over audio input, like push to talk interfaces.

When VAD is disabled, the client will have to manually emit some additional
client events to trigger audio responses:

- Manually send
  [input_audio_buffer.commit](https://platform.openai.com/docs/api-reference/realtime-client-events/input_audio_buffer/commit),
  which will create a new user input item for the conversation.
- Manually send
  [response.create](https://platform.openai.com/docs/api-reference/realtime-client-events/response/create)
  to trigger an audio response from the model.
- Send
  [input_audio_buffer.clear](https://platform.openai.com/docs/api-reference/realtime-client-events/input_audio_buffer/clear)
  before beginning a new user input.

### Keep VAD, but disable automatic responses

If you would like to keep VAD mode enabled, but would just like to retain the
ability to manually decide when a response is generated, you can set
`turn_detection.interrupt_response` and `turn_detection.create_response` to
`false` with the
[session.update](https://platform.openai.com/docs/api-reference/realtime-client-events/session/update)
client event. This will retain all the behavior of VAD but not automatically
create new Responses. Clients can trigger these manually with a
[response.create](https://platform.openai.com/docs/api-reference/realtime-client-events/response/create)
event.

This can be useful for moderation or input validation or RAG patterns, where
you're comfortable trading a bit more latency in the interaction for control
over inputs.

## Create responses outside the default conversation

By default, all responses generated during a session are added to the session's
conversation state (the "default conversation"). However, you may want to
generate model responses outside the context of the session's default
conversation, or have multiple responses generated concurrently. You might also
want to have more granular control over which conversation items are considered
while the model generates a response (e.g. only the last N number of turns).

Generating "out-of-band" responses which are not added to the default
conversation state is possible by setting the `response.conversation` field to
the string `none` when creating a response with the
[response.create](https://platform.openai.com/docs/api-reference/realtime-client-events/response/create)
client event.

When creating an out-of-band response, you will probably also want some way to
identify which server-sent events pertain to this response. You can provide
`metadata` for your model response that will help you identify which response is
being generated for this client-sent event.

```javascript
const prompt = `
Analyze the conversation so far. If it is related to support, output
"support". If it is related to sales, output "sales".
`;

const event = {
  type: "response.create",
  response: {
    // Setting to "none" indicates the response is out of band
    // and will not be added to the default conversation
    conversation: "none",

    // Set metadata to help identify responses sent back from the model
    metadata: { topic: "classification" },

    // Set any other available response fields
    modalities: ["text"],
    instructions: prompt,
  },
};

// WebRTC data channel and WebSocket both have .send()
dataChannel.send(JSON.stringify(event));
```

```python
prompt = """
Analyze the conversation so far. If it is related to support, output
"support". If it is related to sales, output "sales".
"""

event = {
    "type": "response.create",
    "response": {
        # Setting to "none" indicates the response is out of band,
        # and will not be added to the default conversation
        "conversation": "none",

        # Set metadata to help identify responses sent back from the model
        "metadata": { "topic": "classification" },

        # Set any other available response fields
        "modalities": [ "text" ],
        "instructions": prompt,
    },
}

ws.send(json.dumps(event))
```

Now, when you listen for the
[response.done](https://platform.openai.com/docs/api-reference/realtime-server-events/response/done)
server event, you can identify the result of your out-of-band response.

```javascript
function handleEvent(e) {
  const serverEvent = JSON.parse(e.data);
  if (
    serverEvent.type === "response.done" &&
    serverEvent.response.metadata?.topic === "classification"
  ) {
    // this server event pertained to our OOB model response
    console.log(serverEvent.response.output[0]);
  }
}

// Listen for server messages (WebRTC)
dataChannel.addEventListener("message", handleEvent);

// Listen for server messages (WebSocket)
// ws.on("message", handleEvent);
```

```python
def on_message(ws, message):
    server_event = json.loads(message)
    topic = ""

    # See if metadata is present
    try:
        topic = server_event.response.metadata.topic
    except AttributeError:
        print("topic not set")

    if server_event.type == "response.done" and topic == "classification":
        # this server event pertained to our OOB model response
        print(server_event.response.output[0])
```

### Create a custom context for responses

You can also construct a custom context that the model will use to generate a
response, outside the default/current conversation. This can be done using the
`input` array on a
[response.create](https://platform.openai.com/docs/api-reference/realtime-client-events/response/create)
client event. You can use new inputs, or reference existing input items in the
conversation by ID.

```javascript
const event = {
  type: "response.create",
  response: {
    conversation: "none",
    metadata: { topic: "pizza" },
    modalities: ["text"],

    // Create a custom input array for this request with whatever context
    // is appropriate
    input: [
      // potentially include existing conversation items:
      {
        type: "item_reference",
        id: "some_conversation_item_id",
      },
      {
        type: "message",
        role: "user",
        content: [
          {
            type: "input_text",
            text: "Is it okay to put pineapple on pizza?",
          },
        ],
      },
    ],
  },
};

// WebRTC data channel and WebSocket both have .send()
dataChannel.send(JSON.stringify(event));
```

```python
event = {
    "type": "response.create",
    "response": {
        "conversation": "none",
        "metadata": { "topic": "pizza" },
        "modalities": [ "text" ],

        # Create a custom input array for this request with whatever
        # context is appropriate
        "input": [
            # potentially include existing conversation items:
            {
                "type": "item_reference",
                "id": "some_conversation_item_id"
            },

            # include new content as well
            {
                "type": "message",
                "role": "user",
                "content": [
                    {
                        "type": "input_text",
                        "text": "Is it okay to put pineapple on pizza?",
                    }
                ],
            }
        ],
    },
}

ws.send(json.dumps(event))
```

### Create responses with no context

You can also insert responses into the default conversation, ignoring all other
instructions and context. Do this by setting `input` to an empty array.

```javascript
const prompt = `
Say exactly the following:
I'm a little teapot, short and stout! 
This is my handle, this is my spout!
`;

const event = {
  type: "response.create",
  response: {
    // An empty input array removes existing context
    input: [],
    instructions: prompt,
  },
};

// WebRTC data channel and WebSocket both have .send()
dataChannel.send(JSON.stringify(event));
```

```python
prompt = """
Say exactly the following:
I'm a little teapot, short and stout!
This is my handle, this is my spout!
"""

event = {
    "type": "response.create",
    "response": {
        # An empty input array removes all prior context
        "input": [],
        "instructions": prompt,
    },
}

ws.send(json.dumps(event))
```

## Function calling

The Realtime models also support **function calling**, which enables you to
execute custom code to extend the capabilities of the model. Here's how it works
at a high level:

1.  When
    [updating the session](https://platform.openai.com/docs/api-reference/realtime-client-events/session/update)
    or
    [creating a response](https://platform.openai.com/docs/api-reference/realtime-client-events/response/create),
    you can specify a list of available functions for the model to call.
2.  If when processing input, the model determines it should make a function
    call, it will add items to the conversation representing arguments to a
    function call.
3.  When the client detects conversation items that contain function call
    arguments, it will execute custom code using those arguments
4.  When the custom code has been executed, the client will create new
    conversation items that contain the output of the function call, and ask the
    model to respond.

Let's see how this would work in practice by adding a callable function that
will provide today's horoscope to users of the model. We'll show the shape of
the client event objects that need to be sent, and what the server will emit in
turn.

### Configure callable functions

First, we must give the model a selection of functions it can call based on user
input. Available functions can be configured either at the session level, or the
individual response level.

- Session: `session.tools` property in
  [session.update](https://platform.openai.com/docs/api-reference/realtime-client-events/session/update)
- Response: `response.tools` property in
  [response.create](https://platform.openai.com/docs/api-reference/realtime-client-events/response/create)

Here's an example client event payload for a `session.update` that configures a
horoscope generation function, that takes a single argument (the astrological
sign for which the horoscope should be generated):

[session.update](https://platform.openai.com/docs/api-reference/realtime-client-events/session/update)

```json
{
  "type": "session.update",
  "session": {
    "tools": [
      {
        "type": "function",
        "name": "generate_horoscope",
        "description": "Give today's horoscope for an astrological sign.",
        "parameters": {
          "type": "object",
          "properties": {
            "sign": {
              "type": "string",
              "description": "The sign for the horoscope.",
              "enum": [
                "Aries",
                "Taurus",
                "Gemini",
                "Cancer",
                "Leo",
                "Virgo",
                "Libra",
                "Scorpio",
                "Sagittarius",
                "Capricorn",
                "Aquarius",
                "Pisces"
              ]
            }
          },
          "required": ["sign"]
        }
      }
    ],
    "tool_choice": "auto"
  }
}
```

The `description` fields for the function and the parameters help the model
choose whether or not to call the function, and what data to include in each
parameter. If the model receives input that indicates the user wants their
horoscope, it will call this function with a `sign` parameter.

### Detect when the model wants to call a function

Based on inputs to the model, the model may decide to call a function in order
to generate the best response. Let's say our application adds the following
conversation item and attempts to generate a response:

[conversation.item.create](https://platform.openai.com/docs/api-reference/realtime-client-events/conversation/item/create)

```json
{
  "type": "conversation.item.create",
  "item": {
    "type": "message",
    "role": "user",
    "content": [
      {
        "type": "input_text",
        "text": "What is my horoscope? I am an aquarius."
      }
    ]
  }
}
```

Followed by a client event to generate a response:

[response.create](https://platform.openai.com/docs/api-reference/realtime-client-events/response/create)

```json
{
  "type": "response.create"
}
```

Instead of immediately returning a text or audio response, the model will
instead generate a response that contains the arguments that should be passed to
a function in the developer's application. You can listen for realtime updates
to function call arguments using the
[response.function_call_arguments.delta](https://platform.openai.com/docs/api-reference/realtime-server-events/response/function_call_arguments/delta)
server event, but `response.done` will also have the complete data we need to
call our function.

[response.done](https://platform.openai.com/docs/api-reference/realtime-server-events/response/done)

```json
{
  "type": "response.done",
  "event_id": "event_AeqLA8iR6FK20L4XZs2P6",
  "response": {
    "object": "realtime.response",
    "id": "resp_AeqL8XwMUOri9OhcQJIu9",
    "status": "completed",
    "status_details": null,
    "output": [
      {
        "object": "realtime.item",
        "id": "item_AeqL8gmRWDn9bIsUM2T35",
        "type": "function_call",
        "status": "completed",
        "name": "generate_horoscope",
        "call_id": "call_sHlR7iaFwQ2YQOqm",
        "arguments": "{\"sign\":\"Aquarius\"}"
      }
    ],
    "usage": {
      "total_tokens": 541,
      "input_tokens": 521,
      "output_tokens": 20,
      "input_token_details": {
        "text_tokens": 292,
        "audio_tokens": 229,
        "cached_tokens": 0,
        "cached_tokens_details": { "text_tokens": 0, "audio_tokens": 0 }
      },
      "output_token_details": {
        "text_tokens": 20,
        "audio_tokens": 0
      }
    },
    "metadata": null
  }
}
```

In the JSON emitted by the server, we can detect that the model wants to call a
custom function:

| Property                       | Function calling purpose                                                                                                   |
| ------------------------------ | -------------------------------------------------------------------------------------------------------------------------- |
| `response.output[0].type`      | When set to `function_call`, indicates this response contains arguments for a named function call.                         |
| `response.output[0].name`      | The name of the configured function to call, in this case `generate_horoscope`                                             |
| `response.output[0].arguments` | A JSON string containing arguments to the function. In our case, `"{\"sign\":\"Aquarius\"}"`.                              |
| `response.output[0].call_id`   | A system-generated ID for this function call - **you will need this ID to pass a function call result back to the model**. |

Given this information, we can execute code in our application to generate the
horoscope, and then provide that information back to the model so it can
generate a response.

### Provide the results of a function call to the model

Upon receiving a response from the model with arguments to a function call, your
application can execute code that satisfies the function call. This could be
anything you want, like talking to external APIs or accessing databases.

Once you are ready to give the model the results of your custom code, you can
create a new conversation item containing the result via the
`conversation.item.create` client event.

[conversation.item.create](https://platform.openai.com/docs/api-reference/realtime-client-events/conversation/item/create)

```json
{
  "type": "conversation.item.create",
  "item": {
    "type": "function_call_output",
    "call_id": "call_sHlR7iaFwQ2YQOqm",
    "output": "{\"horoscope\": \"You will soon meet a new friend.\"}"
  }
}
```

- The conversation item type is `function_call_output`
- `item.call_id` is the same ID we got back in the `response.done` event above
- `item.output` is a JSON string containing the results of our function call

Once we have added the conversation item containing our function call results,
we again emit the `response.create` event from the client. This will trigger a
model response using the data from the function call.

[response.create](https://platform.openai.com/docs/api-reference/realtime-client-events/response/create)

```json
{
  "type": "response.create"
}
```

## Error handling

The
[error](https://platform.openai.com/docs/api-reference/realtime-server-events/error)
event is emitted by the server whenever an error condition is encountered on the
server during the session. Occasionally, these errors can be traced to a client
event that was emitted by your application.

Unlike HTTP requests and responses, where a response is implicitly tied to a
request from the client, we need to use an `event_id` property on client events
to know when one of them has triggered an error condition on the server. This
technique is shown in the code below, where the client attempts to emit an
unsupported event type.

```javascript
const event = {
  event_id: "my_awesome_event",
  type: "scooby.dooby.doo",
};

dataChannel.send(JSON.stringify(event));
```

This unsuccessful event sent from the client will emit an error event like the
following:

```json
{
  "type": "invalid_request_error",
  "code": "invalid_value",
  "message": "Invalid value: 'scooby.dooby.doo' ...",
  "param": "type",
  "event_id": "my_awesome_event"
}
```