# Load Parameters

Set load-time parameters such as the context length, GPU offload ratio, and more.

### Set Load Parameters with `.model()`

The `.model()` retrieves a handle to a model that has already been loaded, or loads a new one on demand (JIT loading).

**Note**: if the model is already loaded, the configuration will be **ignored**.

```lms_code_snippet
  variants:
    TypeScript:
      language: typescript
      code: |
        const model = await client.llm.model("qwen2.5-7b-instruct", {
          config: {
            contextLength: 8192,
            gpu: {
              ratio: 0.5,
            },
          },
        });
```

See [`LLMLoadModelConfig`](./../api-reference/llm-load-model-config) for all configurable fields.

### Set Load Parameters with `.load()`

The `.load()` method creates a new model instance and loads it with the specified configuration.

```lms_code_snippet
  variants:
    TypeScript:
      language: typescript
      code: |
        const model = await client.llm.load("qwen2.5-7b-instruct", {
          config: {
            contextLength: 8192,
            gpu: {
              ratio: 0.5,
            },
          },
        });
```

See [`LLMLoadModelConfig`](./../api-reference/llm-load-model-config) for all configurable fields.


## agent

### The `.act()` call

> How to use the `.act()` call to turn LLMs into autonomous agents that can perform tasks on your local machine.


## Automatic tool calling

We introduce the concept of execution "rounds" to describe the combined process of running a tool, providing its output to the LLM, and then waiting for the LLM to decide what to do next.

**Execution Round**

```
 â¢ run a tool ->
 â   â¢ provide the result to the LLM ->
 â       â¢ wait for the LLM to generate a response
 â
 ââââââââââââââââââââââââââââââââââââââââââ ââ (return)
```

A model might choose to run tools multiple times before returning a final result. For example, if the LLM is writing code, it might choose to compile or run the program, fix errors, and then run it again, rinse and repeat until it gets the desired result.

With this in mind, we say that the `.act()` API is an automatic "multi-round" tool calling API.


### Quick Example

```lms_code_snippet
  variants:
    TypeScript:
      language: typescript
      code: |
        import { LMStudioClient, tool } from "@lmstudio/sdk";
        import { z } from "zod";

        const client = new LMStudioClient();

        const multiplyTool = tool({
          name: "multiply",
          description: "Given two numbers a and b. Returns the product of them.",
          parameters: { a: z.number(), b: z.number() },
          implementation: ({ a, b }) => a * b,
        });

        const model = await client.llm.model("qwen2.5-7b-instruct");
        await model.act("What is the result of 12345 multiplied by 54321?", [multiplyTool], {
          onMessage: (message) => console.info(message.toString()),
        });
```

> **_NOTE:_**  at this time, this code expects zod v3

### What does it mean for an LLM to "use a tool"?

LLMs are largely text-in, text-out programs. So, you may ask "how can an LLM use a tool?". The answer is that some LLMs are trained to ask the human to call the tool for them, and expect the tool output to to be provided back in some format.

Imagine you're giving computer support to someone over the phone. You might say things like "run this command for me ... OK what did it output? ... OK now click there and tell me what it says ...". In this case you're the LLM! And you're "calling tools" vicariously through the person on the other side of the line.


### Important: Model Selection

The model selected for tool use will greatly impact performance.

Some general guidance when selecting a model:

- Not all models are capable of intelligent tool use
- Bigger is better (i.e., a 7B parameter model will generally perform better than a 3B parameter model)
- We've observed [Qwen2.5-7B-Instruct](https://model.lmstudio.ai/download/lmstudio-community/Qwen2.5-7B-Instruct-GGUF) to perform well in a wide variety of cases
- This guidance may change

### Example: Multiple Tools

The following code demonstrates how to provide multiple tools in a single `.act()` call.

```lms_code_snippet
  variants:
    TypeScript:
      language: typescript
      code: |
        import { LMStudioClient, tool } from "@lmstudio/sdk";
        import { z } from "zod";

        const client = new LMStudioClient();

        const additionTool = tool({
          name: "add",
          description: "Given two numbers a and b. Returns the sum of them.",
          parameters: { a: z.number(), b: z.number() },
          implementation: ({ a, b }) => a + b,
        });

        const isPrimeTool = tool({
          name: "isPrime",
          description: "Given a number n. Returns true if n is a prime number.",
          parameters: { n: z.number() },
          implementation: ({ n }) => {
            if (n < 2) return false;
            const sqrt = Math.sqrt(n);
            for (let i = 2; i <= sqrt; i++) {
              if (n % i === 0) return false;
            }
            return true;
          },
        });

        const model = await client.llm.model("qwen2.5-7b-instruct");
        await model.act(
          "Is the result of 12345 + 45668 a prime? Think step by step.",
          [additionTool, isPrimeTool],
          { onMessage: (message) => console.info(message.toString()) },
        );
```

### Example: Chat Loop with Create File Tool

The following code creates a conversation loop with an LLM agent that can create files.

```lms_code_snippet
  variants:
    TypeScript:
      language: typescript
      code: |
        import { Chat, LMStudioClient, tool } from "@lmstudio/sdk";
        import { existsSync } from "fs";
        import { writeFile } from "fs/promises";
        import { createInterface } from "readline/promises";
        import { z } from "zod";

        const rl = createInterface({ input: process.stdin, output: process.stdout });
        const client = new LMStudioClient();
        const model = await client.llm.model();
        const chat = Chat.empty();

        const createFileTool = tool({
          name: "createFile",
          description: "Create a file with the given name and content.",
          parameters: { name: z.string(), content: z.string() },
          implementation: async ({ name, content }) => {
            if (existsSync(name)) {
              return "Error: File already exists.";
            }
            await writeFile(name, content, "utf-8");
            return "File created.";
          },
        });

        while (true) {
          const input = await rl.question("You: ");
          // Append the user input to the chat
          chat.append("user", input);

          process.stdout.write("Bot: ");
          await model.act(chat, [createFileTool], {
            // When the model finish the entire message, push it to the chat
            onMessage: (message) => chat.append(message),
            onPredictionFragment: ({ content }) => {
              process.stdout.write(content);
            },
          });
          process.stdout.write("\n");
        }
```


### Tool Definition

> Define tools with the `tool()` function and pass them to the model in the `act()` call.


You can define tools with the `tool()` function and pass them to the model in the `act()` call.

## Anatomy of a Tool

Follow this standard format to define functions as tools:

```lms_code_snippet
  title: "index.ts"
  variants:
    TypeScript:
      language: typescript
      code: |
        import { tool } from "@lmstudio/sdk";
        import { z } from "zod";

        const exampleTool = tool({
          // The name of the tool
          name: "add",

          // A description of the tool
          description: "Given two numbers a and b. Returns the sum of them.",

          // zod schema of the parameters
          parameters: { a: z.number(), b: z.number() },

          // The implementation of the tool. Just a regular function.
          implementation: ({ a, b }) => a + b,
        });
```

**Important**: The tool name, description, and the parameter definitions are all passed to the model!

This means that your wording will affect the quality of the generation. Make sure to always provide a clear description of the tool so the model knows how to use it.

## Tools with External Effects (like Computer Use or API Calls)

Tools can also have external effects, such as creating files or calling programs and even APIs. By implementing tools with external effects, you
can essentially turn your LLMs into autonomous agents that can perform tasks on your local machine.

## Example: `createFileTool`

### Tool Definition

```lms_code_snippet
  title: "createFileTool.ts"
  variants:
    TypeScript:
      language: typescript
      code: |
        import { tool } from "@lmstudio/sdk";
        import { existsSync } from "fs";
        import { writeFile } from "fs/promises";
        import { z } from "zod";

        const createFileTool = tool({
          name: "createFile",
          description: "Create a file with the given name and content.",
          parameters: { name: z.string(), content: z.string() },
          implementation: async ({ name, content }) => {
            if (existsSync(name)) {
              return "Error: File already exists.";
            }
            await writeFile(name, content, "utf-8");
            return "File created.";
          },
        });
```

### Example code using the `createFile` tool:

```lms_code_snippet
  title: "index.ts"
  variants:
    TypeScript:
      language: typescript
      code: |
        import { LMStudioClient } from "@lmstudio/sdk";
        import { createFileTool } from "./createFileTool";

        const client = new LMStudioClient();

        const model = await client.llm.model("qwen2.5-7b-instruct");
        await model.act(
          "Please create a file named output.txt with your understanding of the meaning of life.",
          [createFileTool],
        );
```


## plugins

## Introduction to Plugins

> A brief introduction to making plugins for LM Studio using TypeScript.


Plugins extend LM Studio's functionality by providing "hook functions" that execute at specific points during operation.

Plugins are currently written in JavaScript/TypeScript and run on Node.js v22.21.1. Python support is in development.

## Getting Started

LM Studio includes Node.js, so no separate installation is required.

### Create a new plugin

To create a new plugin, navigate to LM Studio... [TO BE CONTINUED]

### Run a plugin in development mode

Once you've created a plugin, run this command in the plugin directory to start development mode:

```bash
lms dev
```

Your plugin will appear in LM Studio's plugin list. Development mode automatically rebuilds and reloads your plugin when you make code changes.

You only need `lms dev` during development. When the plugin is installed, LM Studio automatically runs them as needed. Learn more about distributing and installing plugins in the [Sharing Plugins](./plugins/publish-plugins) section.

## Next Steps

- [Tools Providers](./plugins/tools-provider)

  Give models extra capabilities by creating tools they can use during generation, like accessing external APIs or performing calculations.

- [Prompt Preprocessors](./plugins/prompt-preprocessor)

  Modify user input before it reaches the model - handle file uploads, inject context, or transform queries.

- [Generators](./plugins/generator)

  Create custom text generation sources that replace the local model, perfect for online model adapters.

- [Custom Configurations](./plugins/custom-configuration)

  Add configuration UIs so users can customize your plugin's behavior.

- [Third-Party Dependencies](./plugins/dependencies)

  Use npm packages to leverage existing libraries in your plugins.

- [Sharing Plugins](./plugins/publish-plugins)

  Package and share your plugins with the community.


### Using `npm` Dependencies

> How to use npm packages in LM Studio plugins


## Add dependencies to your plugin with `npm`

LM Studio plugins supports `npm` packages. You can just install them using `npm install`.

When the plugin is installed, LM Studio will automatically download all the required dependencies that are declared in `package.json` and `package-lock.json`. (The user does not need to have Node.js/npm installed.)

### `postinstall` scripts

For safety reasons, we do **not** run `postinstall` scripts. Thus please make sure you are not using any npm packages that require postinstall scripts to work.

## Using Other Package Managers

Since we rely on `package-lock.json`, lock files produced by other package managers will not work. Thus we recommend only using `npm` when developing LM Studio plugins.


### tools-provider

### Introduction to Tools Provider

> Writing tools providers for LM Studio plugins using TypeScript


Tools provider is a function that returns an array of tools that the model can use during generation.

## Examples

The following are some plugins that make use of tools providers:

- [lmstudio/wikipedia](https://lmstudio.ai/lmstudio/wikipedia)

  Gives the LLM tools to search and read Wikipedia articles.

- [lmstudio/js-code-sandbox](https://lmstudio.ai/lmstudio/js-code-sandbox)

  Gives the LLM tools to run JavaScript/TypeScript code in a sandbox environment using [deno](https://deno.com/).

- [lmstudio/dice](https://lmstudio.ai/lmstudio/dice)

  Allows the LLM to generate random numbers using "dice".


#### Single Tool

> 


To setup a tools provider, first create the a file `toolsProvider.ts` in your plugin's `src` directory:

```lms_code_snippet
  title: "src/toolsProvider.ts"
  variants:
    TypeScript:
      language: typescript
      code: |
        import { tool, Tool, ToolsProviderController } from "@lmstudio/sdk";
        import { z } from "zod";
        import { existsSync } from "fs";
        import { writeFile } from "fs/promises";
        import { join } from "path";

        export async function toolsProvider(ctl: ToolsProviderController) {
          const tools: Tool[] = [];

          const createFileTool = tool({
            // Name of the tool, this will be passed to the model. Aim for concise, descriptive names
            name: `create_file`,
            // Your description here, more details will help the model to understand when to use the tool
            description: "Create a file with the given name and content.",
            parameters: { file_name: z.string(), content: z.string() },
            implementation: async ({ file_name, content }) => {
              const filePath = join(ctl.getWorkingDirectory(), file_name);
              if (existsSync(filePath)) {
                return "Error: File already exists.";
              }
              await writeFile(filePath, content, "utf-8");
              return "File created.";
            },
          });
          tools.push(createFileTool);

          return tools;
        }
```

The above tools provider defines a single tool called `create_file` that allows the model to create a file with a specified name and content inside the working directory. You can learn more about defining tools in [Tool Definition](../agent/tools).

Then register the tools provider in your plugin's `index.ts`:

```lms_code_snippet
  title: "src/index.ts"
  variants:
    TypeScript:
      language: typescript
      code: |
        // ... other imports ...
        import { toolsProvider } from "./toolsProvider";

        export async function main(context: PluginContext) {
          // ... other plugin setup code ...

          // Register the tools provider.
          context.withToolsProvider(toolsProvider); // <-- Register the tools provider

          // ... other plugin setup code ...
        }
```

Now, you can try to ask the LLM to create a file, and it should be able to do so using the tool you just created.

## Tips

- **Use Descriptive Names and Descriptions**: When defining tools, use descriptive names and detailed descriptions. This helps the model understand when and how to use each tool effectively.
- **Return Errors as Strings**: Sometimes, the model may make a mistake when calling a tool. In such cases, you can return an error message as a string. In most cases, the model will try to correct itself and call the tool again with the correct parameters.


#### Multiple Tools

> 


A tools provider can define multiple tools for the model to use. Simply create additional tool instances and add them to the tools array.

In the example below, we add a second tool to read the content of a file:

```lms_code_snippet
  title: "src/toolsProvider.ts"
  variants:
    TypeScript:
      language: typescript
      code: |
        import { tool, Tool, ToolsProviderController } from "@lmstudio/sdk";
        import { z } from "zod";
        import { existsSync } from "fs";
        import { readFile, writeFile } from "fs/promises";
        import { join } from "path";

        export async function toolsProvider(ctl: ToolsProviderController) {
          const tools: Tool[] = [];

          const createFileTool = tool({
            name: `create_file`,
            description: "Create a file with the given name and content.",
            parameters: { file_name: z.string(), content: z.string() },
            implementation: async ({ file_name, content }) => {
              const filePath = join(ctl.getWorkingDirectory(), file_name);
              if (existsSync(filePath)) {
                return "Error: File already exists.";
              }
              await writeFile(filePath, content, "utf-8");
              return "File created.";
            },
          });
          tools.push(createFileTool); // First tool

          const readFileTool = tool({
            name: `read_file`,
            description: "Read the content of a file with the given name.",
            parameters: { file_name: z.string() },
            implementation: async ({ file_name }) => {
              const filePath = join(ctl.getWorkingDirectory(), file_name);
              if (!existsSync(filePath)) {
                return "Error: File does not exist.";
              }
              const content = await readFile(filePath, "utf-8");
              return content;
            },
          });
          tools.push(readFileTool); // Second tool

          return tools; // Return the tools array
        }
```


#### Custom Configuration

> Add custom configuration options to your tools provider


You can add custom configuration options to your tools provider, so the user of plugin can customize the behavior without modifying the code.

In the example below, we will ask the user to specify a folder name, and we will create files inside that folder within the working directory.

First, add the config field to `config.ts`:

```lms_code_snippet
  title: "src/config.ts"
  variants:
    TypeScript:
      language: typescript
      code: |
        export const configSchematics = createConfigSchematics()
          .field(
            "folderName", // Key of the configuration field
            "string", // Type of the configuration field
            {
              displayName: "Folder Name",
              subtitle: "The name of the folder where files will be created.",
            },
            "default_folder", // Default value
          )
          .build();
```

```lms_info
In this example, we added the field to `configSchematics`, which is the "per-chat" configuration. If you want to add a global configuration field that is shared across different chats, you should add it under the section `globalConfigSchematics` in the same file.

Learn more about configurations in [Custom Configurations](../plugins/configurations).
```

Then, modify the tools provider to use the configuration value:

```lms_code_snippet
  title: "src/toolsProvider.ts"
  variants:
    TypeScript:
      language: typescript
      code: |
        import { tool, Tool, ToolsProviderController } from "@lmstudio/sdk";
        import { existsSync } from "fs";
        import { mkdir, writeFile } from "fs/promises";
        import { join } from "path";
        import { z } from "zod";
        import { configSchematics } from "./config";

        export async function toolsProvider(ctl: ToolsProviderController) {
          const tools: Tool[] = [];

          const createFileTool = tool({
            name: `create_file`,
            description: "Create a file with the given name and content.",
            parameters: { file_name: z.string(), content: z.string() },
            implementation: async ({ file_name, content }) => {
              // Read the config field
              const folderName = ctl.getPluginConfig(configSchematics).get("folderName");
              const folderPath = join(ctl.getWorkingDirectory(), folderName);

              // Ensure the folder exists
              await mkdir(folderPath, { recursive: true });

              // Create the file
              const filePath = join(folderPath, file_name);
              if (existsSync(filePath)) {
                return "Error: File already exists.";
              }
              await writeFile(filePath, content, "utf-8");
              return "File created.";
            },
          });
          tools.push(createFileTool); // First tool

          return tools; // Return the tools array
        }
```


#### Status Reports & Warnings

> 


Sometimes, a tool may take a long time to execute. In such cases, it will be helpful to provide status updates, so the user knows what is happening. In order times, you may want to warn the user about potential issues.

You can use `status` and `warn` methods on the second parameter of the tool's implementation function to send status updates and warnings.

The following example shows how to implement a tool that waits for a specified number of seconds, providing status updates and warnings if the wait time exceeds 10 seconds:

```lms_code_snippet
  title: "src/toolsProvider.ts"
  variants:
    TypeScript:
      language: typescript
      code: |
        import { tool, Tool, ToolsProviderController } from "@lmstudio/sdk";
        import { z } from "zod";

        export async function toolsProvider(ctl: ToolsProviderController) {
          const tools: Tool[] = [];

          const waitTool = tool({
            name: `wait`,
            description: "Wait for a specified number of seconds.",
            parameters: { seconds: z.number().min(1) },
            implementation: async ({ seconds }, { status, warn }) => {
              if (seconds > 10) {
                warn("The model asks to wait for more than 10 seconds.");
              }
              for (let i = 0; i < seconds; i++) {
                status(`Waiting... ${i + 1}/${seconds} seconds`);
                await new Promise((resolve) => setTimeout(resolve, 1000));
              }
            },
          });
          tools.push(waitTool);

          return tools; // Return the tools array
        }
```

Note status updates and warnings are only visible to the user. If you want the model to also see those messages, you should return them as part of the tool's return value.

## Handling Aborts

A prediction may be aborted by the user while your tool is still running. In such cases, you should handle the abort gracefully by handling the `AbortSignal` object passed as the second parameter to the tool's implementation function.

```lms_code_snippet
  title: "src/toolsProvider.ts"
  variants:
    TypeScript:
      language: typescript
      code: |
        import { tool, Tool, ToolsProviderController } from "@lmstudio/sdk";
        import { z } from "zod";

        export async function toolsProvider(ctl: ToolsProviderController) {
          const tools: Tool[] = [];

          const fetchTool = tool({
            name: `fetch`,
            description: "Fetch a URL using GET method.",
            parameters: { url: z.string() },
            implementation: async ({ url }, { signal }) => {
              const response = await fetch(url, {
                method: "GET",
                signal, // <-- Here, we pass the signal to fetch to allow cancellation
              });
              if (!response.ok) {
                return `Error: Failed to fetch ${url}: ${response.statusText}`;
              }
              const data = await response.text();
              return {
                status: response.status,
                headers: Object.fromEntries(response.headers.entries()),
                data: data.substring(0, 1000), // Limit to 1000 characters
              };
            },
          });
          tools.push(fetchTool);

          return tools;
        }
```

You can learn more about `AbortSignal` in the [MDN documentation](https://developer.mozilla.org/en-US/docs/Web/API/AbortSignal).


#### Handling Aborts

> Gracefully handle user-aborted tool executions in your tools provider


A prediction may be aborted by the user while your tool is still running. In such cases, you should handle the abort gracefully by handling the `AbortSignal` object passed as the second parameter to the tool's implementation function.

```lms_code_snippet
  title: "src/toolsProvider.ts"
  variants:
    TypeScript:
      language: typescript
      code: |
        import { tool, Tool, ToolsProviderController } from "@lmstudio/sdk";
        import { z } from "zod";

        export async function toolsProvider(ctl: ToolsProviderController) {
          const tools: Tool[] = [];

          const fetchTool = tool({
            name: `fetch`,
            description: "Fetch a URL using GET method.",
            parameters: { url: z.string() },
            implementation: async ({ url }, { signal }) => {
              const response = await fetch(url, {
                method: "GET",
                signal, // <-- Here, we pass the signal to fetch to allow cancellation
              });
              if (!response.ok) {
                return `Error: Failed to fetch ${url}: ${response.statusText}`;
              }
              const data = await response.text();
              return {
                status: response.status,
                headers: Object.fromEntries(response.headers.entries()),
                data: data.substring(0, 1000), // Limit to 1000 characters
              };
            },
          });
          tools.push(fetchTool);

          return tools;
        }
```

You can learn more about `AbortSignal` in the [MDN documentation](https://developer.mozilla.org/en-US/docs/Web/API/AbortSignal).


### prompt-preprocessor

### Introduction

> Writing prompt preprocessors for LM Studio plugins using TypeScript


Prompt Preprocessor is a function that is called upon the user hitting the "Send" button. It receives the user input and can modify it before it reaches the model. If multiple prompt preprocessors are registered, they will be chained together, with each one receiving the output of the previous one.

The modified result will be saved in the chat history, meaning that even if your plugin is disabled afterwards, the modified input will still be used.

Prompt preprocessors will only be triggered for the current user input. It will not be triggered for previous messages in the chat history even if they were not preprocessed.

Prompt preprocessors takes in a `ctl` object for controlling the preprocessing and a `userMessage` it needs to preprocess. It returns either a string or a message object which will replace the user message.

### Examples

The following are some plugins that make use of prompt preprocessors:

- [lmstudio/rag-v1](https://lmstudio.ai/lmstudio/rag-v1)

  Retrieval Augmented Generation (RAG) for LM Studio. This is the plugin that gives document handling capabilities to LM Studio.


#### Examples

> 


### Example: Inject Current Time

The following is an example preprocessor that injects the current time before each user message.

```lms_code_snippet
  title: "src/promptPreprocessor.ts"
  variants:
    TypeScript:
      language: typescript
      code: |
        import { type PromptPreprocessorController, type ChatMessage } from "@lmstudio/sdk";
        export async function preprocess(ctl: PromptPreprocessorController, userMessage: ChatMessage) {
          const textContent = userMessage.getText();
          const transformed = `Current time: ${new Date().toString()}\n\n${textContent}`;
          return transformed;
        }
```

### Example: Replace Trigger Words

Another example you can do it with simple text only processing is by replacing certain trigger words. For example, you can replace a `@init` trigger with a special initialization message.

```lms_code_snippet
  title: "src/promptPreprocessor.ts"
  variants:
    TypeScript:
      language: typescript
      code: |
        import { type PromptPreprocessorController, type ChatMessage, text } from "@lmstudio/sdk";

        const mySpecialInstructions = text`
          Here are some special instructions...
        `;

        export async function preprocess(ctl: PromptPreprocessorController, userMessage: ChatMessage) {
          const textContent = userMessage.getText();
          const transformed = textContent.replaceAll("@init", mySpecialInstructions);
          return transformed;
        }
```


#### Custom Configuration

> 


You can access custom configurations via `ctl.getPluginConfig` and `ctl.getGlobalPluginConfig`. See [Custom Configurations](./configurations) for more details.

The following is an example of how you can make the `specialInstructions` and `triggerWord` configurable:

First, add the config field to `config.ts`:

```lms_code_snippet
  title: "src/config.ts"
  variants:
    TypeScript:
      language: typescript
      code: |
        import { createConfigSchematics } from "@lmstudio/sdk";
        export const configSchematics = createConfigSchematics()
          .field(
            "specialInstructions",
            "string",
            {
              displayName: "Special Instructions",
              subtitle: "Special instructions to be injected when the trigger word is found.",
            },
            "Here is some default special instructions.",
          )
          .field(
            "triggerWord",
            "string",
            {
              displayName: "Trigger Word",
              subtitle: "The word that will trigger the special instructions.",
            },
            "@init",
          )
          .build();
```

```lms_info
In this example, we added the field to `configSchematics`, which is the "per-chat" configuration. If you want to add a global configuration field that is shared across different chats, you should add it under the section `globalConfigSchematics` in the same file.

Learn more about configurations in [Custom Configurations](../plugins/configurations).
```

Then, modify the prompt preprocessor to use the configuration:

```lms_code_snippet
  title: "src/promptPreprocessor.ts"
  variants:
    TypeScript:
      language: typescript
      code: |
        import { type PromptPreprocessorController, type ChatMessage } from "@lmstudio/sdk";
        import { configSchematics } from "./config";

        export async function preprocess(ctl: PromptPreprocessorController, userMessage: ChatMessage) {
          const textContent = userMessage.getText();
          const pluginConfig = ctl.getPluginConfig(configSchematics);

          const triggerWord = pluginConfig.get("triggerWord");
          const specialInstructions = pluginConfig.get("specialInstructions");

          const transformed = textContent.replaceAll(triggerWord, specialInstructions);
          return transformed;
        }
```


#### Custom Status Report

> 


Depending on the task, the prompt preprocessor may take some time to complete, for example, it may need to fetch some data from the internet or perform some heavy computation. In such cases, you can report the status of the preprocessing using `ctl.setStatus`.

```lms_code_snippet
  title: "src/promptPreprocessor.ts"
  variants:
    TypeScript:
      language: typescript
      code: |
        const status = ctl.createStatus({
          status: "loading",
          text: "Preprocessing.",
        });
```

You can update the status at any time by calling `status.setState`.

```lms_code_snippet
  title: "src/promptPreprocessor.ts"
  variants:
    TypeScript:
      language: typescript
      code: |
        status.setState({
          status: "done",
          text: "Preprocessing done.",
        })
```

You can even add sub status to the status:

```lms_code_snippet
  title: "src/promptPreprocessor.ts"
  variants:
    TypeScript:
      language: typescript
      code: |
        const subStatus = status.addSubStatus({
          status: "loading",
          text: "I am a sub status."
        });
```


#### Handling Aborts

> 


A prediction may be aborted by the user while your generator is still running. In such cases, you should handle the abort gracefully by handling the `ctl.abortSignal`.

You can learn more about `AbortSignal` in the [MDN documentation](https://developer.mozilla.org/en-US/docs/Web/API/AbortSignal).


### generator

### Introduction

> Writing generators for LM Studio plugins using TypeScript


Generators are replacement for local LLMs. They act like a token source. When a plugin with a generator is used, LM Studio will no longer use the local model to generate text. The generator will be used instead.

Generators are useful for implementing adapters for external models, such as using a remote LM Studio instance or other online models.

If a plugin contains a generator, it will no longer show up in the plugins list. Instead, it will show up in the model dropdown and act as a model. If your plugins contains [Tools Provider](./tools-providers.md) or [Prompt Preprocessor](./prompt-preprocessors.md), they will be used when your generator is being selected.

## Examples

The following are some plugins that make use of generators:

- [lmstudio/remote-lmstudio](https://lmstudio.ai/lmstudio/remote-lmstudio)

  Basic support for using a remote LM Studio instance to generate text.

- [lmstudio/openai-compat-endpoint](https://lmstudio.ai/lmstudio/openai-compat-endpoint)

  Use any OpenAI-compatible API in LM Studio.


#### Text-only Generators

> 


Generators take in the the generator controller and the current conversation state, start the generation, and then report the generated text using the `ctl.fragmentGenerated` method.

The following is an example of a simple generator that echos back the last user message with 200 ms delay between each word:

```lms_code_snippet
  title: "src/toolsProvider.ts"
  variants:
    TypeScript:
      language: typescript
      code: |
        import { Chat, GeneratorController } from "@lmstudio/sdk";

        export async function generate(ctl: GeneratorController, chat: Chat) {
          // Just echo back the last message
          const lastMessage = chat.at(-1).getText();
          // Split the last message into words
          const words = lastMessage.split(/(?= )/);
          for (const word of words) {
            ctl.fragmentGenerated(word); // Send each word as a fragment
            ctl.abortSignal.throwIfAborted(); // Allow for cancellation
            await new Promise((resolve) => setTimeout(resolve, 200)); // Simulate some processing time
          }
        }
```

## Custom Configurations

You can access custom configurations via `ctl.getPluginConfig` and `ctl.getGlobalPluginConfig`. See [Custom Configurations](./configurations) for more details.

## Handling Aborts

A prediction may be aborted by the user while your generator is still running. In such cases, you should handle the abort gracefully by handling the `ctl.abortSignal`.

You can learn more about `AbortSignal` in the [MDN documentation](https://developer.mozilla.org/en-US/docs/Web/API/AbortSignal).


#### Tool calling generators

> 


To enable tool use, it is slightly more involved. To see a comprehensive example that adapts OpenAI API, see the [openai-compat-endpoint plugin](https://lmstudio.ai/lmstudio/openai-compat-endpoint).

You can read the definition of tools available using `ctl.getToolDefinitions()`. For example, if you are making an online model adapter, you need to pass the tool definition to the model.

Once the model starts to make tool calls, you need to tell LM Studio about those calls.

Use `ctl.toolCallGenerationStarted` to report the start of a tool call generation (i.e. the model starts to generate a tool call).

Use `ctl.toolCallGenerationEnded` to report a successful generation of a tool call or use `ctl.toolCallGenerationFailed` to report a failed generation of a tool call.

Optionally, you can also `ctl.toolCallGenerationNameReceived` to eagerly report the name of the tool being called once that is available. You can also use `ctl.toolCallGenerationArgumentFragmentGenerated` to report fragments of the tool call arguments as they are generated. These two methods are useful for providing better user experience, but are not strictly necessary.

Overall, your generator must call these ctl methods in the following order:

1. 0 - N calls to `ctl.fragmentGenerated` to report the generated non-tool-call text fragments.
2. For each tool call:
   1. Call `ctl.toolCallGenerationStarted` to indicate the start of a tool call generation.
   2. (Optionally) Call `ctl.toolCallGenerationNameReceived` to report the name of the tool being called.
   3. (Optionally) Call any times of `ctl.toolCallGenerationArgumentFragmentGenerated` to report the generated fragments of the tool call arguments.
   4. Call either `ctl.toolCallGenerationEnded` to report a successful generation of the tool call or `ctl.toolCallGenerationFailed` to report a failed generation of the tool call.
   5. If the model generates more text between/after the tool call, 0 - N calls to `ctl.fragmentGenerated` to report the generated non-tool-call text fragments. (This should not happen normally, but it is technically possible for some smaller models to do this. **Critically: this is not the same as model receiving the tool results and continuing the conversation. This is just model refusing to stop talking after made a tool request - the tool result is not available to the model yet.** When multi-round prediction happens, i.e. the model actually receives the tool call, your generator function will be called again with the updated conversation state.)

Some API formats may report the tool name together with the beginning of the tool call generation, in which case you can call `ctl.toolCallGenerationNameReceived` immediately after `ctl.toolCallGenerationStarted`.

Some API formats may not have incremental tool call updates (i.e. the entire tool call request is given at once), in which case you can just call `ctl.toolCallGenerationStarted` then immediately `ctl.toolCallGenerationEnded`.


### custom-configuration

### Introduction

> Add custom configurations to LM Studio plugins using TypeScript


LM Studio plugins support custom configurations. That is, you can define a configuration schema and LM Studio will present a UI to the user so they can configure your plugin without having to edit any code.

There are two types of configurations:

- **Per-chat configuration**: tied to a specific chat. Different chats can have different configurations. Most configurations that affects the behavior of the plugin should be of this type.
- **Global configuration**: apply to _all_ chats and are shared across the application. This is useful for global settings such as API keys.

## Types of Configurations

You can define configurations in TypeScript using the `createConfigSchematics` function from the `@lmstudio/sdk` package. This function allows you to define fields with various types and options.

Supported types include:

- `string`: A text input field.
- `numeric`: A number input field with optional validation and slider UI.
- `boolean`: A checkbox or toggle input field.
- `stringArray`: An array of string values with configurable constraints.
- `select`: A dropdown selection field with predefined options.

See the [Defining New Fields](./custom-configuration/defining-new-fields) section for more details on how to define these fields.

## Examples

The following are some plugins that make use of custom configurations

- [lmstudio/wikipedia](https://lmstudio.ai/lmstudio/wikipedia)

  Gives the LLM tools to search and read Wikipedia articles.

- [lmstudio/openai-compat-endpoint](https://lmstudio.ai/lmstudio/openai-compat-endpoint)

  Use any OpenAI-compatible API in LM Studio.


#### `config.ts` File

> 


By default, the plugin scaffold will create a `config.ts` file in the `src/` directory which will contain the schematics of the configurations. If the files does not exist, you can create it manually:

```lms_code_snippet
  title: "src/toolsProvider.ts"
  variants:
    TypeScript:
      language: typescript
      code: |
        import { createConfigSchematics } from "@lmstudio/sdk";

        export const configSchematics = createConfigSchematics()
          .field(
            "myCustomField", // The key of the field.
            "numeric", // Type of the field.
            // Options for the field. Different field types will have different options.
            {
              displayName: "My Custom Field",
              hint: "This is my custom field. Doesn't do anything special.",
              slider: { min: 0, max: 100, step: 1 }, // Add a slider to the field.
            },
            80, // Default Value
          )
          // You can add more fields by chaining the field method.
          // For example:
          //   .field("anotherField", ...)
          .build();

        export const globalConfigSchematics = createConfigSchematics()
          .field(
            "myGlobalCustomField", // The key of the field.
            "string",
            {
              displayName: "My Global Custom Field",
              hint: "This is my global custom field. Doesn't do anything special.",
            },
            "default value", // Default Value
          )
          // You can add more fields by chaining the field method.
          // For example:
          //  .field("anotherGlobalField", ...)
          .build();
```

If you've added your config schematics manual, you will also need to register the configurations in your plugin's `index.ts` file.

This is done by calling `context.withConfigSchematics(configSchematics)` and `context.withGlobalConfigSchematics(globalConfigSchematics)` in the `main` function of your plugin.

```lms_code_snippet
  title: "src/index.ts"
  variants:
    TypeScript:
      language: typescript
      code: |
        // ... other imports ...
        import { toolsProvider } from "./toolsProvider";

        export async function main(context: PluginContext) {
          // ... other plugin setup code ...

          // Register the configuration schematics.
          context.withConfigSchematics(configSchematics);
          // Register the global configuration schematics.
          context.withGlobalConfigSchematics(globalConfigSchematics);

          // ... other plugin setup code ...
        }
```


#### Accessing Configuration

> 


You can access the configuration using the method `ctl.getPluginConfig(configSchematics)` and `ctl.getGlobalConfig(globalConfigSchematics)` respectively.

For example, here is how to access the config within the promptPreprocessor:

```lms_code_snippet
  title: "src/promptPreprocessor.ts"
  variants:
    TypeScript:
      language: typescript
      code: |
        import { type PreprocessorController, type ChatMessage } from "@lmstudio/sdk";
        import { configSchematics } from "./config";

        export async function preprocess(ctl: PreprocessorController, userMessage: ChatMessage) {
          const pluginConfig = ctl.getPluginConfig(configSchematics);
          const myCustomField = pluginConfig.get("myCustomField");

          const globalPluginConfig = ctl.getGlobalPluginConfig(configSchematics);
          const globalMyCustomField = globalPluginConfig.get("myCustomField");

          return (
            `${userMessage.getText()},` +
            `myCustomField: ${myCustomField}, ` +
            `globalMyCustomField: ${globalMyCustomField}`
          );
        }
```


#### Defining New Fields

> 


We support the following field types:

- `string`: A text input field.

  ```lms_code_snippet
    variants:
      TypeScript:
        language: typescript
        code: |
          // ... other fields ...
          .field(
            "stringField", // The key of the field.
            "string", // Type of the field.
            {
              displayName: "A string field",
              subtitle: "Subtitle", // Optional subtitle for the field. (Show below the field)
              hint: "Hint", // Optional hint for the field. (Show on hover)
              isParagraph: false, // Whether to show a large text input area for this field.
              isProtected: false, // Whether the value should be obscured in the UI (e.g., for passwords).
              placeholder: "Placeholder text", // Optional placeholder text for the field.
            },
            "default value", // Default Value
          )
          // ... other fields ...
  ```

- `numeric`: A number input field with optional validation and slider UI.

  ```lms_code_snippet
    variants:
      TypeScript:
        language: typescript
        code: |
          // ... other fields ...
          .field(
            "numberField", // The key of the field.
            "numeric", // Type of the field.
            {
              displayName: "A number field",
              subtitle: "Subtitle for", // Optional subtitle for the field. (Show below the field)
              hint: "Hint for number field", // Optional hint for the field. (Show on hover)
              int: false, // Whether the field should accept only integer values.
              min: 0, // Minimum value for the field.
              max: 100, // Maximum value for the field.
              slider: {
                // If present, configurations for the slider UI
                min: 0, // Minimum value for the slider.
                max: 100, // Maximum value for the slider.
                step: 1, // Step value for the slider.
              },
            },
            42, // Default Value
          )
          // ... other fields ...
  ```

- `boolean`: A checkbox or toggle input field.

  ```lms_code_snippet
    variants:
      TypeScript:
        language: typescript
        code: |
          // ... other fields ...
          .field(
            "booleanField", // The key of the field.
            "boolean", // Type of the field.
            {
              displayName: "A boolean field",
              subtitle: "Subtitle", // Optional subtitle for the field. (Show below the field)
              hint: "Hint", // Optional hint for the field. (Show on hover)
            },
            true, // Default Value
          )
          // ... other fields ...
  ```

- `stringArray`: An array of string values with configurable constraints.

  ```lms_code_snippet
    variants:
      TypeScript:
        language: typescript
        code: |
          // ... other fields ...
          .field(
            "stringArrayField",
            "stringArray",
            {
              displayName: "A string array field",
              subtitle: "Subtitle", // Optional subtitle for the field. (Show below the field)
              hint: "Hint", // Optional hint for the field. (Show on hover)
              allowEmptyStrings: true, // Whether to allow empty strings in the array.
              maxNumItems: 5, // Maximum number of items in the array.
            },
            ["default", "values"], // Default Value
          )
          // ... other fields ...
  ```

- `select`: A dropdown selection field with predefined options.

  ```lms_code_snippet
    variants:
      TypeScript:
        language: typescript
        code: |
          // ... other fields ...
          .field(
            "selectField",
            "select",
            {
              displayName: "A select field",
              options: [
                { value: "option1", displayName: "Option 1" },
                { value: "option2", displayName: "Option 2" },
                { value: "option3", displayName: "Option 3" },
              ],
              subtitle: "Subtitle", // Optional subtitle for the field. (Show below the field)
              hint: "Hint", // Optional hint for the field. (Show on hover)
            },
            "option1", // Default Value
          )
          // ... other fields ...
  ```


### publish-plugins

### Sharing Plugins

> How to publish your LM Studio plugins so they can be used by others


To share publish your LM Studio plugin, open the plugin directory in a terminal and run:

```bash
lms push
```

This command will package your plugin and upload it to the LM Studio Hub. You can use this command to create new plugins or update existing ones.

### Changing Plugin Names

If you wish to change the name of the plugin, you can do so by editing the `manifest.json` file in the root of your plugin directory. Look for the `name` field and update it to your desired plugin name. Note the `name` must be kebab-case.

When you `lms push` the plugin, it will be treated as a new plugin if the name has changed. You can delete the old plugin from the LM Studio Hub if you no longer need it.

### Publishing Plugins to an Organization

If you are in an organization and wish to publish the plugin to the organization, you can do so by editing the `manifest.json` file in the root of your plugin directory. Look for the `owner` field and set it to the name of your organization. When you run `lms push`, the plugin will be published to the organization instead of your personal account.

### Private Plugins

If your account supports private plugins, you can publish your plugins privately by using the `--private` flag when running `lms push`:

```bash
lms push --private
```

Private artifact is in test. Get in touch if you are interested.


## embedding

## Embedding

> Generate text embeddings from input text


Generate embeddings for input text. Embeddings are vector representations of text that capture semantic meaning. Embeddings are a building block for RAG (Retrieval-Augmented Generation) and other similarity-based tasks.

### Prerequisite: Get an Embedding Model

If you don't yet have an embedding model, you can download a model like `nomic-ai/nomic-embed-text-v1.5` using the following command:

```bash
lms get nomic-ai/nomic-embed-text-v1.5
```

## Create Embeddings

To convert a string to a vector representation, pass it to the `embed` method on the corresponding embedding model handle.

```lms_code_snippet
  title: "index.ts"
  variants:
    TypeScript:
      language: typescript
      code: |
        import { LMStudioClient } from "@lmstudio/sdk";
        const client = new LMStudioClient();

        const model = await client.embedding.model("nomic-embed-text-v1.5");

        const { embedding } = await model.embed("Hello, world!");
```


## tokenization

## Tokenization

> Tokenize text using a model's tokenizer


Models use a tokenizer to internally convert text into "tokens" they can deal with more easily. LM Studio exposes this tokenizer for utility.

## Tokenize

You can tokenize a string with a loaded LLM or embedding model using the SDK. In the below examples, `llm` can be replaced with an embedding model `emb`.

```lms_code_snippet
  variants:
    TypeScript:
      language: typescript
      code: |
        import { LMStudioClient } from "@lmstudio/sdk";

        const client = new LMStudioClient();
        const model = await client.llm.model();

        const tokens = await model.tokenize("Hello, world!");

        console.info(tokens); // Array of token IDs.
```

## Count tokens

If you only care about the number of tokens, you can use the `.countTokens` method instead.

```lms_code_snippet
  variants:
    TypeScript:
      language: typescript
      code: |
        const tokenCount = await model.countTokens("Hello, world!");
        console.info("Token count:", tokenCount);
```

### Example: Count Context

You can determine if a given conversation fits into a model's context by doing the following:

1. Convert the conversation to a string using the prompt template.
2. Count the number of tokens in the string.
3. Compare the token count to the model's context length.

```lms_code_snippet
  variants:
    TypeScript:
      language: typescript
      code: |
        import { Chat, type LLM, LMStudioClient } from "@lmstudio/sdk";

        async function doesChatFitInContext(model: LLM, chat: Chat) {
          // Convert the conversation to a string using the prompt template.
          const formatted = await model.applyPromptTemplate(chat);
          // Count the number of tokens in the string.
          const tokenCount = await model.countTokens(formatted);
          // Get the current loaded context length of the model
          const contextLength = await model.getContextLength();
          return tokenCount < contextLength;
        }

        const client = new LMStudioClient();
        const model = await client.llm.model();

        const chat = Chat.from([
          { role: "user", content: "What is the meaning of life." },
          { role: "assistant", content: "The meaning of life is..." },
          // ... More messages
        ]);

        console.info("Fits in context:", await doesChatFitInContext(model, chat));
```

<!-- ### Context length comparisons

The below examples check whether a conversation is over a LLM's context length
(replace `llm` with `emb` to check for an embedding model).

```lms_code_snippet
  variants:
    TypeScript:
      language: typescript
      code: |
        import { LMStudioClient, Chat } from "@lmstudio/sdk";

        const client = new LMStudioClient();
        const llm = await client.llm.model();

        // To check for a string, simply tokenize
        var tokens = await llm.tokenize("Hello, world!");

        // To check for a Chat, apply the prompt template first
        const chat = Chat.createEmpty().withAppended("user", "Hello, world!");
        const templatedChat = await llm.applyPromptTemplate(chat);
        tokens = await llm.tokenize(templatedChat);

        // If the prompt's length in tokens is less than the context length, you're good!
        const contextLength = await llm.getContextLength()
        const isOkay = (tokens.length < contextLength)
``` -->


## manage-models

### List Local Models

> APIs to list the available models in a given local environment


You can iterate through locally available models using the `listLocalModels` method.

## Available Model on the Local Machine

`listLocalModels` lives under the `system` namespace of the `LMStudioClient` object.

```lms_code_snippet
  variants:
    TypeScript:
      language: typescript
      code: |
        import { LMStudioClient } from "@lmstudio/sdk";
        const client = new LMStudioClient();

        console.info(await client.system.listDownloadedModels());
```

This will give you results equivalent to using [`lms ls`](../../cli/ls) in the CLI.

### Example output:

```json
[
  {
    "type": "llm",
    "modelKey": "qwen2.5-7b-instruct",
    "format": "gguf",
    "displayName": "Qwen2.5 7B Instruct",
    "path": "lmstudio-community/Qwen2.5-7B-Instruct-GGUF/Qwen2.5-7B-Instruct-Q4_K_M.gguf",
    "sizeBytes": 4683073952,
    "paramsString": "7B",
    "architecture": "qwen2",
    "vision": false,
    "trainedForToolUse": true,
    "maxContextLength": 32768
  },
  {
    "type": "embedding",
    "modelKey": "text-embedding-nomic-embed-text-v1.5@q4_k_m",
    "format": "gguf",
    "displayName": "Nomic Embed Text v1.5",
    "path": "nomic-ai/nomic-embed-text-v1.5-GGUF/nomic-embed-text-v1.5.Q4_K_M.gguf",
    "sizeBytes": 84106624,
    "architecture": "nomic-bert",
    "maxContextLength": 2048
  }
]
```

<!-- Learn more about the `client.system` namespace in the [System API Reference](../api-reference/system-namespace). -->


### List Loaded Models

> Query which models are currently loaded


You can iterate through models loaded into memory using the `listLoaded` method. This method lives under the `llm` and `embedding` namespaces of the `LMStudioClient` object.

## List Models Currently Loaded in Memory

This will give you results equivalent to using [`lms ps`](../../cli/ps) in the CLI.

```lms_code_snippet
  variants:
    TypeScript:
      language: typescript
      code: |
        import { LMStudioClient } from "@lmstudio/sdk";

        const client = new LMStudioClient();

        const llmOnly = await client.llm.listLoaded();
        const embeddingOnly = await client.embedding.listLoaded();
```

<!-- Learn more about `client.llm` namespace in the [API Reference](../api-reference/llm-namespace). -->


### Manage Models in Memory

> APIs to load, access, and unload models from memory


AI models are huge. It can take a while to load them into memory. LM Studio's SDK allows you to precisely control this process.

**Most commonly:**

- Use `.model()` to get any currently loaded model
- Use `.model("model-key")` to use a specific model

**Advanced (manual model management):**

- Use `.load("model-key")` to load a new instance of a model
- Use `model.unload()` to unload a model from memory

## Get the Current Model with `.model()`

If you already have a model loaded in LM Studio (either via the GUI or `lms load`), you can use it by calling `.model()` without any arguments.

```lms_code_snippet
  variants:
    TypeScript:
      language: typescript
      code: |
        import { LMStudioClient } from "@lmstudio/sdk";
        const client = new LMStudioClient();

        const model = await client.llm.model();
```

## Get a Specific Model with `.model("model-key")`

If you want to use a specific model, you can provide the model key as an argument to `.model()`.

#### Get if Loaded, or Load if not

Calling `.model("model-key")` will load the model if it's not already loaded, or return the existing instance if it is.

```lms_code_snippet
  variants:
    TypeScript:
      language: typescript
      code: |
        import { LMStudioClient } from "@lmstudio/sdk";
        const client = new LMStudioClient();

        const model = await client.llm.model("qwen/qwen3-4b-2507");
```

<!-- Learn more about the `.model()` method and the parameters it accepts in the [API Reference](../api-reference/model). -->

## Load a New Instance of a Model with `.load()`

Use `load()` to load a new instance of a model, even if one already exists. This allows you to have multiple instances of the same or different models loaded at the same time.

```lms_code_snippet
  variants:
    TypeScript:
      language: typescript
      code: |
        import { LMStudioClient } from "@lmstudio/sdk";
        const client = new LMStudioClient();

        const llama = await client.llm.load("qwen/qwen3-4b-2507");
        const another_llama = await client.llm.load("qwen/qwen3-4b-2507", {
          identifier: "second-llama"
        });
```

<!-- Learn more about the `.load()` method and the parameters it accepts in the [API Reference](../api-reference/load). -->

### Note about Instance Identifiers

If you provide an instance identifier that already exists, the server will throw an error.
So if you don't really care, it's safer to not provide an identifier, in which case
the server will generate one for you. You can always check in the server tab in LM Studio, too!

## Unload a Model from Memory with `.unload()`

Once you no longer need a model, you can unload it by simply calling `unload()` on its handle.

```lms_code_snippet
  variants:
    TypeScript:
      language: typescript
      code: |
        import { LMStudioClient } from "@lmstudio/sdk";

        const client = new LMStudioClient();

        const model = await client.llm.model();
        await model.unload();
```

## Set Custom Load Config Parameters

You can also specify the same load-time configuration options when loading a model, such as Context Length and GPU offload.

See [load-time configuration](../llm-prediction/parameters) for more.

## Set an Auto Unload Timer (TTL)

You can specify a _time to live_ for a model you load, which is the idle time (in seconds)
after the last request until the model unloads. See [Idle TTL](/docs/api/ttl-and-auto-evict) for more on this.

```lms_code_snippet
  variants:
    "Using .load":
      language: typescript
      code: |
        import { LMStudioClient } from "@lmstudio/sdk";

        const client = new LMStudioClient();

        const model = await client.llm.load("qwen/qwen3-4b-2507", {
          ttl: 300, // 300 seconds
        });
    "Using .model":
      language: typescript
      code: |
        import { LMStudioClient } from "@lmstudio/sdk";

        const client = new LMStudioClient();

        const model = await client.llm.model("qwen/qwen3-4b-2507", {
          // Note: specifying ttl in `.model` will only set the TTL for the model if the model is
          // loaded from this call. If the model was already loaded, the TTL will not be updated.
          ttl: 300, // 300 seconds
        });
```


## api-reference

### `LLMLoadModelConfig`

> API Reference for `LLMLoadModelConfig`


### Parameters

```lms_params
- name: gpu
  description: |
    How to distribute the work to your GPUs. See {@link GPUSetting} for more information.
  public: true
  type: GPUSetting
  optional: true

- name: contextLength
  description: |
    The size of the context length in number of tokens. This will include both the prompts and the
    responses. Once the context length is exceeded, the value set in
    {@link LLMPredictionConfigBase#contextOverflowPolicy} is used to determine the behavior.

    See {@link LLMContextOverflowPolicy} for more information.
  type: number
  optional: true

- name: ropeFrequencyBase
  description: |
    Custom base frequency for rotary positional embeddings (RoPE).

    This advanced parameter adjusts how positional information is embedded in the model's
    representations. Increasing this value may enable better performance at high context lengths by
    modifying how the model processes position-dependent information.
  type: number
  optional: true

- name: ropeFrequencyScale
  description: |
    Scaling factor for RoPE (Rotary Positional Encoding) frequency.

    This factor scales the effective context window by modifying how positional information is
    encoded. Higher values allow the model to handle longer contexts by making positional encoding
    more granular, which can be particularly useful for extending a model beyond its original
    training context length.
  type: number
  optional: true

- name: evalBatchSize
  description: |
    Number of input tokens to process together in a single batch during evaluation.

    Increasing this value typically improves processing speed and throughput by leveraging
    parallelization, but requires more memory. Finding the optimal batch size often involves
    balancing between performance gains and available hardware resources.
  type: number
  optional: true

- name: flashAttention
  description: |
    Enables Flash Attention for optimized attention computation.

    Flash Attention is an efficient implementation that reduces memory usage and speeds up
    generation by optimizing how attention mechanisms are computed. This can significantly
    improve performance on compatible hardware, especially for longer sequences.
  type: boolean
  optional: true

- name: keepModelInMemory
  description: |
    When enabled, prevents the model from being swapped out of system memory.

    This option reserves system memory for the model even when portions are offloaded to GPU,
    ensuring faster access times when the model needs to be used. Improves performance
    particularly for interactive applications, but increases overall RAM requirements.
  type: boolean
  optional: true

- name: seed
  description: |
    Random seed value for model initialization to ensure reproducible outputs.

    Setting a specific seed ensures that random operations within the model (like sampling)
    produce the same results across different runs, which is important for reproducibility
    in testing and development scenarios.
  type: number
  optional: true

- name: useFp16ForKVCache
  description: |
    When enabled, stores the key-value cache in half-precision (FP16) format.

    This option significantly reduces memory usage during inference by using 16-bit floating
    point numbers instead of 32-bit for the attention cache. While this may slightly reduce
    numerical precision, the impact on output quality is generally minimal for most applications.
  type: boolean
  optional: true

- name: tryMmap
  description: |
    Attempts to use memory-mapped (mmap) file access when loading the model.

    Memory mapping can improve initial load times by mapping model files directly from disk to
    memory, allowing the operating system to handle paging. This is particularly beneficial for
    quick startup, but may reduce performance if the model is larger than available system RAM,
    causing frequent disk access.
  type: boolean
  optional: true

- name: numExperts
  description: |
    Specifies the number of experts to use for models with Mixture of Experts (MoE) architecture.

    MoE models contain multiple "expert" networks that specialize in different aspects of the task.
    This parameter controls how many of these experts are active during inference, affecting both
    performance and quality of outputs. Only applicable for models designed with the MoE architecture.
  type: number
  optional: true

- name: llamaKCacheQuantizationType
  description: |
    Quantization type for the Llama model's key cache.

    This option determines the precision level used to store the key component of the attention
    mechanism's cache. Lower precision values (e.g., 4-bit or 8-bit quantization) significantly
    reduce memory usage during inference but may slightly impact output quality. The effect varies
    between different models, with some being more robust to quantization than others.

    Set to false to disable quantization and use full precision.
  type: LLMLlamaCacheQuantizationType | false
  optional: true

- name: llamaVCacheQuantizationType
  description: |
    Quantization type for the Llama model's value cache.

    Similar to the key cache quantization, this option controls the precision used for the value
    component of the attention mechanism's cache. Reducing precision saves memory but may affect
    generation quality. This option requires Flash Attention to be enabled to function properly.

    Different models respond differently to value cache quantization, so experimentation may be
    needed to find the optimal setting for a specific use case. Set to false to disable quantization.
  type: LLMLlamaCacheQuantizationType | false
  optional: true
```


### `LLMPredictionConfigInput`

> 


### Fields

```lms_params
- name: "maxTokens"
  type: "number | false"
  optional: true
  description: "Number of tokens to predict at most. If set to false, the model will predict as many tokens as it wants.\n\nWhen the prediction is stopped because of this limit, the `stopReason` in the prediction stats will be set to `maxPredictedTokensReached`."

- name: "temperature"
  type: "number"
  optional: true
  description: "The temperature parameter for the prediction model. A higher value makes the predictions more random, while a lower value makes the predictions more deterministic. The value should be between 0 and 1."

- name: "stopStrings"
  type: "Array<string>"
  optional: true
  description: "An array of strings. If the model generates one of these strings, the prediction will stop.\n\nWhen the prediction is stopped because of this limit, the `stopReason` in the prediction stats will be set to `stopStringFound`."

- name: "toolCallStopStrings"
  type: "Array<string>"
  optional: true
  description: "An array of strings. If the model generates one of these strings, the prediction will stop with the `stopReason` `toolCalls`."

- name: "contextOverflowPolicy"
  type: "LLMContextOverflowPolicy"
  optional: true
  description: "The behavior for when the generated tokens length exceeds the context window size. The allowed values are:\n\n- `stopAtLimit`: Stop the prediction when the generated tokens length exceeds the context window size. If the generation is stopped because of this limit, the `stopReason` in the prediction stats will be set to `contextLengthReached`\n- `truncateMiddle`: Keep the system prompt and the first user message, truncate middle.\n- `rollingWindow`: Maintain a rolling window and truncate past messages."

- name: "structured"
  type: "ZodType<TStructuredOutputType> | LLMStructuredPredictionSetting"
  optional: true
  description: "Configures the model to output structured JSON data that follows a specific schema defined using Zod.\n\nWhen you provide a Zod schema, the model will be instructed to generate JSON that conforms to that schema rather than free-form text.\n\nThis is particularly useful for extracting specific data points from model responses or when you need the output in a format that can be directly used by your application."

- name: "topKSampling"
  type: "number"
  optional: true
  description: "Controls token sampling diversity by limiting consideration to the K most likely next tokens.\n\nFor example, if set to 40, only the 40 tokens with the highest probabilities will be considered for the next token selection. A lower value (e.g., 20) will make the output more focused and conservative, while a higher value (e.g., 100) allows for more creative and diverse outputs.\n\nTypical values range from 20 to 100."

- name: "repeatPenalty"
  type: "number | false"
  optional: true
  description: "Applies a penalty to repeated tokens to prevent the model from getting stuck in repetitive patterns.\n\nA value of 1.0 means no penalty. Values greater than 1.0 increase the penalty. For example, 1.2 would reduce the probability of previously used tokens by 20%. This is particularly useful for preventing the model from repeating phrases or getting stuck in loops.\n\nSet to false to disable the penalty completely."

- name: "minPSampling"
  type: "number | false"
  optional: true
  description: "Sets a minimum probability threshold that a token must meet to be considered for generation.\n\nFor example, if set to 0.05, any token with less than 5% probability will be excluded from consideration. This helps filter out unlikely or irrelevant tokens, potentially improving output quality.\n\nValue should be between 0 and 1. Set to false to disable this filter."

- name: "topPSampling"
  type: "number | false"
  optional: true
  description: "Implements nucleus sampling by only considering tokens whose cumulative probabilities reach a specified threshold.\n\nFor example, if set to 0.9, the model will consider only the most likely tokens that together add up to 90% of the probability mass. This helps balance between diversity and quality by dynamically adjusting the number of tokens considered based on their probability distribution.\n\nValue should be between 0 and 1. Set to false to disable nucleus sampling."

- name: "xtcProbability"
  type: "number | false"
  optional: true
  description: "Controls how often the XTC (Exclude Top Choices) sampling technique is applied during generation.\n\nXTC sampling can boost creativity and reduce clichÃ©s by occasionally filtering out common tokens. For example, if set to 0.3, there's a 30% chance that XTC sampling will be applied when generating each token.\n\nValue should be between 0 and 1. Set to false to disable XTC completely."

- name: "xtcThreshold"
  type: "number | false"
  optional: true
  description: "Defines the lower probability threshold for the XTC (Exclude Top Choices) sampling technique.\n\nWhen XTC sampling is activated (based on xtcProbability), the algorithm identifies tokens with probabilities between this threshold and 0.5, then removes all such tokens except the least probable one. This helps introduce more diverse and unexpected tokens into the generation.\n\nOnly takes effect when xtcProbability is enabled."

- name: "cpuThreads"
  type: "number"
  optional: true
  description: "Specifies the number of CPU threads to allocate for model inference.\n\nHigher values can improve performance on multi-core systems but may compete with other processes. For example, on an 8-core system, a value of 4-6 might provide good performance while leaving resources for other tasks.\n\nIf not specified, the system will use a default value based on available hardware."

- name: "draftModel"
  type: "string"
  optional: true
  description: "The draft model to use for speculative decoding. Speculative decoding is a technique that can drastically increase the generation speed (up to 3x for larger models) by paring a main model with a smaller draft model.\n\nSee here for more information: https://lmstudio.ai/docs/advanced/speculative-decoding\n\nYou do not need to load the draft model yourself. Simply specifying its model key here is enough."
```


## model-info

### Get Context Length

> API to get the maximum context length of a model.


LLMs and embedding models, due to their fundamental architecture, have a property called `context length`, and more specifically a **maximum** context length. Loosely speaking, this is how many tokens the models can "keep in memory" when generating text or embeddings. Exceeding this limit will result in the model behaving erratically.

## Use the `getContextLength()` Function on the Model Object

It's useful to be able to check the context length of a model, especially as an extra check before providing potentially long input to the model.

```lms_code_snippet
  title: "index.ts"
  variants:
    TypeScript:
      language: typescript
      code: |
        const contextLength = await model.getContextLength();
```

The `model` in the above code snippet is an instance of a loaded model you get from the `llm.model` method. See [Manage Models in Memory](../manage-models/loading) for more information.

### Example: Check if the input will fit in the model's context window

You can determine if a given conversation fits into a model's context by doing the following:

1. Convert the conversation to a string using the prompt template.
2. Count the number of tokens in the string.
3. Compare the token count to the model's context length.

```lms_code_snippet
  variants:
    TypeScript:
      language: typescript
      code: |
        import { Chat, type LLM, LMStudioClient } from "@lmstudio/sdk";

        async function doesChatFitInContext(model: LLM, chat: Chat) {
          // Convert the conversation to a string using the prompt template.
          const formatted = await model.applyPromptTemplate(chat);
          // Count the number of tokens in the string.
          const tokenCount = await model.countTokens(formatted);
          // Get the current loaded context length of the model
          const contextLength = await model.getContextLength();
          return tokenCount < contextLength;
        }

        const client = new LMStudioClient();
        const model = await client.llm.model();

        const chat = Chat.from([
          { role: "user", content: "What is the meaning of life." },
          { role: "assistant", content: "The meaning of life is..." },
          // ... More messages
        ]);

        console.info("Fits in context:", await doesChatFitInContext(model, chat));
```


### Get Model Info

> Get information about the model


You can access information about a loaded model using the `getInfo` method.

```lms_code_snippet
  variants:
    LLM:
      language: typescript
      code: |
        import { LMStudioClient } from "@lmstudio/sdk";

        const client = new LMStudioClient();
        const model = await client.llm.model();

        const modelInfo = await model.getInfo();

        console.info("Model Key", modelInfo.modelKey);
        console.info("Current Context Length", model.contextLength);
        console.info("Model Trained for Tool Use", modelInfo.trainedForToolUse);
        // etc.
    Embedding Model:
      language: typescript
      code: |
        import { LMStudioClient } from "@lmstudio/sdk";

        const client = new LMStudioClient();
        const model = await client.embedding.model();

        const modelInfo = await model.getInfo();

        console.info("Model Key", modelInfo.modelKey);
        console.info("Current Context Length", modelInfo.contextLength);
        // etc.
```