# Source: https://huggingface.co/docs/transformers/v5.0.0rc1/model_doc/xlstm.md

# xLSTM

## Overview

The xLSTM model was proposed in [xLSTM: Extended Long Short-Term Memory](https://huggingface.co/papers/2405.04517) by Maximilian Beck*, Korbinian Pöppel*, Markus Spanring, Andreas Auer, Oleksandra Prudnikova, Michael Kopp, Günter Klambauer, Johannes Brandstetter and Sepp Hochreiter.
xLSTM updates the original LSTM architecture to be competitive with Transformer models by introducing exponential gating, matrix memory expansion, and parallelizable training and ingestion.

The [7B model](https://huggingface.co/NX-AI/xLSTM-7b) variant was trained by the xLSTM team Maximilian Beck, Korbinian Pöppel, Phillip Lippe, Richard Kurle, Patrick Blies, Sebastian Böck and Sepp Hochreiter at NXAI.

The abstract from the paper is the following:

*In the 1990s, the constant error carousel and gating were introduced as the central ideas of the Long Short-Term Memory (LSTM). Since then, LSTMs have stood the test of time and contributed to numerous deep learning success stories, in particular they constituted the first Large Language Models (LLMs). However, the advent of the Transformer technology with parallelizable self-attention at its core marked the dawn of a new era, outpacing LSTMs at scale. We now raise a simple question: How far do we get in language modeling when scaling LSTMs to billions of parameters, leveraging the latest techniques from modern LLMs, but mitigating known limitations of LSTMs? Firstly, we introduce exponential gating with appropriate normalization and stabilization techniques. Secondly, we modify the LSTM memory structure, obtaining: (i) sLSTM with a scalar memory, a scalar update, and new memory mixing, (ii) mLSTM that is fully parallelizable with a matrix memory and a covariance update rule. Integrating these LSTM extensions into residual block backbones yields xLSTM blocks that are then residually stacked into xLSTM architectures. Exponential gating and modified memory structures boost xLSTM capabilities to perform favorably when compared to state-of-the-art Transformers and State Space Models, both in performance and scaling.*

This model was contributed by [NX-AI](https://huggingface.co/NX-AI).
The original code can be found [here](https://github.com/NX-AI/xlstm).

## xLSTMConfig[[transformers.xLSTMConfig]]

#### transformers.xLSTMConfig[[transformers.xLSTMConfig]]

[Source](https://github.com/huggingface/transformers/blob/v5.0.0rc1/src/transformers/models/xlstm/configuration_xlstm.py#L60)

This is the configuration class to store the configuration of a `xLSTM`. It is used to instantiate a xLSTM
model according to the specified arguments, defining the model architecture. Instantiating a configuration with the
defaults will yield a similar configuration to that of the xLSTM-7b [NX-AI/xLSTM-7b](https://huggingface.co/NX-AI/xLSTM-7b) model.

Configuration objects inherit from [PreTrainedConfig](/docs/transformers/v5.0.0rc1/en/main_classes/configuration#transformers.PreTrainedConfig) and can be used to control the model outputs. Read the
documentation from [PreTrainedConfig](/docs/transformers/v5.0.0rc1/en/main_classes/configuration#transformers.PreTrainedConfig) for more information.

Example:

```python
>>> from transformers import xLSTMConfig, xLSTMModel

>>> # Initializing a xLSTM configuration
>>> configuration = xLSTMConfig()

>>> # Initializing a model (with random weights) from the configuration
>>> model = xLSTMModel(configuration)

>>> # Accessing the model configuration
>>> configuration = model.config
```

**Parameters:**

vocab_size (int, optional, *optional*, defaults to 50304) : Vocabulary size of the xLSTM model. Defines the number of different tokens that can be represented by the `inputs_ids` passed when calling [xLSTMModel](/docs/transformers/v5.0.0rc1/en/model_doc/xlstm#transformers.xLSTMModel). Defaults to the GPT2-NeoX tokenizer size.

hidden_size (int, optional, *optional*, defaults to 4096) : Dimensionality of the embeddings or hidden states.

embedding_dim (int, optional, *optional*, defaults to 4096) : Dimensionality of the embeddings or hidden states, use hidde_size if None.

num_hidden_layers (int, optional, *optional*, defaults to 32) : Number of blocks of the xLSTM model.

num_blocks (int, optional, *optional*, defaults to 32) : Number of blocks of the xLSTM model, use num_hidden_layers if None.

num_heads (int, optional, *optional*, defaults to 8) : Number of heads for the xLSTM Layer/Cell.

use_bias (bool, optional, *optional*, defaults to `False`) : Whether to use biases in the xLSTM model.

norm_reduction_force_float32 (bool, optional, *optional*, defaults to `True`) : Whether to force the float32 norm reduction op to be done in fp32 precision.

tie_word_embeddings (bool, optional, *optional*, defaults to `False`) : Whether to tie word embeddings to the lm head weights.

add_out_norm (bool, optional, *optional*, defaults to `True`) : Whether to add an output norm after the blocks before the LMHead.

norm_eps (float, optional, *optional*, defaults to 1e-06) : Norm eps for RMSNorm and Layer Norm.

qk_dim_factor (float, optional, *optional*, defaults to 0.5) : Scale factor for the query and key dimension.

v_dim_factor (float, optional, *optional*, defaults to 1.0) : Scale factor for the value dimension.

chunkwise_kernel (ChunkwiseKernelType, optional, *optional*, defaults to `"chunkwise--native_autograd"`) : Kernel type for chunkwise processing mode.

sequence_kernel (SequenceKernelType, optional, *optional*, defaults to `"native_sequence__native"`) : Kernel type for sequence processing mode.

step_kernel (StepKernelType, optional, *optional*, defaults to `"native"`) : Kernel type for step processing mode.

mode (BackendModeType, optional, *optional*, defaults to `"inference"`) : Operation mode (inference is needed for generation).

chunk_size (int, optional, *optional*, defaults to 64) : Internal chunk size.

return_last_states (bool, optional, *optional*, defaults to `True`) : If to return the last states / cache internally. Needed as True for generation.

autocast_kernel_dtype (DtypeType, optional, *optional*, defaults to `"bfloat16"`) : Kernel dtype for the states.

eps (float, optional, *optional*, defaults to 1e-06) : Epsilon for the mLSTM cell post norm.

inference_state_dtype (DtypeType, optional, *optional*, defaults to `"float32"`) : Kernel dtype for states in inference.

ffn_proj_factor (float, optional, *optional*, defaults to 2.667) : Size factor of the post-up projection gated Feed Forward network.

ffn_round_up_to_multiple_of (int, optional, *optional*, defaults to 64) : Size factor round value of the post-up projection gated Feed Forward network.

gate_soft_cap (float, optional, *optional*, defaults to 15.0) : Gate soft cap scale.

output_logit_soft_cap (float, optional, *optional*, defaults to 30.0) : Output logit soft cap scale.

weight_mode (`Literal`, *optional*, defaults to `"single"`) : Whether parallel linear layers are separated or fused (single).

use_cache (bool, optional, *optional*, defaults to `True`) : Whether to use the cache (xLSTMCache).

pad_token_id (int, optional, *optional*, defaults to 1) : Pad token id needed for generation.

bos_token_id (int, optional, *optional*, defaults to 0) : BOS token id needed for generation.

eos_token_id (int, optional, *optional*, defaults to 2) : EOS token id needed for generation.

max_inference_chunksize (int, optional, *optional*, defaults to 16384) : Limit the chunk size for inference to save memory.

## xLSTMModel[[transformers.xLSTMModel]]

#### transformers.xLSTMModel[[transformers.xLSTMModel]]

[Source](https://github.com/huggingface/transformers/blob/v5.0.0rc1/src/transformers/models/xlstm/modeling_xlstm.py#L1392)

The bare Xlstm Model outputting raw hidden-states without any specific head on top.

This model inherits from [PreTrainedModel](/docs/transformers/v5.0.0rc1/en/main_classes/model#transformers.PreTrainedModel). Check the superclass documentation for the generic methods the
library implements for all its model (such as downloading or saving, resizing the input embeddings, pruning heads
etc.)

This model is also a PyTorch [torch.nn.Module](https://pytorch.org/docs/stable/nn.html#torch.nn.Module) subclass.
Use it as a regular PyTorch Module and refer to the PyTorch documentation for all matter related to general usage
and behavior.

forwardtransformers.xLSTMModel.forwardhttps://github.com/huggingface/transformers/blob/v5.0.0rc1/src/transformers/models/xlstm/modeling_xlstm.py#L1409[{"name": "input_ids", "val": ": typing.Optional[torch.LongTensor] = None"}, {"name": "inputs_embeds", "val": ": typing.Optional[torch.LongTensor] = None"}, {"name": "cache_params", "val": ": typing.Optional[transformers.models.xlstm.modeling_xlstm.xLSTMCache] = None"}, {"name": "use_cache", "val": ": typing.Optional[bool] = None"}, {"name": "output_hidden_states", "val": ": typing.Optional[bool] = None"}, {"name": "**kwargs", "val": ""}]- **input_ids** (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*) --
  Indices of input sequence tokens in the vocabulary. Padding will be ignored by default.

  Indices can be obtained using [AutoTokenizer](/docs/transformers/v5.0.0rc1/en/model_doc/auto#transformers.AutoTokenizer). See [PreTrainedTokenizer.encode()](/docs/transformers/v5.0.0rc1/en/internal/tokenization_utils#transformers.PreTrainedTokenizerBase.encode) and
  [PreTrainedTokenizer.__call__()](/docs/transformers/v5.0.0rc1/en/internal/tokenization_utils#transformers.PreTrainedTokenizerBase.__call__) for details.

  [What are input IDs?](../glossary#input-ids)
- **inputs_embeds** (`torch.LongTensor` of shape `(batch_size, sequence_length, hidden_size)`, *optional*) --
  Optionally, instead of passing `input_ids` you can choose to directly pass an embedded representation. This
  is useful if you want more control over how to convert `input_ids` indices into associated vectors than the
  model's internal embedding lookup matrix.
- **cache_params** (`xLSTMCache`, *optional*) --
  The xLSTMCache that carries the RNN states.
- **use_cache** (`bool`, *optional*) --
  If set to `True`, `past_key_values` key value states are returned and can be used to speed up decoding (see
  `past_key_values`).
- **output_hidden_states** (`bool`, *optional*) --
  Whether or not to return the hidden states of all layers. See `hidden_states` under returned tensors for
  more detail.0`transformers.models.xlstm.modeling_xlstm.xLSTMOutput` or `tuple(torch.FloatTensor)`A `transformers.models.xlstm.modeling_xlstm.xLSTMOutput` or a tuple of
`torch.FloatTensor` (if `return_dict=False` is passed or when `config.return_dict=False`) comprising various
elements depending on the configuration ([xLSTMConfig](/docs/transformers/v5.0.0rc1/en/model_doc/xlstm#transformers.xLSTMConfig)) and inputs.

- **last_hidden_state** (`torch.FloatTensor` of shape `(batch_size, sequence_length, hidden_size)`, *optional*) -- Sequence of hidden-states at the output of the last layer of the model.
- **cache_params** (`~models.xlstm.modeling_xlstm.xLSTMCache`, *optional*, defaults to `None`) -- The state of the model at the last time step. Can be used in a forward method with the next `input_ids` to
  avoid providing the old `input_ids`.
- **hidden_states** (`tuple[torch.FloatTensor]`, *optional*, returned when `output_hidden_states=True` is passed or when `config.output_hidden_states=True`) -- Tuple of `torch.FloatTensor` (one for the output of the embeddings, if the model has an embedding layer, +
  one for the output of each layer) of shape `(batch_size, sequence_length, hidden_size)`.

  Hidden-states of the model at the output of each layer plus the optional initial embedding outputs.
The [xLSTMModel](/docs/transformers/v5.0.0rc1/en/model_doc/xlstm#transformers.xLSTMModel) forward method, overrides the `__call__` special method.

Although the recipe for forward pass needs to be defined within this function, one should call the `Module`
instance afterwards instead of this since the former takes care of running the pre and post processing steps while
the latter silently ignores them.

**Parameters:**

config ([xLSTMModel](/docs/transformers/v5.0.0rc1/en/model_doc/xlstm#transformers.xLSTMModel)) : Model configuration class with all the parameters of the model. Initializing with a config file does not load the weights associated with the model, only the configuration. Check out the [from_pretrained()](/docs/transformers/v5.0.0rc1/en/main_classes/model#transformers.PreTrainedModel.from_pretrained) method to load the model weights.

**Returns:**

``transformers.models.xlstm.modeling_xlstm.xLSTMOutput` or `tuple(torch.FloatTensor)``

A `transformers.models.xlstm.modeling_xlstm.xLSTMOutput` or a tuple of
`torch.FloatTensor` (if `return_dict=False` is passed or when `config.return_dict=False`) comprising various
elements depending on the configuration ([xLSTMConfig](/docs/transformers/v5.0.0rc1/en/model_doc/xlstm#transformers.xLSTMConfig)) and inputs.

- **last_hidden_state** (`torch.FloatTensor` of shape `(batch_size, sequence_length, hidden_size)`, *optional*) -- Sequence of hidden-states at the output of the last layer of the model.
- **cache_params** (`~models.xlstm.modeling_xlstm.xLSTMCache`, *optional*, defaults to `None`) -- The state of the model at the last time step. Can be used in a forward method with the next `input_ids` to
  avoid providing the old `input_ids`.
- **hidden_states** (`tuple[torch.FloatTensor]`, *optional*, returned when `output_hidden_states=True` is passed or when `config.output_hidden_states=True`) -- Tuple of `torch.FloatTensor` (one for the output of the embeddings, if the model has an embedding layer, +
  one for the output of each layer) of shape `(batch_size, sequence_length, hidden_size)`.

  Hidden-states of the model at the output of each layer plus the optional initial embedding outputs.

## xLSTMLMHeadModel[[transformers.xLSTMForCausalLM]]

#### transformers.xLSTMForCausalLM[[transformers.xLSTMForCausalLM]]

[Source](https://github.com/huggingface/transformers/blob/v5.0.0rc1/src/transformers/models/xlstm/modeling_xlstm.py#L1524)

The Xlstm Model for causal language modeling.

This model inherits from [PreTrainedModel](/docs/transformers/v5.0.0rc1/en/main_classes/model#transformers.PreTrainedModel). Check the superclass documentation for the generic methods the
library implements for all its model (such as downloading or saving, resizing the input embeddings, pruning heads
etc.)

This model is also a PyTorch [torch.nn.Module](https://pytorch.org/docs/stable/nn.html#torch.nn.Module) subclass.
Use it as a regular PyTorch Module and refer to the PyTorch documentation for all matter related to general usage
and behavior.

forwardtransformers.xLSTMForCausalLM.forwardhttps://github.com/huggingface/transformers/blob/v5.0.0rc1/src/transformers/models/xlstm/modeling_xlstm.py#L1576[{"name": "input_ids", "val": ": typing.Optional[torch.LongTensor] = None"}, {"name": "inputs_embeds", "val": ": typing.Optional[torch.FloatTensor] = None"}, {"name": "cache_params", "val": ": typing.Optional[transformers.models.xlstm.modeling_xlstm.xLSTMCache] = None"}, {"name": "labels", "val": ": typing.Optional[torch.LongTensor] = None"}, {"name": "use_cache", "val": ": typing.Optional[bool] = None"}, {"name": "output_hidden_states", "val": ": typing.Optional[bool] = None"}, {"name": "**kwargs", "val": ""}]- **input_ids** (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*) --
  Indices of input sequence tokens in the vocabulary. Padding will be ignored by default.

  Indices can be obtained using [AutoTokenizer](/docs/transformers/v5.0.0rc1/en/model_doc/auto#transformers.AutoTokenizer). See [PreTrainedTokenizer.encode()](/docs/transformers/v5.0.0rc1/en/internal/tokenization_utils#transformers.PreTrainedTokenizerBase.encode) and
  [PreTrainedTokenizer.__call__()](/docs/transformers/v5.0.0rc1/en/internal/tokenization_utils#transformers.PreTrainedTokenizerBase.__call__) for details.

  [What are input IDs?](../glossary#input-ids)
- **inputs_embeds** (`torch.FloatTensor` of shape `(batch_size, sequence_length, hidden_size)`, *optional*) --
  Optionally, instead of passing `input_ids` you can choose to directly pass an embedded representation. This
  is useful if you want more control over how to convert `input_ids` indices into associated vectors than the
  model's internal embedding lookup matrix.
- **cache_params** (`xLSTMCache`, *optional*) --
  The xLSTMCache that carries the RNN states.
- **labels** (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*) --
  Labels for computing the masked language modeling loss. Indices should either be in `[0, ...,
  config.vocab_size]` or -100 (see `input_ids` docstring). Tokens with indices set to `-100` are ignored
  (masked), the loss is only computed for the tokens with labels in `[0, ..., config.vocab_size]`.
- **use_cache** (`bool`, *optional*) --
  If set to `True`, `past_key_values` key value states are returned and can be used to speed up decoding (see
  `past_key_values`).
- **output_hidden_states** (`bool`, *optional*) --
  Whether or not to return the hidden states of all layers. See `hidden_states` under returned tensors for
  more detail.0`transformers.models.xlstm.modeling_xlstm.xLSTMCausalLMOutput` or `tuple(torch.FloatTensor)`A `transformers.models.xlstm.modeling_xlstm.xLSTMCausalLMOutput` or a tuple of
`torch.FloatTensor` (if `return_dict=False` is passed or when `config.return_dict=False`) comprising various
elements depending on the configuration ([xLSTMConfig](/docs/transformers/v5.0.0rc1/en/model_doc/xlstm#transformers.xLSTMConfig)) and inputs.

- **loss** (`torch.FloatTensor` of shape `(1,)`, *optional*, returned when `labels` is provided) -- Language modeling loss (for next-token prediction).
- **logits** (`torch.FloatTensor` of shape `(batch_size, sequence_length, config.vocab_size)`) -- Prediction scores of the language modeling head (scores for each vocabulary token before SoftMax).
- **cache_params** (`xLSTMCache`, *optional*, carrying the RNN states) -- The state of the model at the last time step. Can be used in a forward method with the next `input_ids` to
  avoid providing the old `input_ids`.
- **hidden_states** (`tuple[torch.FloatTensor]`, *optional*, returned when `output_hidden_states=True` is passed or when `config.output_hidden_states=True`) -- Tuple of `torch.FloatTensor` (one for the output of the embeddings, if the model has an embedding layer, +
  one for the output of each layer) of shape `(batch_size, sequence_length, hidden_size)`.

  Hidden-states of the model at the output of each layer plus the optional initial embedding outputs.
The [xLSTMForCausalLM](/docs/transformers/v5.0.0rc1/en/model_doc/xlstm#transformers.xLSTMForCausalLM) forward method, overrides the `__call__` special method.

Although the recipe for forward pass needs to be defined within this function, one should call the `Module`
instance afterwards instead of this since the former takes care of running the pre and post processing steps while
the latter silently ignores them.

Example:

```python
```

**Parameters:**

config ([xLSTMForCausalLM](/docs/transformers/v5.0.0rc1/en/model_doc/xlstm#transformers.xLSTMForCausalLM)) : Model configuration class with all the parameters of the model. Initializing with a config file does not load the weights associated with the model, only the configuration. Check out the [from_pretrained()](/docs/transformers/v5.0.0rc1/en/main_classes/model#transformers.PreTrainedModel.from_pretrained) method to load the model weights.

**Returns:**

``transformers.models.xlstm.modeling_xlstm.xLSTMCausalLMOutput` or `tuple(torch.FloatTensor)``

A `transformers.models.xlstm.modeling_xlstm.xLSTMCausalLMOutput` or a tuple of
`torch.FloatTensor` (if `return_dict=False` is passed or when `config.return_dict=False`) comprising various
elements depending on the configuration ([xLSTMConfig](/docs/transformers/v5.0.0rc1/en/model_doc/xlstm#transformers.xLSTMConfig)) and inputs.

- **loss** (`torch.FloatTensor` of shape `(1,)`, *optional*, returned when `labels` is provided) -- Language modeling loss (for next-token prediction).
- **logits** (`torch.FloatTensor` of shape `(batch_size, sequence_length, config.vocab_size)`) -- Prediction scores of the language modeling head (scores for each vocabulary token before SoftMax).
- **cache_params** (`xLSTMCache`, *optional*, carrying the RNN states) -- The state of the model at the last time step. Can be used in a forward method with the next `input_ids` to
  avoid providing the old `input_ids`.
- **hidden_states** (`tuple[torch.FloatTensor]`, *optional*, returned when `output_hidden_states=True` is passed or when `config.output_hidden_states=True`) -- Tuple of `torch.FloatTensor` (one for the output of the embeddings, if the model has an embedding layer, +
  one for the output of each layer) of shape `(batch_size, sequence_length, hidden_size)`.

  Hidden-states of the model at the output of each layer plus the optional initial embedding outputs.