# Source: https://www.gigaspaces.com/data-terms/direct-preference-optimization-dpo

Skip to content

[ ![GigaSpaces Logo](https://www.gigaspaces.com/wp-content/uploads/2020/02/logo-header-2.svg) ![GigaSpaces Logo](https://www.gigaspaces.com/wp-content/uploads/2020/02/logo-header-2.svg) ](https://www.gigaspaces.com/)

Search for:

  * Products![](https://www.gigaspaces.com/wp-content/uploads/2022/11/Path-16484.svg)

    * Our Products

      * [eRAG](https://www.gigaspaces.com/products/erag)
        * [GenAI Catalyst](https://www.gigaspaces.com/products/genai-catalyst)
        * [Instant Data](https://www.gigaspaces.com/products/instant-data)
        * [Respond Proactively](https://www.gigaspaces.com/products/proactive-ai)
        * [Act Autonomously](https://www.gigaspaces.com/products/data-autonomy)
      * [Smart DIH](https://www.gigaspaces.com/products/smart-digital-integration-hub-lobby)
      * [XAP](https://www.gigaspaces.com/products/xap-imdg)
    * Solutions for

      * [Pharma](https://www.gigaspaces.com/products/erag/pharma)
      * [Procurement](https://www.gigaspaces.com/products/erag/procurement)
    * [![vid-icon](https://www.gigaspaces.com/wp-content/uploads/2025/02/Product-menu-thumb.jpg)](https://go.gigaspaces.com/webinar/why-conventional-rag-falls-short)

Conventional RAG Falls Short with Enterprise Databases

[Watch the Webinar![icon](/wp-content/uploads/2023/04/Group-30743.png)](https://go.gigaspaces.com/webinar/why-conventional-rag-falls-short)

  * Solutions![](https://www.gigaspaces.com/wp-content/uploads/2022/11/Path-16484.svg)

    * Business Solutions

      * [Digital Innovation Over Legacy Systems](https://www.gigaspaces.com/use-cases/digital-innovation-acceleration)
      * [Integration Data Hub](https://www.gigaspaces.com/technology/integration-data-hub)
      * [API Scaling](https://www.gigaspaces.com/use-cases/application-acceleration)
      * [Hybrid / Multi-cloud Integration](https://www.gigaspaces.com/use-cases/hybrid-or-multi-cloud-integration)
      * [Customer 360](https://www.gigaspaces.com/use-cases/customer-360)
      * Industry Solutions
      * [Retail](https://www.gigaspaces.com/retail-and-ecommerce-2)
      * [Financial Services](https://www.gigaspaces.com/financial-services)
      * [Insurance Companies](https://www.gigaspaces.com/industry-solutions/insurance)
    * [![vid-icon](https://www.gigaspaces.com/wp-content/uploads/2024/07/solutons.png) Massimo Pezzini, Gartner Analyst Emeritus  ](https://go.gigaspaces.com/5-top-use-cases-for-driving-business-with-data-hub-architecture)

5 Top Use Cases For Driving Business With Data Hub Architecture 

[Watch the Webinar![icon](/wp-content/uploads/2023/04/Group-30743.png)](https://go.gigaspaces.com/5-top-use-cases-for-driving-business-with-data-hub-architecture)

  * How it Works![](https://www.gigaspaces.com/wp-content/uploads/2022/11/Path-16484.svg)

    * [eRAG Technology Overview](/how-it-works/erag-how-it-works)

      * [AI-Ready, IT-Friendly](https://www.gigaspaces.com/products/ai-ready)
      * [Semantic Reasoning](https://www.gigaspaces.com/how-it-works/semantic-reasoning)
      * [Questions to SQL Queries](https://www.gigaspaces.com/how-it-works/questions-into-sql-queries)
      * [Asked & Answered in Natural Language](https://www.gigaspaces.com/how-it-works/asked-answered-natural-language)
      * [Multiple Data Sources](https://www.gigaspaces.com/how-it-works/supported-data-sources)
      * [Proactive AI Governance](https://www.gigaspaces.com/how-it-works/proactive-ai-governance)
    * [![vid-icon](https://www.gigaspaces.com/wp-content/uploads/2025/02/How-it-works-menu.jpg)](https://go.gigaspaces.com/effective-ai-governance-wp)

Ensure GenAI compliance and governance 

[Read the Whitepaper![icon](/wp-content/uploads/2023/04/Group-30743.png)](https://go.gigaspaces.com/effective-ai-governance-wp)

  * [Success Stories![](https://www.gigaspaces.com/wp-content/uploads/2022/11/Path-16484.svg)](https://www.gigaspaces.com/case-studies)

    * By Use Case

      * [Procurement](https://www.gigaspaces.com/use-cases/procurement-supply-chain)
      * [Operations](https://www.gigaspaces.com/use-cases/enhancing-pharmaceutical-operations)
      * [Budget Management](https://www.gigaspaces.com/use-cases/educational-organizations-budget-resources)
      * [Sales Operations](https://www.gigaspaces.com/use-cases/retail-boost-sales)
      * [Service Providers](https://www.gigaspaces.com/use-cases/contract-control-cost-savings)
      * [Utilities Management](https://www.gigaspaces.com/use-cases/cut-sla-penalties)
      * [Restaurant Management](https://www.gigaspaces.com/use-cases/power-smarter-restaurant-growth-erag)
    * By Industry

      * [Logistics](https://www.gigaspaces.com/case-studies/optimizing-supply-chain-with-ai-powered-insights)
      * [Pharma](https://www.gigaspaces.com/case-studies/transforming-patient-outcomes)
      * [Education](https://www.gigaspaces.com/case-studies/achieving-educational-budget-control)
      * [Retail](https://www.gigaspaces.com/case-studies/retail-intelligence-with-erag)
      * [Shipping](https://www.gigaspaces.com/case-studies/redefining-global-logistics)
      * [Energy](https://www.gigaspaces.com/case-studies/boosting-operations-reducing-penalties)
      * [Hospitality](https://www.gigaspaces.com/case-studies/restaurant-increase-profit-erag)
    * [![vid-icon](https://www.gigaspaces.com/wp-content/uploads/2025/02/Company-menu-thumb.jpg)](https://go.gigaspaces.com/webinarod/ecurely-unlocking-structured-data-for-enterprise-ai-0)

Monkey See, AI Do - All about CUA

[Watch Webinar![icon](/wp-content/uploads/2023/04/Group-30743.png)](https://go.gigaspaces.com/webinarod/ecurely-unlocking-structured-data-for-enterprise-ai-0)

  * Resources![](https://www.gigaspaces.com/wp-content/uploads/2022/11/Path-16484.svg)

    * Content Hub

      * [Case Studies](/resources/case-studies)
      * [Webinars](https://www.gigaspaces.com/resources/webinars)
      * [Q&As](https://www.gigaspaces.com/answers)
      * [Videos](https://www.gigaspaces.com/resources/videos)
      * [Whitepapers & Brochures](https://www.gigaspaces.com/resources/brochures)
      * [Events](/resources/events/)
      * [Glossary](https://www.gigaspaces.com/data-terms)
      * [Blog](https://www.gigaspaces.com/blog)
      * [FAQs](https://www.gigaspaces.com/products/erag/faq)
      * [Technical Documentation](https://docs.gigaspaces.com/latest/landing.html)
    * [![vid-icon](https://www.gigaspaces.com/wp-content/uploads/2025/02/Resources-menu-thumb.jpg)](https://www.gigaspaces.com/blog/from-rag-to-tag)

Taking the AI leap from RAG to TAG

[Read the Blog![icon](/wp-content/uploads/2023/04/Group-30743.png)](https://www.gigaspaces.com/blog/from-rag-to-tag)

  * Company![](https://www.gigaspaces.com/wp-content/uploads/2022/11/Path-16484.svg)

    * Our Company

      * [About](https://www.gigaspaces.com/about)
      * [Customers](https://www.gigaspaces.com/customers)
      * [Management](https://www.gigaspaces.com/leadership)
      * [Board Members](https://www.gigaspaces.com/board-members)
      * [Investors](https://www.gigaspaces.com/investors)
      * [News](https://www.gigaspaces.com/news)
      * [Press Releases](/press-releases)
      * [Careers](https://www.gigaspaces.com/career-lobby)
    * col2

      * [Partners](https://www.gigaspaces.com/partners-lobby)
      * [OEM Partners](https://www.gigaspaces.com/partners-lobby/oem)
      * [System Integrators](https://www.gigaspaces.com/partners-lobby/system-integrators)
      * [Technology Partners](https://www.gigaspaces.com/partners-lobby/technology)
      * [Value Added Resellers](https://www.gigaspaces.com/partners-lobby/value-added-resellers)
      * Support & Services
      * [Services](https://www.gigaspaces.com/service-overview)
      * [Support](https://www.gigaspaces.com/support-lobby)
    * [![vid-icon](https://www.gigaspaces.com/wp-content/uploads/2025/02/Company-menu-thumb.jpg)](https://www.gigaspaces.com/news/gigaspaces-partners-with-ibm-and-aws-to-deliver-accurate-safe-genai-with-dbs)

GigaSpaces, IBM & AWS make AI safer

[Read How![icon](/wp-content/uploads/2023/04/Group-30743.png)](https://www.gigaspaces.com/news/gigaspaces-partners-with-ibm-and-aws-to-deliver-accurate-safe-genai-with-dbs)

  * [Book a Demo](https://www.gigaspaces.com/book-a-demo)



Search for:

  * Products![](https://www.gigaspaces.com/wp-content/uploads/2022/11/Path-16484.svg)

    * Our Products

      * [eRAG](https://www.gigaspaces.com/products/erag)
        * [GenAI Catalyst](https://www.gigaspaces.com/products/genai-catalyst)
        * [Instant Data](https://www.gigaspaces.com/products/instant-data)
        * [Respond Proactively](https://www.gigaspaces.com/products/proactive-ai)
        * [Act Autonomously](https://www.gigaspaces.com/products/data-autonomy)
      * [Smart DIH](https://www.gigaspaces.com/products/smart-digital-integration-hub-lobby)
      * [XAP](https://www.gigaspaces.com/products/xap-imdg)
    * Solutions for

      * [Pharma](https://www.gigaspaces.com/products/erag/pharma)
      * [Procurement](https://www.gigaspaces.com/products/erag/procurement)
    * [![vid-icon](https://www.gigaspaces.com/wp-content/uploads/2025/02/Product-menu-thumb.jpg)](https://go.gigaspaces.com/webinar/why-conventional-rag-falls-short)

Conventional RAG Falls Short with Enterprise Databases

[Watch the Webinar![icon](/wp-content/uploads/2023/04/Group-30743.png)](https://go.gigaspaces.com/webinar/why-conventional-rag-falls-short)

  * Solutions![](https://www.gigaspaces.com/wp-content/uploads/2022/11/Path-16484.svg)

    * Business Solutions

      * [Digital Innovation Over Legacy Systems](https://www.gigaspaces.com/use-cases/digital-innovation-acceleration)
      * [Integration Data Hub](https://www.gigaspaces.com/technology/integration-data-hub)
      * [API Scaling](https://www.gigaspaces.com/use-cases/application-acceleration)
      * [Hybrid / Multi-cloud Integration](https://www.gigaspaces.com/use-cases/hybrid-or-multi-cloud-integration)
      * [Customer 360](https://www.gigaspaces.com/use-cases/customer-360)
      * Industry Solutions
      * [Retail](https://www.gigaspaces.com/retail-and-ecommerce-2)
      * [Financial Services](https://www.gigaspaces.com/financial-services)
      * [Insurance Companies](https://www.gigaspaces.com/industry-solutions/insurance)
    * [![vid-icon](https://www.gigaspaces.com/wp-content/uploads/2024/07/solutons.png) Massimo Pezzini, Gartner Analyst Emeritus  ](https://go.gigaspaces.com/5-top-use-cases-for-driving-business-with-data-hub-architecture)

5 Top Use Cases For Driving Business With Data Hub Architecture 

[Watch the Webinar![icon](/wp-content/uploads/2023/04/Group-30743.png)](https://go.gigaspaces.com/5-top-use-cases-for-driving-business-with-data-hub-architecture)

  * How it Works![](https://www.gigaspaces.com/wp-content/uploads/2022/11/Path-16484.svg)

    * [eRAG Technology Overview](/how-it-works/erag-how-it-works)

      * [AI-Ready, IT-Friendly](https://www.gigaspaces.com/products/ai-ready)
      * [Semantic Reasoning](https://www.gigaspaces.com/how-it-works/semantic-reasoning)
      * [Questions to SQL Queries](https://www.gigaspaces.com/how-it-works/questions-into-sql-queries)
      * [Asked & Answered in Natural Language](https://www.gigaspaces.com/how-it-works/asked-answered-natural-language)
      * [Multiple Data Sources](https://www.gigaspaces.com/how-it-works/supported-data-sources)
      * [Proactive AI Governance](https://www.gigaspaces.com/how-it-works/proactive-ai-governance)
    * [![vid-icon](https://www.gigaspaces.com/wp-content/uploads/2025/02/How-it-works-menu.jpg)](https://go.gigaspaces.com/effective-ai-governance-wp)

Ensure GenAI compliance and governance 

[Read the Whitepaper![icon](/wp-content/uploads/2023/04/Group-30743.png)](https://go.gigaspaces.com/effective-ai-governance-wp)

  * [Success Stories![](https://www.gigaspaces.com/wp-content/uploads/2022/11/Path-16484.svg)](https://www.gigaspaces.com/case-studies)

    * By Use Case

      * [Procurement](https://www.gigaspaces.com/use-cases/procurement-supply-chain)
      * [Operations](https://www.gigaspaces.com/use-cases/enhancing-pharmaceutical-operations)
      * [Budget Management](https://www.gigaspaces.com/use-cases/educational-organizations-budget-resources)
      * [Sales Operations](https://www.gigaspaces.com/use-cases/retail-boost-sales)
      * [Service Providers](https://www.gigaspaces.com/use-cases/contract-control-cost-savings)
      * [Utilities Management](https://www.gigaspaces.com/use-cases/cut-sla-penalties)
      * [Restaurant Management](https://www.gigaspaces.com/use-cases/power-smarter-restaurant-growth-erag)
    * By Industry

      * [Logistics](https://www.gigaspaces.com/case-studies/optimizing-supply-chain-with-ai-powered-insights)
      * [Pharma](https://www.gigaspaces.com/case-studies/transforming-patient-outcomes)
      * [Education](https://www.gigaspaces.com/case-studies/achieving-educational-budget-control)
      * [Retail](https://www.gigaspaces.com/case-studies/retail-intelligence-with-erag)
      * [Shipping](https://www.gigaspaces.com/case-studies/redefining-global-logistics)
      * [Energy](https://www.gigaspaces.com/case-studies/boosting-operations-reducing-penalties)
      * [Hospitality](https://www.gigaspaces.com/case-studies/restaurant-increase-profit-erag)
    * [![vid-icon](https://www.gigaspaces.com/wp-content/uploads/2025/02/Company-menu-thumb.jpg)](https://go.gigaspaces.com/webinarod/ecurely-unlocking-structured-data-for-enterprise-ai-0)

Monkey See, AI Do - All about CUA

[Watch Webinar![icon](/wp-content/uploads/2023/04/Group-30743.png)](https://go.gigaspaces.com/webinarod/ecurely-unlocking-structured-data-for-enterprise-ai-0)

  * Resources![](https://www.gigaspaces.com/wp-content/uploads/2022/11/Path-16484.svg)

    * Content Hub

      * [Case Studies](/resources/case-studies)
      * [Webinars](https://www.gigaspaces.com/resources/webinars)
      * [Q&As](https://www.gigaspaces.com/answers)
      * [Videos](https://www.gigaspaces.com/resources/videos)
      * [Whitepapers & Brochures](https://www.gigaspaces.com/resources/brochures)
      * [Events](/resources/events/)
      * [Glossary](https://www.gigaspaces.com/data-terms)
      * [Blog](https://www.gigaspaces.com/blog)
      * [FAQs](https://www.gigaspaces.com/products/erag/faq)
      * [Technical Documentation](https://docs.gigaspaces.com/latest/landing.html)
    * [![vid-icon](https://www.gigaspaces.com/wp-content/uploads/2025/02/Resources-menu-thumb.jpg)](https://www.gigaspaces.com/blog/from-rag-to-tag)

Taking the AI leap from RAG to TAG

[Read the Blog![icon](/wp-content/uploads/2023/04/Group-30743.png)](https://www.gigaspaces.com/blog/from-rag-to-tag)

  * Company![](https://www.gigaspaces.com/wp-content/uploads/2022/11/Path-16484.svg)

    * Our Company

      * [About](https://www.gigaspaces.com/about)
      * [Customers](https://www.gigaspaces.com/customers)
      * [Management](https://www.gigaspaces.com/leadership)
      * [Board Members](https://www.gigaspaces.com/board-members)
      * [Investors](https://www.gigaspaces.com/investors)
      * [News](https://www.gigaspaces.com/news)
      * [Press Releases](/press-releases)
      * [Careers](https://www.gigaspaces.com/career-lobby)
    * col2

      * [Partners](https://www.gigaspaces.com/partners-lobby)
      * [OEM Partners](https://www.gigaspaces.com/partners-lobby/oem)
      * [System Integrators](https://www.gigaspaces.com/partners-lobby/system-integrators)
      * [Technology Partners](https://www.gigaspaces.com/partners-lobby/technology)
      * [Value Added Resellers](https://www.gigaspaces.com/partners-lobby/value-added-resellers)
      * Support & Services
      * [Services](https://www.gigaspaces.com/service-overview)
      * [Support](https://www.gigaspaces.com/support-lobby)
    * [![vid-icon](https://www.gigaspaces.com/wp-content/uploads/2025/02/Company-menu-thumb.jpg)](https://www.gigaspaces.com/news/gigaspaces-partners-with-ibm-and-aws-to-deliver-accurate-safe-genai-with-dbs)

GigaSpaces, IBM & AWS make AI safer

[Read How![icon](/wp-content/uploads/2023/04/Group-30743.png)](https://www.gigaspaces.com/news/gigaspaces-partners-with-ibm-and-aws-to-deliver-accurate-safe-genai-with-dbs)

  * [Book a Demo](https://www.gigaspaces.com/book-a-demo)



  * Products
    * Our Products
      * [eRAG](https://www.gigaspaces.com/products/erag)
        * [GenAI Catalyst](https://www.gigaspaces.com/products/genai-catalyst)
        * [Instant Data](https://www.gigaspaces.com/products/instant-data)
        * [Respond Proactively](https://www.gigaspaces.com/products/proactive-ai)
        * [Act Autonomously](https://www.gigaspaces.com/products/data-autonomy)
      * [Smart DIH](https://www.gigaspaces.com/products/smart-digital-integration-hub)
      * [XAP](https://www.gigaspaces.com/products/xap-imdg)
    * Solutions for
      * [Pharma](https://www.gigaspaces.com/products/erag/pharma)
      * [Procurement](https://www.gigaspaces.com/products/erag/procurement)
  * Solutions
    * [Digital Innovation Over Legacy Systems](https://www.gigaspaces.com/use-cases/digital-innovation-acceleration)
    * [Integration Data Hub](https://www.gigaspaces.com/technology/integration-data-hub)
    * [API Scaling](https://www.gigaspaces.com/use-cases/application-acceleration)
    * [Hybrid/Multi-cloud Integration](https://www.gigaspaces.com/use-cases/hybrid-or-multi-cloud-integration)
    * [Customer 360](https://www.gigaspaces.com/use-cases/customer-360)
    * [Retail](https://www.gigaspaces.com/retail-and-ecommerce-2)
    * [Financial Services](https://www.gigaspaces.com/financial-services)
    * [Insurance Companies](https://www.gigaspaces.com/industry-solutions/insurance)
  * How it Works
    * [eRAG Technology Overview](https://www.gigaspaces.com/how-it-works/erag-how-it-works)
      * [AI-Ready, IT-Friendly](https://www.gigaspaces.com/products/ai-ready)
      * [Semantic Reasoning](https://www.gigaspaces.com/how-it-works/semantic-reasoning)
      * [Questions to SQL Queries](https://www.gigaspaces.com/how-it-works/questions-into-sql-queries)
      * [Asked & Answered in Natural Language](https://www.gigaspaces.com/how-it-works/asked-answered-natural-language)
      * [Multiple Data Sources](https://www.gigaspaces.com/how-it-works/supported-data-sources)
      * [Governance](https://www.gigaspaces.com/how-it-works/proactive-ai-governance)
  * [Success Stories](https://www.gigaspaces.com/case-studies)
    * By Use Case
      * [Procurement](https://www.gigaspaces.com/use-cases/procurement-supply-chain)
      * [Operations](https://www.gigaspaces.com/use-cases/enhancing-pharmaceutical-operations)
      * [Budget Management](https://www.gigaspaces.com/use-cases/educational-organizations-budget-resources)
      * [Sales Operations](https://www.gigaspaces.com/use-cases/retail-boost-sales)
      * [Service Providers](https://www.gigaspaces.com/use-cases/contract-control-cost-savings)
      * [Utilities Management](https://www.gigaspaces.com/use-cases/cut-sla-penalties)
      * [Restaurant Management](https://www.gigaspaces.com/use-cases/power-smarter-restaurant-growth-erag)
    * By Industry
      * [Logistics](https://www.gigaspaces.com/case-studies/optimizing-supply-chain-with-ai-powered-insights)
      * [Pharma](https://www.gigaspaces.com/case-studies/transforming-patient-outcomes)
      * [Education](https://www.gigaspaces.com/case-studies/achieving-educational-budget-control)
      * [Retail](https://www.gigaspaces.com/case-studies/retail-intelligence-with-erag)
      * [Shipping](https://www.gigaspaces.com/case-studies/redefining-global-logistics)
      * [Energy](https://www.gigaspaces.com/case-studies/boosting-operations-reducing-penalties)
      * [Hospitality](https://www.gigaspaces.com/case-studies/restaurant-increase-profit-erag)
  * Resources
    * [Webinars](https://www.gigaspaces.com/resources/webinars)
    * [Videos](https://www.gigaspaces.com/resources/videos)
    * [Q&As](https://www.gigaspaces.com/answers)
    * [Whitepapers & Brochures](https://www.gigaspaces.com/resources/brochures)
    * [Customer Case Studies](https://www.gigaspaces.com/resources/case-studies)
    * [Events](https://www.gigaspaces.com/resources/events/)
    * [Glossary](https://www.gigaspaces.com/data-terms)
    * [FAQs](https://www.gigaspaces.com/products/erag/faq)
    * [Blog](https://www.gigaspaces.com/blog)
    * [Technical Documentation](https://docs.www.gigaspaces.com/latest/landing.html)
  * Company
    * [About](https://www.gigaspaces.com/about)
    * [Customers](https://www.gigaspaces.com/customers)
    * [Management](https://www.gigaspaces.com/leadership)
    * [Board Members](https://www.gigaspaces.com/board-members)
    * [Investors](https://www.gigaspaces.com/investors)
    * [News](https://www.gigaspaces.com/news)
    * [Press Releases](https://www.gigaspaces.com/press-releases)
    * [Careers](https://www.gigaspaces.com/career-lobby)
    * [Partners](https://www.gigaspaces.com/partners-lobby)
      * [OEM Partners](https://www.gigaspaces.com/partners-lobby/oem)
      * [System Integrators](https://www.gigaspaces.com/partners-lobby/system-integrators)
      * [Technology Partners](https://www.gigaspaces.com/partners-lobby/technology)
      * [Value Added Resellers](https://www.gigaspaces.com/partners-lobby/value-added-resellers)
    * Support & Services
      * [Services](https://www.gigaspaces.com/service-overview)
      * [Support](https://www.gigaspaces.com/support-lobby)
  * [Pricing](https://www.gigaspaces.com/pricing)
  * [Book a Demo](https://www.gigaspaces.com/?schedule-meeting-form=true)



Search for:

# Direct Preference Optimization (DPO)

[More Glossary Terms](https://www.gigaspaces.com/data-terms)

## What is Direct Preference Optimization (DPO)?

Direct Preference Optimization (DPO) is a new, simpler way to fine-tune large language models (LLMs) using human preferences. Unlike traditional methods like Reinforcement Learning from [Human Feedback](https://www.gigaspaces.com/data-terms/reinforcement-learning-human-feedback) (RLHF), DPO directly trains models to prefer certain responses over others without needing complex reward models or reinforcement learning steps. 

By focusing directly on human preferences, DPO training helps create AI systems that better understand and respond to human needs, while reducing the cost and complexity of model development. And, as AI models become central to business applications — from chatbots to AI assistants — direct preference optimization helps ensure these models align with human expectations safely and efficiently.

However, while DPO is simpler than RLHF overall, training still requires running inference through two models (reference and target), which can double forward-pass computation time compared to supervised [fine-tuning](https://www.gigaspaces.com/question/what-is-instruction-fine-tuning-how-it-works) alone.

## How Direct Preference Optimization Works

At its core, DPO training uses pairs of AI-generated responses where a human has indicated which one is better. Instead of creating a reward score for each response (as in RLHF), DPO focuses on these binary preferences to adjust the model directly.

For instance, if a user prefers ‘response A’ over ‘response B’, the DPO algorithm will update the model so that the preferred response becomes more likely in future interactions. This mechanism is fueled by the DPO equation, which is derived from the Bradley-Terry model of preferences and is used to align language models with human preferences. 

The equation optimizes the relative probabilities of preferred and dispreferred responses directly, without requiring a separate reward model or reinforcement learning steps, and mathematically increases the model’s probability of generating preferred outputs and lowers the probability of generating ones that are rejected.

Because DPO works directly on preferences, it avoids the need for a separate reward model and reinforcement learning loop, making DPO training simpler, faster, and more stable than RLHF.

## The Key Advantages of DPO Over RLHF

Although RLHF has been widely used to align AI models, DPO offers several key benefits that are driving its rapid adoption in AI development:

**Simplicity:** RLHF requires a multi-step process with a reward model and reinforcement learning. DPO reduces this to one step, working directly from preference pairs.

**Stability:** DPO training is more stable because it avoids reinforcement learning loops that can lead to unpredictable outputs, such as repetitive answers or "mode collapse.”

**Efficiency:** By removing the need for a reward model and reinforcement learning, DPO algorithms are much faster and less computationally intensive, cutting down on the cost and time of training large AI models.

**Performance:** Despite its simplicity, DPO LLMs often perform as well as or better than RLHF-tuned models when judged on human preference alignment, making DPO a practical and effective alternative.

**Transparency:** DPO training offers greater transparency because it is easier to trace how human preferences influence model updates, which is important for AI governance and compliance.

## Applications of DPO in AI Development

Direct Preference Optimization is being used across a growing range of AI development tasks, particularly for models that need to align closely with human expectations.

### Chatbots and Virtual Assistants

DPO can help fine-tune AI chatbots to generate polite, accurate, and helpful responses, mirroring user preferences directly instead of depending on unspecific, generic training datasets.

### Generating and Summarizing Content

AI models that are tasked with writing content or generating summaries can use DPO training to adjust their outputs based on readers’ preferences, such as more detailed or concise responses.

### Customer Support AI

In customer service, DPO LLMs help ensure AI-generated responses align with company policy and customer expectations, reducing errors and improving service quality.

### Code Generation and Developer Tools

AI coding tools and assistants use DPO algorithms to better understand which code suggestions developers find most useful, helping to increase accuracy and efficiency.

### AI Safety and Alignment Research

DPO is also a valuable tool for AI safety because it directly encodes human preferences into AI models without depending on unclear reward systems, which makes DPO training an excellent tool for entities that focus on developing ethical and trustworthy AI.

[![](https://www.gigaspaces.com/wp-content/uploads/2020/02/logo-header-2.svg)](https://www.gigaspaces.com/)

#### PRODUCTS & SOLUTIONS

  * Products
    * [eRAG](https://www.gigaspaces.com/products/erag)
    * [Smart DIH](https://www.gigaspaces.com/products/smart-digital-integration-hub/)
    * [XAP](https://www.gigaspaces.com/products/xap-imdg)



  * Our Technology
    * [Semantic Reasoning](https://www.gigaspaces.com/how-it-works/semantic-reasoning)
    * [Natural language to SQL](https://www.gigaspaces.com/how-it-works/asked-answered-natural-language)
    * [RAG for Structured Data](https://www.gigaspaces.com/how-it-works/erag-how-it-works)
    * [In-Memory Data Grid](https://www.gigaspaces.com/technology/in-memory-data-grid)
    * [Data Integration](https://www.gigaspaces.com/technology/data-integration)
    * [Data Operations by Multiple Access Methods](https://www.gigaspaces.com/technology/data-operations-by-multiple-access-methods)
    * [Unified Data Model](https://www.gigaspaces.com/technology/unified-data-model)
    * [Event-Driven Architecture](https://www.gigaspaces.com/technology/event-driven-architecture)



#### RESOURCES

  * [Resource Hub](https://www.gigaspaces.com/resources/)
  * [Webinars](https://www.gigaspaces.com/resources/webinars/)
  * [Q&As](https://www.gigaspaces.com/answers)
  * [Blogs](https://www.gigaspaces.com/blog/)
  * [FAQs](https://www.gigaspaces.com/faq-older)
  * [Videos](https://www.gigaspaces.com/resources/videos/)
  * [Whitepapers & Brochures](https://www.gigaspaces.com/resources/brochures/)
  * [Customer Case Studies](https://www.gigaspaces.com/resources/case-studies)
  * [Events](https://www.gigaspaces.com/resources/events/)
  * [Use Cases](https://www.gigaspaces.com/use-cases)
  * [Analyst Reports](https://www.gigaspaces.com/resources/analyst-reports/)
  * [Technical Documentation](https://docs.gigaspaces.com/latest/landing.html)



#### COMPANY

  * [About](https://www.gigaspaces.com/about/)
  * [Customers](https://www.gigaspaces.com/customers/)
  * [Management](https://www.gigaspaces.com/leadership/)
  * [Board Members](https://www.gigaspaces.com/board-members/)
  * [Investors](https://www.gigaspaces.com/investors/)
  * [News](https://www.gigaspaces.com/news/)
  * [Careers](https://www.gigaspaces.com/career-lobby/)
  * Contact Us
  * [Book A Demo](https://www.gigaspaces.com/book-a-demo)



  * [Partners](https://www.gigaspaces.com/partners-lobby/)
  * [OEM Partners](https://www.gigaspaces.com/partners-lobby/oem/)
  * [System Integrators](https://www.gigaspaces.com/partners-lobby/system-integrators/)
  * [Value Added Resellers](https://www.gigaspaces.com/partners-lobby/value-added-resellers/)
  * [Technology Partners](https://www.gigaspaces.com/partners-lobby/technology/)
  * [Support & Services](https://www.gigaspaces.com/support-lobby/)
  * [Services](https://www.gigaspaces.com/service-overview/)
  * [Support](https://www.gigaspaces.com/support-lobby/)



Copyright © GigaSpaces 2025 All rights reserved | [Privacy Policy](/privacy-policy/) | [Terms of Use](/terms-of-use/)  


[LinkedIn](https://www.linkedin.com/company/gigaspaces/ "LinkedIn")[X](https://twitter.com/GigaSpaces "X")[Facebook](https://www.facebook.com/GigaSpaces/ "Facebook")[YouTube](https://www.youtube.com/user/gigaspacestv "YouTube")

[]()

Manage your privacy

To provide the best experiences, we and our partners use technologies like cookies to store and/or access device information. Consenting to these technologies will allow us and our partners to process personal data such as browsing behavior or unique IDs on this site and show (non-) personalized ads. Not consenting or withdrawing consent, may adversely affect certain features and functions.

Click below to consent to the above or make granular choices. Your choices will be applied to this site only. You can change your settings at any time, including withdrawing your consent, by using the toggles on the Cookie Policy, or by clicking on the manage consent button at the bottom of the screen.

Functional Functional Always active 

The technical storage or access is strictly necessary for the legitimate purpose of enabling the use of a specific service explicitly requested by the subscriber or user, or for the sole purpose of carrying out the transmission of a communication over an electronic communications network.

Preferences Preferences

The technical storage or access is necessary for the legitimate purpose of storing preferences that are not requested by the subscriber or user.

Statistics Statistics

The technical storage or access that is used exclusively for statistical purposes. The technical storage or access that is used exclusively for anonymous statistical purposes. Without a subpoena, voluntary compliance on the part of your Internet Service Provider, or additional records from a third party, information stored or retrieved for this purpose alone cannot usually be used to identify you.

Marketing Marketing

The technical storage or access is required to create user profiles to send advertising, or to track the user on a website or across several websites for similar marketing purposes.

Statistics

Marketing

Features

Always active

Always active

  * Manage options
  * Manage services
  * Manage {vendor_count} vendors
  * [Read more about these purposes](https://cookiedatabase.org/tcf/purposes/)



Accept Deny Manage options Save preferences Manage options

  * {title}
  * {title}
  * {title}



Manage your privacy

To provide the best experiences, we use technologies like cookies to store and/or access device information. Consenting to these technologies will allow us to process data such as browsing behavior or unique IDs on this site. Not consenting or withdrawing consent, may adversely affect certain features and functions.

Functional Functional Always active 

The technical storage or access is strictly necessary for the legitimate purpose of enabling the use of a specific service explicitly requested by the subscriber or user, or for the sole purpose of carrying out the transmission of a communication over an electronic communications network.

Preferences Preferences

The technical storage or access is necessary for the legitimate purpose of storing preferences that are not requested by the subscriber or user.

Statistics Statistics

The technical storage or access that is used exclusively for statistical purposes. The technical storage or access that is used exclusively for anonymous statistical purposes. Without a subpoena, voluntary compliance on the part of your Internet Service Provider, or additional records from a third party, information stored or retrieved for this purpose alone cannot usually be used to identify you.

Marketing Marketing

The technical storage or access is required to create user profiles to send advertising, or to track the user on a website or across several websites for similar marketing purposes.

Statistics

Marketing

Features

Always active

Always active

  * Manage options
  * Manage services
  * Manage {vendor_count} vendors
  * [Read more about these purposes](https://cookiedatabase.org/tcf/purposes/)



Accept Deny Manage options Save preferences Manage options

  * {title}
  * {title}
  * {title}



Manage consent Manage consent

Skip to content

[ Open toolbar Accessibility Tools ](javascript:void\(0\); "Accessibility Tools")

Accessibility Tools

  * Increase TextIncrease Text
  * Decrease TextDecrease Text
  * GrayscaleGrayscale
  * High ContrastHigh Contrast
  * Negative ContrastNegative Contrast
  * Light BackgroundLight Background
  * Links UnderlineLinks Underline
  * Readable FontReadable Font
  * Reset Reset


  * [ SitemapSitemap ](https://www.gigaspaces.com/sitemap_index.xml)



![](https://www.gigaspaces.com/wp-content/uploads/2022/12/close-img.svg)

### Hey  
tell us what  
you need

![](https://www.gigaspaces.com/wp-content/uploads/2022/12/form-dude.svg)

You can unsubscribe from these communications at any time. For more information on how to unsubscribe, our privacy practices, and how we are committed to protecting and respecting your privacy, please review our [Privacy Policy.](https://www.gigaspaces.com/privacy-policy)

### Hey , tell us what you need

First name

Last name

Email

Country

Company name

Job Title

How can We help: Please let us know how We can help...

I agree to receive other communications from GigaSpaces.

You can unsubscribe from these communications at any time. For more information on how to unsubscribe, our privacy practices, and how we are committed to protecting and respecting your privacy, please review our [Privacy Policy.](https://www.gigaspaces.com/privacy-policy)

Oops! Something went wrong, please check email address (work email only).

Thank you!

We will get back to You shortly.

![](https://www.gigaspaces.com/wp-content/uploads/2022/12/close-img.svg)