# Source: https://docs.nvidia.com/nemo-framework/user-guide/latest/datacuration/api/decontamination.html.md

Title: Task Decontamination — NVIDIA NeMo Framework User Guide

URL Source: https://docs.nvidia.com/nemo-framework/user-guide/latest/datacuration/api/decontamination.html

Published Time: Fri, 18 Jul 2025 19:26:26 GMT

Markdown Content:
Task Decontamination[#](https://docs.nvidia.com/nemo-framework/user-guide/latest/datacuration/api/decontamination.html.md#task-decontamination "Link to this heading")
-------------------------------------------------------------------------------------------------------------------------------------------------------------------

Base Class[#](https://docs.nvidia.com/nemo-framework/user-guide/latest/datacuration/api/decontamination.html.md#base-class "Link to this heading")
-----------------------------------------------------------------------------------------------------------------------------------------------

_class_ nemo_curator.tasks.DownstreamTask[#](https://docs.nvidia.com/nemo-framework/user-guide/latest/datacuration/api/decontamination.html.md#nemo_curator.tasks.DownstreamTask "Link to this definition")_class_ nemo_curator.tasks.import_task(_task\_path:str_)[#](https://docs.nvidia.com/nemo-framework/user-guide/latest/datacuration/api/decontamination.html.md#nemo_curator.tasks.import_task "Link to this definition")
Module[#](https://docs.nvidia.com/nemo-framework/user-guide/latest/datacuration/api/decontamination.html.md#module "Link to this heading")
---------------------------------------------------------------------------------------------------------------------------------------

_class_ nemo_curator.TaskDecontamination(_tasks:[DownstreamTask](https://docs.nvidia.com/nemo-framework/user-guide/latest/datacuration/api/decontamination.html.md#nemo\_curator.tasks.DownstreamTask "nemo\_curator.tasks.downstream\_task.DownstreamTask")|Iterable[[DownstreamTask](https://docs.nvidia.com/nemo-framework/user-guide/latest/datacuration/api/decontamination.html.md#nemo\_curator.tasks.DownstreamTask "nemo\_curator.tasks.downstream\_task.DownstreamTask")]_,_text\_field:str='text'_,_max\_ngram\_size:int=13_,_max\_matches:int=10_,_min\_document\_length:int=200_,_remove\_char\_each\_side:int=200_,_max\_splits:int=10_,_removed\_dir:str|None=None_,)[#](https://docs.nvidia.com/nemo-framework/user-guide/latest/datacuration/api/decontamination.html.md#nemo_curator.TaskDecontamination "Link to this definition")call(_dataset:[DocumentDataset](https://docs.nvidia.com/nemo-framework/user-guide/latest/datacuration/api/datasets.html.md#nemo\_curator.datasets.DocumentDataset "nemo\_curator.datasets.doc\_dataset.DocumentDataset")_,)→[DocumentDataset](https://docs.nvidia.com/nemo-framework/user-guide/latest/datacuration/api/datasets.html.md#nemo_curator.datasets.DocumentDataset "nemo_curator.datasets.doc_dataset.DocumentDataset")[#](https://docs.nvidia.com/nemo-framework/user-guide/latest/datacuration/api/decontamination.html.md#nemo_curator.TaskDecontamination.call "Link to this definition")
Performs an arbitrary operation on a dataset

Parameters:
**dataset** ([_DocumentDataset_](https://docs.nvidia.com/nemo-framework/user-guide/latest/datacuration/api/datasets.html.md#nemo_curator.datasets.DocumentDataset "nemo_curator.datasets.DocumentDataset")) – The dataset to operate on

prepare_task_ngram_count()→dict[#](https://docs.nvidia.com/nemo-framework/user-guide/latest/datacuration/api/decontamination.html.md#nemo_curator.TaskDecontamination.prepare_task_ngram_count "Link to this definition")
Computes a dictionary of all ngrams in each task as keys and each value set to 0.

Tasks[#](https://docs.nvidia.com/nemo-framework/user-guide/latest/datacuration/api/decontamination.html.md#tasks "Link to this heading")
-------------------------------------------------------------------------------------------------------------------------------------

_class_ nemo_curator.tasks.Race(_min\_ngram\_size:int=8_, _max\_ngram\_size:int=13_)[#](https://docs.nvidia.com/nemo-framework/user-guide/latest/datacuration/api/decontamination.html.md#nemo_curator.tasks.Race "Link to this definition")_class_ nemo_curator.tasks.Squad(_min\_ngram\_size:int=8_, _max\_ngram\_size:int=13_)[#](https://docs.nvidia.com/nemo-framework/user-guide/latest/datacuration/api/decontamination.html.md#nemo_curator.tasks.Squad "Link to this definition")_class_ nemo_curator.tasks.ArcEasy(_min\_ngram\_size:int=8_, _max\_ngram\_size:int=13_)[#](https://docs.nvidia.com/nemo-framework/user-guide/latest/datacuration/api/decontamination.html.md#nemo_curator.tasks.ArcEasy "Link to this definition")_class_ nemo_curator.tasks.ArcChallenge(_min\_ngram\_size:int=8_, _max\_ngram\_size:int=13_)[#](https://docs.nvidia.com/nemo-framework/user-guide/latest/datacuration/api/decontamination.html.md#nemo_curator.tasks.ArcChallenge "Link to this definition")_class_ nemo_curator.tasks.OpenBookQA(_min\_ngram\_size:int=8_, _max\_ngram\_size:int=13_)[#](https://docs.nvidia.com/nemo-framework/user-guide/latest/datacuration/api/decontamination.html.md#nemo_curator.tasks.OpenBookQA "Link to this definition")_class_ nemo_curator.tasks.BoolQ(_min\_ngram\_size:int=8_, _max\_ngram\_size:int=13_)[#](https://docs.nvidia.com/nemo-framework/user-guide/latest/datacuration/api/decontamination.html.md#nemo_curator.tasks.BoolQ "Link to this definition")_class_ nemo_curator.tasks.Copa(_min\_ngram\_size:int=8_, _max\_ngram\_size:int=13_)[#](https://docs.nvidia.com/nemo-framework/user-guide/latest/datacuration/api/decontamination.html.md#nemo_curator.tasks.Copa "Link to this definition")_class_ nemo_curator.tasks.RTE(_min\_ngram\_size:int=8_, _max\_ngram\_size:int=13_)[#](https://docs.nvidia.com/nemo-framework/user-guide/latest/datacuration/api/decontamination.html.md#nemo_curator.tasks.RTE "Link to this definition")_class_ nemo_curator.tasks.MultiRC(_min\_ngram\_size:int=8_, _max\_ngram\_size:int=13_)[#](https://docs.nvidia.com/nemo-framework/user-guide/latest/datacuration/api/decontamination.html.md#nemo_curator.tasks.MultiRC "Link to this definition")_class_ nemo_curator.tasks.WSC(_min\_ngram\_size:int=8_, _max\_ngram\_size:int=13_)[#](https://docs.nvidia.com/nemo-framework/user-guide/latest/datacuration/api/decontamination.html.md#nemo_curator.tasks.WSC "Link to this definition")_class_ nemo_curator.tasks.CB(_min\_ngram\_size:int=8_, _max\_ngram\_size:int=13_)[#](https://docs.nvidia.com/nemo-framework/user-guide/latest/datacuration/api/decontamination.html.md#nemo_curator.tasks.CB "Link to this definition")_class_ nemo_curator.tasks.ANLI(_min\_ngram\_size:int=8_, _max\_ngram\_size:int=13_)[#](https://docs.nvidia.com/nemo-framework/user-guide/latest/datacuration/api/decontamination.html.md#nemo_curator.tasks.ANLI "Link to this definition")_class_ nemo_curator.tasks.Record(_min\_ngram\_size:int=8_, _max\_ngram\_size:int=13_)[#](https://docs.nvidia.com/nemo-framework/user-guide/latest/datacuration/api/decontamination.html.md#nemo_curator.tasks.Record "Link to this definition")_class_ nemo_curator.tasks.COQA(_file\_path:str|None=None_,_min\_ngram\_size:int=8_,_max\_ngram\_size:int=13_,)[#](https://docs.nvidia.com/nemo-framework/user-guide/latest/datacuration/api/decontamination.html.md#nemo_curator.tasks.COQA "Link to this definition")_class_ nemo_curator.tasks.TriviaQA(_min\_ngram\_size:int=8_, _max\_ngram\_size:int=13_)[#](https://docs.nvidia.com/nemo-framework/user-guide/latest/datacuration/api/decontamination.html.md#nemo_curator.tasks.TriviaQA "Link to this definition")_class_ nemo_curator.tasks.Quac(_min\_ngram\_size:int=8_, _max\_ngram\_size:int=13_)[#](https://docs.nvidia.com/nemo-framework/user-guide/latest/datacuration/api/decontamination.html.md#nemo_curator.tasks.Quac "Link to this definition")_class_ nemo_curator.tasks.WebQA(_min\_ngram\_size:int=8_, _max\_ngram\_size:int=13_)[#](https://docs.nvidia.com/nemo-framework/user-guide/latest/datacuration/api/decontamination.html.md#nemo_curator.tasks.WebQA "Link to this definition")_class_ nemo_curator.tasks.Drop(_min\_ngram\_size:int=8_, _max\_ngram\_size:int=13_)[#](https://docs.nvidia.com/nemo-framework/user-guide/latest/datacuration/api/decontamination.html.md#nemo_curator.tasks.Drop "Link to this definition")_class_ nemo_curator.tasks.WiC(_min\_ngram\_size:int=8_, _max\_ngram\_size:int=13_)[#](https://docs.nvidia.com/nemo-framework/user-guide/latest/datacuration/api/decontamination.html.md#nemo_curator.tasks.WiC "Link to this definition")_class_ nemo_curator.tasks.MMLU(_path:str|None=None_,_min\_ngram\_size:int=8_,_max\_ngram\_size:int=13_,)[#](https://docs.nvidia.com/nemo-framework/user-guide/latest/datacuration/api/decontamination.html.md#nemo_curator.tasks.MMLU "Link to this definition")_class_ nemo_curator.tasks.BigBenchHard(_path:str|None=None_,_min\_ngram\_size:int=8_,_max\_ngram\_size:int=13_,)[#](https://docs.nvidia.com/nemo-framework/user-guide/latest/datacuration/api/decontamination.html.md#nemo_curator.tasks.BigBenchHard "Link to this definition")_class_ nemo_curator.tasks.BigBenchLight(_path:str|None=None_,_min\_ngram\_size:int=8_,_max\_ngram\_size:int=13_,)[#](https://docs.nvidia.com/nemo-framework/user-guide/latest/datacuration/api/decontamination.html.md#nemo_curator.tasks.BigBenchLight "Link to this definition")_class_ nemo_curator.tasks.Multilingual(_path:str|None=None_,_min\_ngram\_size:int=8_,_max\_ngram\_size:int=13_,)[#](https://docs.nvidia.com/nemo-framework/user-guide/latest/datacuration/api/decontamination.html.md#nemo_curator.tasks.Multilingual "Link to this definition")_class_ nemo_curator.tasks.PIQA(_min\_ngram\_size:int=8_, _max\_ngram\_size:int=13_)[#](https://docs.nvidia.com/nemo-framework/user-guide/latest/datacuration/api/decontamination.html.md#nemo_curator.tasks.PIQA "Link to this definition")_class_ nemo_curator.tasks.Winogrande(_min\_ngram\_size:int=8_, _max\_ngram\_size:int=13_)[#](https://docs.nvidia.com/nemo-framework/user-guide/latest/datacuration/api/decontamination.html.md#nemo_curator.tasks.Winogrande "Link to this definition")_class_ nemo_curator.tasks.Lambada(_file\_path:str_,_min\_ngram\_size:int=8_,_max\_ngram\_size:int=13_,)[#](https://docs.nvidia.com/nemo-framework/user-guide/latest/datacuration/api/decontamination.html.md#nemo_curator.tasks.Lambada "Link to this definition")_class_ nemo_curator.tasks.NumDasc(_n:int_,_file\_path:str_,_min\_ngram\_size:int=8_,_max\_ngram\_size:int=13_,)[#](https://docs.nvidia.com/nemo-framework/user-guide/latest/datacuration/api/decontamination.html.md#nemo_curator.tasks.NumDasc "Link to this definition")_class_ nemo_curator.tasks.StoryCloze(_file\_path:str_,_min\_ngram\_size:int=8_,_max\_ngram\_size:int=13_,)[#](https://docs.nvidia.com/nemo-framework/user-guide/latest/datacuration/api/decontamination.html.md#nemo_curator.tasks.StoryCloze "Link to this definition")

Links/Buttons:
- [#](https://docs.nvidia.com/nemo-framework/user-guide/latest/datacuration/api/decontamination.html.md#nemo_curator.tasks.StoryCloze)
- [DocumentDataset](https://docs.nvidia.com/nemo-framework/user-guide/latest/datacuration/api/datasets.html.md#nemo_curator.datasets.DocumentDataset)