Toggle navigation sidebar
Toggle in-page Table of Contents
Ray 3.0.0.dev0
Welcome to Ray!
Ray
Overview
Getting Started
Installation
Use Cases
Example Gallery
Ecosystem
Ray Core
Ray Data
Ray Train
Key Concepts
Distributed PyTorch
More Frameworks
Ray Train Internals
Examples
Ray Train FAQ
Ray Train API
ray.train.trainer.BaseTrainer
ray.train.data_parallel_trainer.DataParallelTrainer
ray.train.DataConfig
ray.train.gbdt_trainer.GBDTTrainer
ray.train.trainer.BaseTrainer.fit
ray.train.trainer.BaseTrainer.setup
ray.train.trainer.BaseTrainer.preprocess_datasets
ray.train.trainer.BaseTrainer.training_loop
ray.train.trainer.BaseTrainer.as_trainable
ray.train.backend.Backend
ray.train.backend.BackendConfig
ray.train.context.TrainContext
ray.train.get_context
ray.train.get_dataset_shard
ray.train.report
ray.train.Checkpoint
ray.train.get_context
ray.train.context.TrainContext
ray.train.torch.TorchTrainer
ray.train.torch.TorchConfig
ray.train.torch.TorchCheckpoint
ray.train.torch.prepare_model
ray.train.torch.prepare_optimizer
ray.train.torch.prepare_data_loader
ray.train.torch.get_device
ray.train.torch.accelerate
ray.train.torch.backward
ray.train.torch.enable_reproducibility
ray.train.lightning.LightningTrainer
ray.train.lightning.LightningConfigBuilder
ray.train.lightning.LightningCheckpoint
ray.train.lightning.LightningPredictor
ray.train.tensorflow.TensorflowTrainer
ray.train.tensorflow.TensorflowConfig
ray.train.tensorflow.TensorflowCheckpoint
ray.train.tensorflow.prepare_dataset_shard
ray.train.horovod.HorovodTrainer
ray.train.horovod.HorovodConfig
ray.train.xgboost.XGBoostTrainer
ray.train.xgboost.XGBoostCheckpoint
ray.train.lightgbm.LightGBMTrainer
ray.train.lightgbm.LightGBMCheckpoint
ray.train.huggingface.TransformersTrainer
ray.train.huggingface.TransformersCheckpoint
ray.train.huggingface.TransformersCheckpoint.__init__
ray.train.huggingface.TransformersCheckpoint.as_directory
ray.train.huggingface.TransformersCheckpoint.from_bytes
ray.train.huggingface.TransformersCheckpoint.from_checkpoint
ray.train.huggingface.TransformersCheckpoint.from_dict
ray.train.huggingface.TransformersCheckpoint.from_directory
ray.train.huggingface.TransformersCheckpoint.from_model
ray.train.huggingface.TransformersCheckpoint.from_uri
ray.train.huggingface.TransformersCheckpoint.get_internal_representation
ray.train.huggingface.TransformersCheckpoint.get_model
ray.train.huggingface.TransformersCheckpoint.get_preprocessor
ray.train.huggingface.TransformersCheckpoint.get_tokenizer
ray.train.huggingface.TransformersCheckpoint.get_training_arguments
ray.train.huggingface.TransformersCheckpoint.set_preprocessor
ray.train.huggingface.TransformersCheckpoint.to_bytes
ray.train.huggingface.TransformersCheckpoint.to_dict
ray.train.huggingface.TransformersCheckpoint.to_directory
ray.train.huggingface.TransformersCheckpoint.to_uri
ray.train.huggingface.TransformersCheckpoint.path
ray.train.huggingface.TransformersCheckpoint.uri
ray.train.huggingface.AccelerateTrainer
ray.train.sklearn.SklearnTrainer
ray.train.sklearn.SklearnCheckpoint
ray.train.mosaic.MosaicTrainer
ray.train.trainer.BaseTrainer.restore
ray.train.data_parallel_trainer.DataParallelTrainer.restore
Ray Tune
Ray Serve
Ray RLlib
More Libraries
Ray Clusters
Monitoring and Debugging
References
Developer Guides
repository
open issue
suggest edit
.rst
.pdf
ray.train.huggingface.TransformersCheckpoint.get_tokenizer
ray.train.huggingface.TransformersCheckpoint.get_tokenizer
#
TransformersCheckpoint.
get_tokenizer
(
tokenizer
:
Type
[
transformers.tokenization_utils.PreTrainedTokenizer
]
,
**
kwargs
)
→
Optional
[
transformers.tokenization_utils.PreTrainedTokenizer
]
[source]
#
Create a tokenizer using the data stored in this checkpoint.