ray.tune.syncer.Syncer#

class ray.tune.syncer.Syncer(sync_period: float = 300, sync_timeout: float = 1800)[source]#

Bases: abc.ABC

Syncer class for synchronizing data between Ray nodes and remote (cloud) storage.

This class handles data transfer for two cases:

  1. Synchronizing data such as experiment checkpoints from the driver to cloud storage.

  2. Synchronizing data such as trial checkpoints from remote trainables to cloud storage.

Synchronizing tasks are usually asynchronous and can be awaited using wait(). The base class implements a wait_or_retry() API that will retry a failed sync command.

The base class also exposes an API to only kick off syncs every sync_period seconds.

Parameters

  • sync_period – The minimum time in seconds between sync operations, as used by sync_up/down_if_needed.

  • sync_timeout – The maximum time to wait for a sync process to finish before issuing a new sync operation. Ex: should be used by wait if launching asynchronous sync tasks.

DeveloperAPI: This API may change across minor Ray releases.

Methods

delete(remote_dir)

Delete directory on remote storage.

retry()

Retry the last sync up, sync down, or delete command.

sync_down(remote_dir, local_dir[, exclude])

Synchronize remote directory to local directory.

sync_down_if_needed(remote_dir, local_dir[, ...])

Syncs down if time since last sync down is greater than sync_period.

sync_up(local_dir, remote_dir[, exclude])

Synchronize local directory to remote directory.

sync_up_if_needed(local_dir, remote_dir[, ...])

Syncs up if time since last sync up is greater than sync_period.

validate_upload_dir(upload_dir)

Checks if upload_dir is supported by the Syncer.

wait()

Wait for asynchronous sync command to finish.