Basic Rclone commands

• rclone config – Enter an interactive configuration session.
• rclone copy – Copy files from source to dest, skipping already copied.
• rclone sync – Make source and dest identical, modifying destination only.
• rclone move – Move files from source to dest.
• rclone delete – Remove the contents of path.
• rclone purge – Remove the path and all of its contents.
• rclone mkdir – Make the path if it doesn’t already exist.
• rclone rmdir – Remove the path.
• rclone ls – List all the objects in the path with size and path.
• rclone lsd – List all directories/containers/buckets in the path

For this example, to copy data between Box and your home directory you could do the following. The remote for Box is this instance is called umichbox. The basic structure for the copy command is this:

rclone copy source:sourcepath dest:destpath

To copy from box to your home directory:

• rclone copy umichbox:/<item to be copied> /home/<uniqname>/

From your home directory to box:

• rclone copy /home/<uniqname>/<item to be copied> umichbox:/

More commands and instructions on command usage can be found here.

Rclone on the clusters

We offer Rclone on our Great Lakes and Cavium clusters.

To access Rclone on Great Lakes please run the following command:

• module load rclone

To access Rclone on the Cavium cluster you will need to paste the following into your command line:

• /sw/dai/aarch64/centos7/rclone-v1.51.0-linux-arm64/rclone

You can also set an alias in your .bashrc file for this so you don’t have to copy and paste or type the whole path. Information on setting aliases in Linux can be found here.

When choosing to create a new remote for Google Drive you should start with the creation of a client ID for your Google Drive if you do not have one already. Instructions on how to set up your own client ID for your Google Derive can be found here. If you choose to leave the client ID blank please be aware that by doing so you share in the public rclone ID which could degrade performance.

After choosing your option for a client ID you’ll need to decide if you want a Client Secret. This is essentially a password. Default is to not set one:

The next step is to choose the level of access that wish to grant cclone. Leaving the selection blank will choose the default but the selections on the screen do not inform you what the default is. It is recommended to choose full access (selection 1) to start as this can be edited later:

The next few steps you will leave blank and choose ‘n’ for both ‘Edit advanced config’ and ‘Use auto config’. This will bring up a link for a webpage that you will need to cut and paste into a browser window. Once you have the webpage open in a browser you will be asked to choose the email address you would like to log in with. Choose your UMich email address. This will take you to a page where you are asked to grant rclone access to your Google account:

Click the ‘Allow’ button at the bottom of the page. This will take you to a page with the verification code to copy back into your active session rclone session on Great Lakes.

The next option is to configure your drive as a team drive. If the remote you are creating is for a team drive then select ‘y’ and follow the remaining instructions. If you select ‘n’, the next option is to verify that the drive information is correct. Once you’ve done that your Rclone Google Drive remote is all set.

The first several steps for setting up a remote for DropBox do not require any special input so you can choose the defaults. For Advanced Config and Auto config choose ‘No’. Great Lakes does not allow for the browser window to open that is required to allow Rclone access to DropBox. You will need a local terminal session open to complete the setup process. Instructions on installing Rclone on your local machine can be found here.

After you have Rclone installed on your local machine you will need to type the following on the command line:

rclone authorize “dropbox”

This will launch a browser window on your local machine. Once you have the browser window open you will be required to login into your DropBox account:

Enter your email into the ‘Email’ field and then click on the ‘Password’ field. This will automatically change it to say the following:

Click the ‘Continue’ button and you will then see this:

Click ‘Allow. This will then prompt your local Rclone session to show an access token. Copy and paste that into your Great Lakes Rclone session. Once that is pasted into your Great Lakes Rclone session choose ‘y’. Your DropBox remote is now set up and ready to use.

The first several steps for setting up a remote for Box do not require any special input so you can choose the defaults. For Advanced Config and Auto config choose ‘No’. Great Lakes doe not allow for the browser window to open that is required to allow Rclone access to Box. You will need a local terminal session open to complete the setup process. Instructions on installing Rclone on your local machine can be found here.

After you have Rclone installed on your local machine you will need to type the following on the command line:

rclone authorize “box”

This will launch a browser window on your local machine. Once you have the browser window open you will be required to login into your Box storage:

If using a UMich Box account click on the ‘User Single Sign On (SSO)’ and enter your UMich email address. You will then be taken to a page with a button that says ‘Grant Access to Box’. Click this and your local Rclone session will be updated and say this:

Copy the access token and paste it into your Rclone session on Great Lakes. Your Rclone session on Great Lakes will now show this:

Once you choose ‘y’ your Rclone remote for Box will be created.

The following instructions are for getting started with RClone on our Great Lakes cluster.

Rclone is a command line utility so the following command will need to be run from your command line prompt:

1. module load rclone
2. rclone config

Rclone config will prompt you to choose one of the following:

Choose ‘n’ to create a new remote.

You will then be asked to choose a name for the remote you are going to create. It is recommended that the name you choose reflects the type of storage you are configuring Rclone for. For example, if you are going to create a remote for Box storage through UMich then name the remote ‘UmichBox’. This will help you keep track of which remote is tied to a specific storage.

Each type of remote created requires some of the same steps. One important step is retrieving a key from a pop up browser. Great Lakes does not allow for a browser to pop up so you will need a separate local terminal session open on your own machine and requires for you to have Rclone installed on your local machine as well. Instructions on installing Rclone on your local machine can be found here.

Specific steps for setting up different storage remotes will be shown in their respective sections.

What can rclone do for you?

Rclone helps you:

• Backup (and encrypt) files to cloud storage
• Restore (and decrypt) files from cloud storage
• Mirror cloud data to other cloud services or locally
• Migrate data to cloud, or between cloud storage vendors
• Mount multiple, encrypted, cached or diverse cloud storage as a disk
• Analyse and account for data held on cloud storage using lsf, ljson, size, ncdu
• Union file systems together to present multiple local and/or cloud file systems as one

Features

• Transfers
  • MD5, SHA1 hashes are checked at all times for file integrity
  • Timestamps are preserved on files
  • Operations can be restarted at any time
  • Can be to and from network, eg two different cloud providers
  • Can use multi-threaded downloads to local disk
• Copy new or changed files to cloud storage
• Sync (one way) to make a directory identical
• Move files to cloud storage deleting the local after verification
• Check hashes and for missing/extra files
• Mount your cloud storage as a network disk
• Serve local or remote files over HTTP/WebDav/FTP/SFTP/dlna
• Experimental Web based GUI