Backup files to Google Drive with Rclone

Backup files to google drive using rclone. Before we get right to this, let us break down the keywords first.

Backup

In information technology, a backup, or data backup is a copy of computer data taken and stored elsewhere. That way, it may be used to restore the original after a data loss event. The verb form referring to the process is known as “back up”. Whereas the noun and adjective form for the same process is “backup”.

Backups can be used to recover lost data as a result of data deletion, corruption, and also to recover data from an earlier time.[2] Backups provide a simple form of disaster recovery. However, not all backup systems can reconstitute a computer system or other complex configuration. Some of which is a computer cluster, active directory server, or database server.

Source: https://en.wikipedia.org/wiki/Backup

Google Drive

Google Drive is a file storage and synchronization service developed by Google. Launched on April 24, 2012, Google Drive allows users to store files on their servers, synchronize files across devices, and share files. In addition to a website, Google Drive offers apps with offline capabilities for Windows and macOS computers, and Android and iOS smartphones and tablets. Google Drive encompasses Google Docs, Google Sheets, and Google Slides, which are a part of the Google Docs Editors office suite that permits collaborative editing of documents, spreadsheets, presentations, drawings, forms, and more.

Drive offers users 15 GB of free storage through Google One. Google One also offers 100 GB, 200 GB, 2 TB, 10 TB, 20 TB, and 30 TB, offered through optional paid plans.

Rclone

Rclone is an open-source, multi-threaded, command-line computer program to manage content on the cloud and other high latency storage. Its capabilities include sync, transfer, crypt, cache, union, compress and mount. The rclone website lists fifty supported backends including S3 services and Google Drive.

Descriptions of rclone often carry the strapline Rclone syncs your files to cloud storage. Those before 2020 include the alternative Rsync for Cloud Storage. Users have called rclone The Swiss Army Knife of cloud storage.

Rclone is well known for its rclone sync and rclone mount commands. It provides further management functions analogous to those ordinarily used for files on local disks, but which tolerate some intermittent and unreliable service. Rclone file transfers optimise for high latency networks. Rclone is commonly a front-end for media servers such as Plex, Emby or Jellyfin to stream content direct from consumer file storage services

Requirements

Before we begin the process of backing up our files on google drive with the help of the swiss army knife (rclone), we need to get the following fundamentals in place.

• A Google account with google drive access
• Rclone installed on the device with the backup files
• File to backup available in one specific location

Google Account

Skip this if you have a google account. All google accounts give you access to their google drive service.

This particular tutorial concentrate on backing up your files to google drive. Hence, we do need a google account to have access to their drive service. The entire process of creating a google should take less than 5 minutes to complete. Follow the create a google account link below to have your account created

Create your Google Account

Rclone Installation

Rclone has various installation methods based on the host system. It works on Windows, macOS, Linux and FreeBSD operating systems. Installation and download instructions for all supported operating systems are available from the official documentation page. Follow the appropriate instructions for your host system.

Step 1: Mounting Google Drive

We are proceeding with the assumption that all previous requirements have been met. You have rclone installed on your host computer and your backup file have been prepared. Rclone installation can be confirmed by running the following command in the terminal/command prompt

rclone version

The above command should print the rclone version number, go version (go lang) and host operating system.

Let us proceed with the process to mount remote ( google drive) as a file system on our host system. The rclone config command provides an interactive wizard that guides through the process of successfully adding a remote as a local file system. Run the rclone config command from your terminal/command prompt

rclone config 

No remotes found - make a new one
n) New remote
s) Set configuration password
q) Quit config
n/s/q> n

The first prompt in the rclone config interactive wizard issues a list option to add, rename, copy remote and set configuration password. Enter n to create a new remote. To confirm our selection let us hit the enter button on the keyboard

Next, provide a name to identify the remote. Let us give the name gdrive

name> gdrive

Cloud drive types in rclone

After successfully setting the name parameter need to select the drive type we want to configure. As captured in the title of this article we need to select the google drive option. Google drive is option number 13 which has a string value of “drive” hence let us type drive into the prompt and hit the enter key as always.

Type of storage to configure.
Enter a string value. Press Enter for the default ("").
Choose a number from below, or type in your own value
 1 / 1Fichier
   \\ "fichier"
 2 / Alias for an existing remote
   \\ "alias"
 3 / Amazon Drive
   \\ "amazon cloud drive"
 4 / Amazon S3 Compliant Storage Provider (AWS, Alibaba, Ceph, Digital Ocean, Dreamhost, IBM COS, Minio, Tencent COS, etc)
   \\ "s3"
 5 / Backblaze B2
   \\ "b2"
 6 / Box
   \\ "box"
 7 / Cache a remote
   \\ "cache"
 8 / Citrix Sharefile
   \\ "sharefile"
 9 / Dropbox
   \\ "dropbox"
10 / Encrypt/Decrypt a remote
   \\ "crypt"
11 / FTP Connection
   \\ "ftp"
12 / Google Cloud Storage (this is not Google Drive)
   \\ "google cloud storage"
13 / Google Drive
   \\ "drive"
14 / Google Photos
   \\ "google photos"
15 / Hubic
   \\ "hubic"
16 / In memory object storage system.
   \\ "memory"
17 / Jottacloud
   \\ "jottacloud"
18 / Koofr
   \\ "koofr"
19 / Local Disk
   \\ "local"
20 / Mail.ru Cloud
   \\ "mailru"
21 / Mega
   \\ "mega"
22 / Microsoft Azure Blob Storage
   \\ "azureblob"
23 / Microsoft OneDrive
   \\ "onedrive"
24 / OpenDrive
   \\ "opendrive"
25 / OpenStack Swift (Rackspace Cloud Files, Memset Memstore, OVH)
   \\ "swift"
26 / Pcloud
   \\ "pcloud"
27 / Put.io
   \\ "putio"
28 / QingCloud Object Storage
   \\ "qingstor"
29 / SSH/SFTP Connection
   \\ "sftp"
30 / Sugarsync
   \\ "sugarsync"
31 / Tardigrade Decentralized Cloud Storage
   \\ "tardigrade"
32 / Transparently chunk/split large files
   \\ "chunker"
33 / Union merges the contents of several upstream fs
   \\ "union"
34 / Webdav
   \\ "webdav"
35 / Yandex Disk
   \\ "yandex"
36 / http Connection
   \\ "http"
37 / premiumize.me
   \\ "premiumizeme"
38 / seafile
   \\ "seafile"
Storage> drive 

Continuing from the last step we will need to accept the default for the next two prompts by hitting the enter key twice in regards to the client id and the client secret.

** See help for drive backend at: <https://rclone.org/drive/> **

Google Application Client Id
Setting your own is recommended.
See <https://rclone.org/drive/#making-your-own-client-id> for how to create your own.
If you leave this blank, it will use an internal key which is low performance.
Enter a string value. Press Enter for the default ("").
client_id>
OAuth Client Secret
Leave blank normally.
Enter a string value. Press Enter for the default ("").
client_secret>

Scope definition is the next step in the configuration wizard. We define the permission or access level the rclone application has on the google drive platform. Full access to all files is what we want hence we need to choose option 1 and proceed by hitting the almighty enter key.

Scope that rclone should use when requesting access from drive.
Enter a string value. Press Enter for the default ("").
Choose a number from below, or type in your own value
 1 / Full access all files, excluding Application Data Folder.
   \\ "drive"
 2 / Read-only access to file metadata and file contents.
   \\ "drive.readonly"
   / Access to files created by rclone only.
 3 | These are visible in the drive website.
   | File authorization is revoked when the user deauthorizes the app.
   \\ "drive.file"
   / Allows read and write access to the Application Data folder.
 4 | This is not visible in the drive website.
   \\ "drive.appfolder"
   / Allows read-only access to file metadata but
 5 | does not allow any access to read or download file content.
   \\ "drive.metadata.readonly"
scope> 1

The root folder provides a base folder for the to be backup files. This should be an existing folder on the google drive platform. If one does not exist we need to create one and type the folder name into the root folder prompt. For our example, we will use “site_performance_result” as our backup folder.

ID of the root folder
Leave blank normally.

Fill in to access "Computers" folders (see docs), or for rclone to use
a non root folder as its starting point.

Enter a string value. Press Enter for the default ("").
root_folder_id> site_performance_result

Let us go with the default value by hitting the enter key.

Service Account Credentials JSON file path
Leave blank normally.
Needed only if you want use SA instead of interactive login.

Leading `~` will be expanded in the file name as will environment variables such as `${RCLONE_CONFIG_DIR}`.

Enter a string value. Press Enter for the default ("").
service_account_file>

Editing the advanced configurations is not relevant to us now therefore we skip it by typing “n” into the prompt to move to the next item on the configuration wizard.

Edit advanced config? (y/n)
y) Yes
n) No (default)
y/n> n

Next up is the remote configurations, here we authenticate rclone and give it access to our google drive folder. On a headless machine that has no graphical user interface and a browser, we might want to use the second option. I personally prefer the headless machine option so I enter n into the prompt and proceed to login.

Remote config
Use auto config?
 * Say Y if not sure
 * Say N if you are working on a remote or headless machine
y) Yes (default)
n) No
y/n> n

Copy and paste link echoed to the command prompt/ terminal into a browser to get the verification codes. In the browser follow the prompt and give the right permission. The verification code will show in the browser, copy the verification code and paste in the terminal/command prompt.

This is not a team drive hence we do not need to configure the team drive option. Let say no by typing “n” into the prompt to proceed.

Configure this as a team drive?
y) Yes
n) No (default)
y/n> n

Finally, let confirm the configuration details by entering “y” into the prompt and hit the enter key for the last time.

y) Yes this is OK (default)
e) Edit this remote
d) Delete this remote
y/e/d> y

On the next prompt, you should see the drive name given above in the list of remote drives. Let complete the process by typing the “q” to quitting the wizard.

Current remotes:

Name                 Type
====                 ====
gdrive               drive

e) Edit existing remote
n) New remote
d) Delete remote
r) Rename remote
c) Copy remote
s) Set configuration password
q) Quit config
e/n/d/r/c/s/q> q

Following all the aforementioned steps help to successfully install and config google drive remote on rclone. Now that rclone installed and a remote in place, it is time to perform the actual backup. The rclone tool and the setup we just concluded is ideal for a headless machine environment that does not have a graphical user interface.

Rclone combines to greater effect with a backup script run on time by a crontab job. For our current example, we shall concentrate on setup, configuration and the various backup/ synchronizations commands available to us.

Step 2 : Backup files using rclone

The second step is to back up our files using the configured remote. Rclone provides a couple of option to copy a file between a source to a remote destination.

The rclone command provides a full list of available commands and a brief description of what they do. Let us run the rclone command in the terminal/command line

For our current example, we will explore the following commands.

	config          Enter an interactive configuration session.
  copy            Copy files from source to dest, skipping already copied.
  listremotes     List all the remotes in the config file.
  sync            Make source and dest identical, modifying destination only.
  tree            List the contents of the remote in a tree like fashion.

List remotes command as the name suggests it provides a list of all configured remotes in the configuration file. Appending the -l flag list the type of remote. The following options are available options of the “listremotes” command.

Flags:
  -h, --help   help for listremotes
      --long   Show the type as well as names.

eg rclone listremotes --long 

Copying files using rclone

To copy files to the remote we use the “rclone copy” command. This command copy files from source to dest, skipping already copied. The command does not transfer unchanged files, testing by size and modification time or MD5SUM. Doesn’t delete files from the destination.

Rclone syncs the contents of the directory, not the directory. Only the content of the directory copied to the destination, not the directory name and contents.

# rclone copy path-to-source-folder remote-name:remote-path

# Let us asume that there 3 files in our source directory

# path-to-source-folder/file-one.pdf
# path-to-source-folder/file-two.pdf
# path-to-source-folder/file-three.pdf

# Only the files in the source folder will be copied to the remote

# path-to-destination-folder/file-one.pdf
# path-to-destination-folder/file-two.pdf
# path-to-destination-folder/file-three.pdf

# per our configuration we run this code to copy our files to the remote

rclone copy --progress path-to-source-folder gdrive:site_performance_result

# the --progress flag as the name states gives you the progression of the file
# copy to remote

The rclone sync command makes source and destination identical, modifying destination only. The command does not transfer unchanged files, testing by size and modification time or MD5SUM. Rclone updates the destination file to match the source, including deleting files if necessary. The source folder or files will always be the single source of truth with regards to the “sync” hence any changes on the remote that does not exist on the source location will be deleted from the destination.

# similar stituation to the copy command applies to the sync command with 
# regards to the content of source folder synced rather than the folder 
# and files 

#example command
rclone sync SOURCE remote:DESTINATION

# per our configuration we run this code to copy our files to the remote

rclone sync  --progress path-to-source-folder gdrive:site_performance_result

# the --progress flag as the name states gives you the progression of the file
# copy to remote

Conclusion

Rclone is a powerful command-line tool that has many use cases. We only concentrated on a fraction of the use case of the Swiss army knife which is rclone. I encourage you to head over to the rlcone documentation page for an in-depth explanation of their enormous command list. You can improve your back up script and usage of rclone by reading more and playing with the various commands to get the perfect one for your use-case.

For Linux users, Dave Mckay has a great step-by-step walkthrough for setting up and using rclone to backup files on Linux.

Backup files with rclone Resources

For more reading on rclone here is a list of resources I find very useful

https://gist.github.com/allanphilipbarku/03027025ccbcb7d75548f1589dd8597e