hoarding.me

rclone – Setting up your remotes and encryption.

rclone is the workhorse behind your server. This is what will automatically encrypt and upload your media to your online storage.

Here’s how to install it;

Fetch and unpack

curl -O https://downloads.rclone.org/rclone-current-linux-amd64.zip
unzip rclone-current-linux-amd64.zip
cd rclone-*-linux-amd64

Copy binary file

sudo cp rclone /usr/bin/
sudo chown root:root /usr/bin/rclone
sudo chmod 755 /usr/bin/rclone

Install manpage


sudo mkdir -p /usr/local/share/man/man1
sudo cp rclone.1 /usr/local/share/man/man1/
sudo mandb

We’re going to encrypt everything before we upload it so this adds another layer to the process. How this works is you create remotes to your cloud storage, then we create an encrypted remote on top of the normal remote. These encrypted remotes, one for TV and one for movies are the ones we’ll be using for uploading. We’ll then be creating two more remotes afterwards to decrypt the plexdrive mounts. So 5 in total.

To run it, rclone config. Select N for a new remote, and just name it ‘gd’ then select 7 for GD. This is the underlying remote we’ll use for our crypts. Follow this link to create a client ID and secret, and use them for the next two prompts in the rclone config. After this, select N, and then copy the link provided and use it in your browser. Verify your google account and paste the code returned, then Y for ‘yes this is ok’ and you have your first remote!

Next we’re going to setup two encrypted remotes. Login to GD and create two folders, tv-gd and m-gd.

Run rclone config again, N for new remote, then set the name as tv-gd, and 5 for a crypt. Next enter gd:/tv-gd, and 2 for standard filenames. Create or generate password and an optional salt, make sure you keep these somewhere safe, as they’re required to access the decrypted data. Select Y for ‘yes this is ok’. Then you can do the same for the second one, using the name m-gd, and the remote gd:/m-gd. There’s our two encrypted remotes setup. Note their names and the folders they’re linked to, IE tv-gd is linked to gd:/tv-gd etc.

Let’s also setup both of our remotes for decrypting your plexdrive. It’s also worth clarifying that instead of mounting and decrypting your files straight from google drive, we’re instead going to be mounting it like this.

google drive -> plexdrive -> rclone -> plex. So plexdrive mounts your entire drive, including your encrypted folders. You then create two new rclone mounts that will read these ‘local’ encrypted folders and re-mount and decrypt them to the folder plex reads.

Run rclone config again, N for new remote, name it dec-tv, select 5 for a crypt, for path enter /mnt/plexdrive/tv-gd, and 2 for standard filenames. Now for the password and salt, select Y to use existing, and use the same password and salt you used for the ‘tv-gd’ crypt we created above. Then select Y to create the remote. Repeat for the movies, using the name dec-m, path /mnt/plexdrive/m-gd, and use the password and salt from the ‘m-gd’ crypt.

Plexdrive

Now that the remotes are setup, let’s take a few minutes to go over Plexdrive. Plexdrive is a necessity for this setup, mainly because Google will temporary ban you for 24hours if you exceed api usage. Plexdrive caches the locations which makes this whole setup possible. First download the newest version from this page, wget paste-link-here


mv plexdrive-linux-amd64 /usr/local/bin/plexdrive
chown root:root /usr/local/bin/plexdrive
chmod 755 /usr/local/bin/plexdrive

To run it once, /usr/local/bin/plexdrive mount /mnt/plexdrive/. This will prompt you to enter more credentials, you can use the same client id and secret as before, and it will get started on building your first cache. What this does is build a local cache of the directories, so API calls to google are cut down and you don’t get API banned.

From here, I cancelled the automatic cache build (ctrl+c) and instead used this script, ensure the filepaths are consistent with what you’re using.

As of plexdrive_5.0 you no longer need to run this rebuildcache script, so feel free to skip it over.


#!/bin/bash
## plexdrive-rebuildcache.sh (chmod a+x plexdrive-rebuildcache.sh)
## The function of this script is rebuild cache in memory and then move it to default location.
## NOTE: This will speed up process significantly as building full cache on hard disk may take hours.

TIMESTAMP=`date +%Y-%m-%d_%H-%M-%S`
LOGFILE="/home/scripts/logs/plexdrive-rebuildcache.log"
PLEXDRIVEEXEC=/usr/local/bin/plexdrive # Set path to plexdrive executable
PLEXDRIVEMNT=/mnt/plexdrive # Set path to plexdrive mount
PLEXDRIVECONF=~/.plexdrive # Set location of your plexdrive config or leave default
PLEXDRIVEMOUNTCMD="/home/scripts/mountplexdrive" # Set path to your plexdrive mount script. If you use systemd replace it with "sudo systemctl plexdrive.service start"

# Specify list of commands to stop apps & unmount drives before proceeding with cache rebuild
## example: sudo service plexmediaserver stop
STOPCMDS=(
     'fusermount -uz /mnt/plexdrive'
)
# Specify list of commands to mount and start apps once cache is rebuilt
## If you mount plexdrive with systemd replace it with: sudo systemctl plexdrive.service star
## example: sudo service plexmediaserver start
STARTCMDS=(
     'echo "/home/plex/scripts/mountplexdrive"'
)


if [[ ! -f $PLEXDRIVECONF/config.json ]]; then
    echo "EXIT: plexdrive config missing at path $PLEXDRIVECONF"
    echo "NOTE: plexdrive must be configured, type: plexdrive /mnt/plexdrive and follow setup"
    exit
fi
echo "///////////////////////////////////////////////////////////////////////////////////////////////"
echo "Your current cache will be deleted, rebuilt in memory (/dev/shm) and moved to default location"
echo "plexdrive conf path = $PLEXDRIVECONF"
echo "plexdrive exec path = $PLEXDRIVEEXEC"
echo "plexdrive mount command: $PLEXDRIVEMOUNTCMD"
echo "NOTE: You can run rebuilt automatically: /path/plexdrive-rebuildcache.sh y"
echo "///////////////////////////////////////////////////////////////////////////////////////////////"
response=$1
if [[ $response != y ]]; then
    read -r -p "Are you sure? [y/N] " response
fi
case "$response" in
    [yY][eE][sS]|[yY])
        echo "$(date "+%d.%m.%Y %T") plexdrive rebuild cache started" | tee -a $LOGFILE
        echo "Unmounting drive and stopping apps" | tee -a $LOGFILE
        for stopcmd in "${STOPCMDS[@]}"
        do
            $stopcmd | tee -a $LOGFILE
        done
        echo "Copying plex drive config to /dev/shm" | tee -a $LOGFILE
        cp $PLEXDRIVECONF/config.json /dev/shm/ | tee -a $LOGFILE
        cp $PLEXDRIVECONF/token.json /dev/shm/ | tee -a $LOGFILE
        if [[ ! -f /dev/shm/cache ]]; then
            echo "Deleting existing cache file" | tee -a $LOGFILE
            rm /dev/shm/cache* | tee -a $LOGFILE
        fi
        echo "Mounting plexdrive and waiting for cache to be rebuilt" | tee -a $LOGFILE
        cachebuildstart=$(date +'%s')
        $PLEXDRIVEEXEC $PLEXDRIVEMNT --config=/dev/shm -v 2 &>>/dev/shm/plexdrive-$TIMESTAMP.log &
        while > /dev/null
        do
          finished=$(grep "First cache build process finished!" /dev/shm/plexdrive-$TIMESTAMP.log)
          if [[ ! -z $finished ]]; then
            echo "First cache build process finished in $(($(date +'%s') - $cachebuildstart)) seconds" | tee -a $LOGFILE
            echo "$(grep "Processed" /dev/shm/plexdrive-$TIMESTAMP.log | tail -1)" | tee -a $LOGFILE
            break
          fi
            echo "$(grep -i "${PLEXDRIVEEXEC:1}" /dev/shm/plexdrive-$TIMESTAMP.log | tail -1)"
            sleep 1
        done
        echo "Unmounting plexdrive" | tee -a $LOGFILE
        fusermount -uz $PLEXDRIVEMNT | tee -a $LOGFILE
        echo "/dev/shm cleanup" | tee -a $LOGFILE
        rm /dev/shm/config.json | tee -a $LOGFILE
        rm /dev/shm/token.json  | tee -a $LOGFILE
        rm /dev/shm/plexdrive-$TIMESTAMP.log | tee -a $LOGFILE
        echo "Moving plexdrive cache back to $PLEXDRIVECONF" | tee -a $LOGFILE
        mv /dev/shm/cache* $PLEXDRIVECONF | tee -a $LOGFILE

        echo "Mounting plexdrive and starting all services" | tee -a $LOGFILE
        for startmd in "${STARTCMDS[@]}"
        do
            $startcmd | tee -a $LOGFILE
        done
        ;;
    *)
        echo "Aborted"
        exit
        ;;
esac
        echo "$(date "+%d.%m.%Y %T") plexdrive rebuild cache finished" | tee -a $LOGFILE
exit

Save it in /home/scripts/ (chmod a+x also) and you can setup a crontab to rebuild the caches every 12 hours or so. crontab -e and enter 0 */12 * * * /home/scripts/rebuildcache.sh >/dev/null 2>&1

fusermount -uz /mnt/plexdrive. Next lets setup a better script to remount plexdrive. In the scripts folder, again create mount-plex.sh


#!/bin/bash
## plexdrive mount (chmod a+x mountplexdrive.sh)
## To mount the drive at reboot & to remount in case of failure type crontab -e and add 2 lines below (without ##):
## @reboot /path/mount-plex.sh
## 0 5 * * *   /path/mount-plex.sh >/dev/null 2>&1

## GLOBAL VARS
LOGFILE="/home/scripts/logs/mount-plex.log"
MPOINT="/mnt/plexdrive/"

## UNMOUNT IF SCRIPT WAS RUN WITH unmount PARAMETER
if [[ $1 = "unmount" ]]; then
    echo "Unmounting $MPOINT"
    fusermount -uz $MPOINT
    exit
fi

## CHECK IF MOUNT ALREADY EXIST AND MOUNT IF NOT
if mountpoint -q $MPOINT ; then
    echo "$MPOINT already mounted"
else
    echo "Mounting $MPOINT"
    /usr/local/bin/plexdrive mount $MPOINT \
                       -o allow_other \
                       -v 2 &>>$LOGFILE &
fi
exit

## Default is with minimal options and if needed use aditional flags copy paste them above line: -v 2 &>>$LOGFILE &
##        Note: Only lines eg options with - or -- in front
##  --chunk-size 5M \
##        The size of each chunk that is downloaded (units: B, K, M, G) (default "5M")
##  --clear-chunk-age 30m0s \
##        The maximum age of a cached chunk file (default 30m0s)
##  --clear-chunk-interval 1m0s \
##        The time to wait till clearing the chunk directory (default 1m0s)
##  --clear-chunk-max-size 100G \
##        The maximum size of the temporary chunk directory (units: B, K, M, G)
##  -c, --config=/home/plex/.plexdrive \
##        The path to the configuration directory (default "/home/plex/.plexdrive")
##  -o allow_other \
##        Fuse mount options (e.g. -fuse-options allow_other,...)
##  --gid 1000 \
##        Set the mounts GID (-1 = default permissions) (default -1)
##  --refresh-interval 5m0s \
##        The time to wait till checking for changes (default 5m0s)
##  --speed-limit 1G \
##        This value limits the download speed, e.g. 5M = 5MB/s per chunk (units: B, K, M, G)
##  -t, --temp=/tmp \
##        Path to a temporary directory to store temporary data (default "/tmp")
##  --uid 1000 \
##        Set the mounts UID (-1 = default permissions) (default -1)
##  --umask value
##        Override the default file permissions
##  -v, --verbosity 2 \
##        Set the log level (0 = error, 1 = warn, 2 = info, 3 = debug, 4 = trace)

Chmod a+x this and the rebuildcaches above if you haven’t already, then mount it again via /home/scripts/mountplex.sh. If it prompts you to setup the google API, just follow the steps. You can use the same client id and secret as before. This will be then saved in ~/.plexdrive/config.json and it’s what plexdrive uses to validate.

I want to also point out the additional options I’ve put in.

-t /home/plex/temp

-t dictates where we set the home for the chunks, by default it goes to root and this can rapidly fill up your partition so ensure it’s somewhere with plenty of space.

Next up, [rclone, scripting and automation with crontab.]