backupagentnq - cloud backup enqueuer agent

Overview

backupagentnq is the cloud backup enqueuer agent. It is responsible for identifying the files to backup and enqueues their pack instruction files into the Spool/Work directory. Its companion backupagentdq is responsible for dequeuing and submitting the pack instruction files to the cncop network.

Pack instruction files

Pack instruction files are the fundamental unit of backups. Each backup file will be converted to a series of instructions, which the cncop cloud network will follow to create a cloud version of the file.

Pack instruction files come in two variations; pack-add and pack-update. A pack-add instruction file provides a complete copy of the source file. A pack-update instruction file provides directions for modifying a previous backup file to match the changes stored on a prime.

Operational phases

The backupagentnq agent operation has three distinct phases

1. Peek - The peek phase is where the agent identifies those files that have either changed, been added, or been deleted from the local device and need backup to the cloud. Each planned action is recorded in the work queues database, e.g., Spool/Work/work.db.

2. Pack - The pack phase is where the agent generates pack instruction files using the cncop.packgen2 system and enqueues them into the Spool/Work directory.

3. Prune - The prune phrase is where the agent deletes files from the cloud that have been deleted from the local device.

Move promotions

From time to time, a user will reorganize their files and folders. An example is at the end of a reporting quarter or the end of a year when they might rename an old document tree to include the last year, e.g., Proposals/ to 20NN-Proposal/. The agent needs to identify this type of folder maintenance; otherwise, it would interpret the action as a mass of DELETES and ADDS. And the resulting batch of ADDS could cause a lengthy backlog that could take days or weeks to re-upload to the cloud.

To avoid this, the peek phase searches for move promotions. This involves scanning the ADD and DELETE work queues for potential matches and removing and replacing these two operations with a MOVE and an UPDATE operation if a match is detected.

The full details of move promotion and what attributes it considers are described and protected by US Patent # 7,668,880

Skip backup when nothing has changed

To conserve Internet bandwidth, the backupagentnq will skip performing a cloud peek when it determines nothing has changed on the local device. Instead, it employs a last_write check system that allows it to know for each job when the newest file was last written, and if nothing more recent has been added or changed, the backupagentnq will end without peeking the cloud.

Use the --forcepeek command line option to override this check and force a complete peek of the local device and the cloud.

Protections against unintended deletes

The backupagentnq will limit the number of deletes it will perform in any single backup run to a delete collar specified in the job (default 1,500). This is to protect against accidental mass deletes. Use the --nomaxdelete command line options to suppress the delete collar and perform all scheduled deletes.

Singleton operation

Two singleton protections are embedded in the backupagentnq system; a joblock and an applock.

The joblock protection prevents multiple running copies of the same backup job. Any attempt to run a second copy of the job will cause the agent to wait for the first instance to finish and release its joblock.

The applock will protect against running more than one instance of the backupagentnq software. The reason is to prevent excessive write-lock contention for the work database and to limit the number of writers to the default logging file (ex: Logs/NQ-YYYY-MM-DD.log) to a single writer.

When there is a single long-running backupagentnq agent, the use of the --noapplock option will disable the applock protection and allow another copy of the software to run. When invoked with this option, the log file prefix will change to NQU (the extra U is applied to indicate it was invoked unlocked).

The --noapplock does not eliminate the joblock singleton pattern, which is to say, you cannot use this option to run the same job more than once.

Command line options

usage: backupagentnq [-?] [-d] [--debug2] [--debug3] [-c CONFIG] [-n] [--nomaxdelete] [--noapplock] [--skipmoves] [--skipdeletes] [--skippeeksource] [--skippeekcloud] [--usefork] [--nofork] [--usevss] [--forcepeek] [--nicld] [--suppressmissing] [--chunksize CHUNKSIZE] [--savepeek SAVEPEEK] [--peeklog] [--nolinks] [job [job ...]]

Positional arguments

job

Jobid or jobtitle of job to run.

Optional argument:

?, -h, --help

Display help and exit

-d, --debug

Generate diagnostic logging. The output is directed to the Spool/Logs folder and can be viewed using the cncop watcher script.

--debug2

Advanced diagnostic logging. Will emit verbose details of pack file generation. This option is intended for developers and not recommended for production.

--debug3

Extreme diagnostic logging. Not recommended for production use.

-c CONFIG, --config CONFIG

Alternate configuration file.

-n, --dryrun

Run agent in reporting mode. Does not perform a backup, but instead reports on what it would do. Note: use of this option will invalidate our source peek information, and cause the next invocation of this agent to perform a full cloud peek.

--nomaxdelete

Removes the maximum delete collar and allows all cloud deletes to take place.

--noapplock

Disable application locks. The default is to prevent more than one copy of backupagentnq from running at a time. If a second copy tries to run, it will wait until the first one finishes before running. When the agent is invoked with --noapplock, the application lock is disabled and another backupagentnq can run. If invoked with --noapplock, the agent will write it’s log information to Logs\NQU-YYYY-MM-DD.log

--skipmoves

Instruct the peek phase to skip checking for MOVED files. This option is only useful when there are a large number of ADD’s and DELETE’s, and scanning these two work queues for MOVE matches could consume excessive time and memory.

--skipdeletes

Do not delete files from the cloud. This is useful for the 1st sync with the cloud after it’s been seeded from a shuttled hard drive.

--skippeeksource

Do not peek the source filesystem. This option is primarily for diagnostics and should not be used in production.

--skippeekcloud

Do not peek the cloud. This option will reuse the most recently downloaded cloud peek. It should only be used for diagnostics.

--usefork

Override fork detection logic and force peek to use filesystem forks. This option is for development use and not recommended for production.

--nofork

Override fork detection logic and disable the use of filesystem forks. This option is for development use and not recommended for production.

--usevss

Backup using VSS. This option will cause the agent to make a new VSS snapshot and backup the specified path using the snapshot.

--forcepeek

Override the job’s newness detection system and force the agent to perform a cloud peek.

--nicld

Suppress logging of PEEK-NICLD messages. If include filters are used, this agent will log all files that do not match the filter, which, if the source path is large, can cause excessive log messages. This message disables this logging.

--suppressmissing

Suppress warnings of missing sourcefiles. If the filesystem is undergoing rapid changes while the backup job is running, some files may disappear after they’ve been scheduled for backup. This option suppresses generating job warning messages about any missing sourcefiles.

--chunksize CHUNKSIZE

The cncopagent process calls the cloud enqueuer and dequeuer agents with the same command line parameters. The enqueuer agent ignores this parameter.

--savepeek SAVEPEEK

Save the peek contents received from the cloud a local xml file. This option is for debugging cloud responses and likely only useful for s/w developers.

--peeklog

Similar to --savepeek, this will cause the peek data to be preserved, but rather than specifying the file’s name, the system will select a unique basename and store this file in the cncop log directory with the prefix NQPK. This feature enables debugging infrequent peek errors that reoccur with unpredictable frequency.

--nolinks

Do no backup file system links. Use of this option will cause the agent to skip backing up symlinks and soft links. Without this option, the agent will “follow” the link and backup what the link points to.