Jobs

A given cback workload consists of multiple jobs that nominally work together to provide a fully lifecycled backup. It is important to understand that even so, job definitions are independent from each other, and therefore can be modified, ran and even deleted outside of the context of the other jobs in the same workload.

All cback jobs consist of a set of fields that are stored in the job database in a specific, per job type table. There is some variance between the fields available depending on the job, but the three most important are:

  • id - A unique (per job class) integer used to target said job in invocation
  • Group - A group name string defining what cback group the job belongs to
  • Repository - An S3 URL that points to the restic repository the job belongs to and will work on

Job types

There are four main job types in cback, backup, restore, verify and prune, each corresponding to an equivalent underlying restic repository operation. Their relation to each other is described by the bellow table:

Job Type Backup Restore Prune Verify
Restic equivalent command Restic backup restic restore restic prune restic restore
cback job function Create new restic snapshots restore from a specific restic snapshot archive old restic snapshots verify snapshot integrity
Picking Agent/s Backup, Switch Restore Prune, Switch Verify, Switch
Schedulable? Yes (Switch) No Yes (Switch) Yes (Switch)
Created as part of a workload? Yes No, restore jobs are nominally ephemeral Yes Yes
Portal manageable? Yes Yes Partially (creation) Partially (creation)

Bellow is an example of each of the major job types in cback from a common workload, showing some of the variation found within the fields of their definitions. If you wish to see all fields of a given job, you can use cback <job-agent-sub-command> status --long <job-id>

[17:38][root@devstack-cback-71abd95b7d (ceph_devstack:cback/dev) cback]# cback backup status 1
╒══════╤═════════╤═══════════════════════════════════════════════╤═════════════════════════════════════════════════════════════════════════════════════════════════════════════╤════════════╤═══════════════════════════════════════════════════════════════════════════════════════════╤══════════╤═════════════════════╤═════════════════════╤═══════════════╕
│   Id │ Group   │ Name                                          │ Source                                                                                                      │ Instance   │ Repository                                                                                │ Status   │ Started             │ Finished            │ Info          │
╞══════╪═════════╪═══════════════════════════════════════════════╪═════════════════════════════════════════════════════════════════════════════════════════════════════════════╪════════════╪═══════════════════════════════════════════════════════════════════════════════════════════╪══════════╪═════════════════════╪═════════════════════╪═══════════════╡
│    1 │ manila  │ devstack-70e1c17f-a565-4f48-9d26-c2c550c4f12f │ /cephfs-devstack/volumes/_nogroup/8dabc902-5d08-475f-8738-adef46510c60/f7139a00-1be9-4314-a6f3-7b95cde1070e │            │ s3:https://s3-fr-prevessin-1.cern.ch/manila-devstack-70e1c17f-a565-4f48-9d26-c2c550c4f12f │ pending  │ 2024-05-07 18:05:17 │ 2024-05-07 18:05:19 │ Job completed │
╘══════╧═════════╧═══════════════════════════════════════════════╧═════════════════════════════════════════════════════════════════════════════════════════════════════════════╧════════════╧═══════════════════════════════════════════════════════════════════════════════════════════╧══════════╧═════════════════════╧═════════════════════╧═══════════════╛

[17:38][root@devstack-cback-71abd95b7d (ceph_devstack:cback/dev) cback]# cback prune status 1
╒══════╤═════════╤═══════════════════════════════════════════════╤═════════════════════════════════════════════════════════════════════════════════════════════════════════════╤════════════╤══════════╤═════════════════════╤═════════════════════╤═══════════════╕
│   Id │ Group   │ Name                                          │ Source Backup                                                                                               │ Instance   │ Status   │ Started             │ Finished            │ Info          │
╞══════╪═════════╪═══════════════════════════════════════════════╪═════════════════════════════════════════════════════════════════════════════════════════════════════════════╪════════════╪══════════╪═════════════════════╪═════════════════════╪═══════════════╡
│    1 │ manila  │ devstack-70e1c17f-a565-4f48-9d26-c2c550c4f12f │ /cephfs-devstack/volumes/_nogroup/8dabc902-5d08-475f-8738-adef46510c60/f7139a00-1be9-4314-a6f3-7b95cde1070e │            │ pending  │ 2024-05-07 17:26:25 │ 2024-05-07 17:26:28 │ Job completed │
╘══════╧═════════╧═══════════════════════════════════════════════╧═════════════════════════════════════════════════════════════════════════════════════════════════════════════╧════════════╧══════════╧═════════════════════╧═════════════════════╧═══════════════╛

[17:38][root@devstack-cback-71abd95b7d (ceph_devstack:cback/dev) cback]# cback verify status 1
╒══════╤═════════╤═══════════════════════════════════════════════╤═════════════════════════════════════════════════════════════════════════════════════════════════════════════╤════════════╤══════════╤═══════════╤════════════╤════════╕
│   Id │ Group   │ Name                                          │ Source Backup                                                                                               │ Instance   │ Status   │ Started   │ Finished   │ Info   │
╞══════╪═════════╪═══════════════════════════════════════════════╪═════════════════════════════════════════════════════════════════════════════════════════════════════════════╪════════════╪══════════╪═══════════╪════════════╪════════╡
│    1 │ manila  │ devstack-70e1c17f-a565-4f48-9d26-c2c550c4f12f │ /cephfs-devstack/volumes/_nogroup/8dabc902-5d08-475f-8738-adef46510c60/f7139a00-1be9-4314-a6f3-7b95cde1070e │            │ pending  │           │            │        │
╘══════╧═════════╧═══════════════════════════════════════════════╧═════════════════════════════════════════════════════════════════════════════════════════════════════════════╧════════════╧══════════╧═══════════╧════════════╧════════╛

[17:38][root@devstack-cback-71abd95b7d (ceph_devstack:cback/dev) cback]# cback restore status 1
╒══════════════╤═════════╤═══════════════════════════════════════════════╤═══════════╤══════════════════════════════════════════════════════════════════════════════════════════════════════════════╤═══════════╤═════════════════════╤═════════════════════╤═════════════════════╤═══════════════╕
│   Restore Id │ Group   │ Name                                          │ Pattern   │ Destination                                                                                                  │ Status    │ Created             │ Started             │ Finished            │ Info          │
╞══════════════╪═════════╪═══════════════════════════════════════════════╪═══════════╪══════════════════════════════════════════════════════════════════════════════════════════════════════════════╪═══════════╪═════════════════════╪═════════════════════╪═════════════════════╪═══════════════╡
│            1 │ manila  │ devstack-70e1c17f-a565-4f48-9d26-c2c550c4f12f │           │ /cephfs-devstack/volumes/_nogroup/8dabc902-5d08-475f-8738-adef46510c60/f7139a00-1be9-4314-a6f3-7b95cde1070e/ │ completed │ 2024-05-07 15:57:33 │ 2024-05-07 15:57:42 │ 2024-05-07 15:58:08 │ Job completed │
╘══════════════╧═════════╧═══════════════════════════════════════════════╧═══════════╧══════════════════════════════════════════════════════════════════════════════════════════════════════════════╧═══════════╧═════════════════════╧═════════════════════╧═════════════════════╧═══════════════╛

[note] cbak job IDs are NOT guaranteed to be identical across different job types within a workload, therefore when making modifications to a set of jobs, always verify the jobs are identical in regards to target s3 repository URL rather then with job-id/s.

Execution state

Status or the execution field of the job defines how agents will interact with the job in question while also providing some feedback regarding the progress and health of the job. It can have the following values:

  • Pending: Job is ready to be picked and ran by an available agent.
  • Running: The job has been picked and is being executed by an agent.
  • Failed: Something happened during the running of a job that caused a failure.
  • Completed: The job concluded successfully or was marked as completed artificially by an administrator. It will now lay dormant until switched again.
  • Rescheduled: Another job had a lock on the jobs restic repo during its run. Job will be rerun.
  • Archived: Backup job specific, the backup is disabled, and a life-cycle policy is applied to the repository that casues the prune agent to delete all snapshots but the latest.

Job Status is nominally set and updated as an action of the different cback agents, however there exist two mechanisms to 'reset' this state to allow the job to immediately be reran:

  • asynchronously: cback <job-agent-subcommand> reset <job-id> sets a job back to pending state
  • synchronously: cback <job-agent-subcommand> run <job-id> immediately attempts to run the job with an ephemeral agent on the worker node the command was issued.

Control state

Enabled or the control field of the job, can be either Enabled or Disabled and defines whether a job should be allowed to be picked up by a corresponding cback agent. For example even if a job is in pending Status, if its Enabled state is Disabled, the job will not be picked and run.

Jobs default to Enabled on creation but this can be set administratively via the cback-cli with cback <job-agent-sub-command> <disable/enable> <job-id> for example, bellow I disable a backup job with id 1 in the manner just described:

[17:01][root@devstack-cback-71abd95b7d (ceph_devstack:cback/dev) cback]# cback backup disable 1
╒══════╤═════════╤═══════════════════════════════════════════════╤═════════════════════════════════════════════════════════════════════════════════════════════════════════════╤════════════╤═══════════════════════════════════════════════════════════════════════════════════════════╤══════════╤═════════════════════╤═════════════════════╤═══════════════╕
│   Id │ Group   │ Name                                          │ Source                                                                                                      │ Instance   │ Repository                                                                                │ Status   │ Started             │ Finished            │ Info          │
╞══════╪═════════╪═══════════════════════════════════════════════╪═════════════════════════════════════════════════════════════════════════════════════════════════════════════╪════════════╪═══════════════════════════════════════════════════════════════════════════════════════════╪══════════╪═════════════════════╪═════════════════════╪═══════════════╡
│    1 │ manila  │ devstack-70e1c17f-a565-4f48-9d26-c2c550c4f12f │ /cephfs-devstack/volumes/_nogroup/8dabc902-5d08-475f-8738-adef46510c60/f7139a00-1be9-4314-a6f3-7b95cde1070e │            │ s3:https://s3-fr-prevessin-1.cern.ch/manila-devstack-70e1c17f-a565-4f48-9d26-c2c550c4f12f │ pending  │ 2024-05-07 18:05:17 │ 2024-05-07 18:05:19 │ Job completed │
╘══════╧═════════╧═══════════════════════════════════════════════╧═════════════════════════════════════════════════════════════════════════════════════════════════════════════╧════════════╧═══════════════════════════════════════════════════════════════════════════════════════════╧══════════╧═════════════════════╧═════════════════════╧═══════════════╛

[17:01][root@devstack-cback-71abd95b7d (ceph_devstack:cback/dev) cback]# cback backup status -l 1
╒══════╤═════════╤═══════════╤═══════════════════════════════════════════════╤════════════╤═════════════════════════════════════════════════════════════════════════════════════════════════════════════╤════════════╤═══════════════════════════════════════════════════════════════════════════════════════════╤══════════╤══════════════════════════════════════════╤═══════════════════════════════════════╤═══════════════╤════════════════════════╤═════════════════════╤═════════════════════╤═══════════════╕
│   Id │ Group   │   Enabled │ Name                                          │ Username   │ Source                                                                                                      │ Instance   │ Repository                                                                                │ Status   │ Worker                                   │ Stats                                 │ Pinned Node   │ Desidered start time   │ Started             │ Finished            │ Info          │
╞══════╪═════════╪═══════════╪═══════════════════════════════════════════════╪════════════╪═════════════════════════════════════════════════════════════════════════════════════════════════════════════╪════════════╪═══════════════════════════════════════════════════════════════════════════════════════════╪══════════╪══════════════════════════════════════════╪═══════════════════════════════════════╪═══════════════╪════════════════════════╪═════════════════════╪═════════════════════╪═══════════════╡
│    1 │ manila  │         0 │ devstack-70e1c17f-a565-4f48-9d26-c2c550c4f12f │ root       │ /cephfs-devstack/volumes/_nogroup/8dabc902-5d08-475f-8738-adef46510c60/f7139a00-1be9-4314-a6f3-7b95cde1070e │            │ s3:https://s3-fr-prevessin-1.cern.ch/manila-devstack-70e1c17f-a565-4f48-9d26-c2c550c4f12f │ pending  │ devstack-cback-71abd95b7d.cern.ch:771736 │ N[0/0] C[0/0] U[2/6] - Added: 0 bytes │               │                        │ 2024-05-07 18:05:17 │ 2024-05-07 18:05:19 │ Job completed │
╘══════╧═════════╧═══════════╧═══════════════════════════════════════════════╧════════════╧═════════════════════════════════════════════════════════════════════════════════════════════════════════════╧════════════╧═══════════════════════════════════════════════════════════════════════════════════════════╧══════════╧══════════════════════════════════════════╧═══════════════════════════════════════╧═══════════════╧════════════════════════╧═════════════════════╧═════════════════════╧═══════════════╛

Creating a workload

To create a complete workload in cback, nominally you will use the cback backup add command. It is expected, that this will create a backup, prune and verify job for the target filepath concerned.

[note] You may notice that cback also has cback <prune|verify|restore> add commands, these can be used to add individual jobs, likewise, when using cback backup add if you wish to NOT create a workload (only a backup job) you can use the --no-workload flag.

Bellow is an example of adding a workload into cback:

[13:34][root@devstack-cback-71abd95b7d (ceph_devstack:cback/dev) ~]# cback backup add example /example-path -g manila
╒══════╤═════════╤═════════╤═══════════════╤════════════╤═════════════════════════════════════════════════════╤══════════╤═══════════╤════════════╤════════╕
│   Id │ Group   │ Name    │ Source        │ Instance   │ Repository                                          │ Status   │ Started   │ Finished   │ Info   │
╞══════╪═════════╪═════════╪═══════════════╪════════════╪═════════════════════════════════════════════════════╪══════════╪═══════════╪════════════╪════════╡
│    2 │ manila  │ example │ /example-path │ default    │ s3:https://s3-fr-prevessin-1.cern.ch/manila-example │ pending  │           │            │        │
╘══════╧═════════╧═════════╧═══════════════╧════════════╧═════════════════════════════════════════════════════╧══════════╧═══════════╧════════════╧════════╛

[13:34][root@devstack-cback-71abd95b7d (ceph_devstack:cback/dev) ~]# cback prune ls
╒══════╤═════════╤═══════════════════════════════════════════════╤═════════════════════════════════════════════════════════════════════════════════════════════════════════════╤════════════╤══════════╤═════════════════════╤═════════════════════╤═══════════════╕
│   Id │ Group   │ Name                                          │ Source Backup                                                                                               │ Instance   │ Status   │ Started             │ Finished            │ Info          │
╞══════╪═════════╪═══════════════════════════════════════════════╪═════════════════════════════════════════════════════════════════════════════════════════════════════════════╪════════════╪══════════╪═════════════════════╪═════════════════════╪═══════════════╡
│    2 │ manila  │ example                                       │ /example-path                                                                                               │ default    │ pending  │                     │                     │               │
╘══════╧═════════╧═══════════════════════════════════════════════╧═════════════════════════════════════════════════════════════════════════════════════════════════════════════╧════════════╧══════════╧═════════════════════╧═════════════════════╧═══════════════╛

[13:34][root@devstack-cback-71abd95b7d (ceph_devstack:cback/dev) ~]# cback verify ls
╒══════╤═════════╤═══════════════════════════════════════════════╤═════════════════════════════════════════════════════════════════════════════════════════════════════════════╤════════════╤══════════╤═══════════╤════════════╤════════╕
│   Id │ Group   │ Name                                          │ Source Backup                                                                                               │ Instance   │ Status   │ Started   │ Finished   │ Info   │
╞══════╪═════════╪═══════════════════════════════════════════════╪═════════════════════════════════════════════════════════════════════════════════════════════════════════════╪════════════╪══════════╪═══════════╪════════════╪════════╡
│    2 │ manila  │ example                                       │ /example-path                                                                                               │ default    │ pending  │           │            │        │
╘══════╧═════════╧═══════════════════════════════════════════════╧═════════════════════════════════════════════════════════════════════════════════════════════════════════════╧════════════╧══════════╧═══════════╧════════════╧════════╛

[warning❗] the mechanism for ensuring the filepath you wish to target in your workload is accessible on the cback systems worker nodes, is left to the reader. As an example, at CERN, we ensure all worker nodes locally mount target eos and ceph clusters file systems.