Jobs

A given cback workload consists of multiple jobs that nominally work together to provide a fully lifecycled backup. Its important to understand that even so, job definitions are independent from eachother, and therefore can be modfied, ran and even deleted outside of the context of the other jobs in the same workload.

all cback jobs consist of a set of fields that are stored in the job database in a specific, per job type table, There is some variance between the fields availible depending on the job, but the three most important are:

id - A unique (per job class) integer used to target said job in invocation
Group - A group name string defining what cback group the job belongs to
Repository - A S3 URL that points to the restic repository the job belongs to and will work on

Job types

There are four main job types in cback, backup, restore, verify and prune, each corrosponding to a equivilent underlying restic repository operation. Their relation to each other is described by the bellow table:

Job Type	Backup	Restore	Prune	Verify
Restic equivilent command	Restic backup	restic restore	restic prune	restic restore
cback job function	Create new restic snapshots	restore from a specific restic snapshot	archive old restic snapshots	verify snapshot integrity
Picking Agent/s	Backup, Switch	Restore	Prune, Switch	Verify, Switch
Scheduable?	Yes (switch)	No	Yes (Switch)	Yes (switch)
Created as part of a workload?	Yes	No, restore jobs are nominally ephemeral	Yes	Yes
Portal managable?	Yes	Yes	Partially (creation)	Partially (creation)

Bellow is an example of each of the major job types in cback from a common workload, showing some of the variation found within the fields of their definitions. if you wish to see all fields of a given job, you can use cback <job-agent-sub-command> status --long <job-id>

[17:38][root@devstack-cback-71abd95b7d (ceph_devstack:cback/dev) cback]# cback backup status 1
╒══════╤═════════╤═══════════════════════════════════════════════╤═════════════════════════════════════════════════════════════════════════════════════════════════════════════╤════════════╤═══════════════════════════════════════════════════════════════════════════════════════════╤══════════╤═════════════════════╤═════════════════════╤═══════════════╕
│   Id │ Group   │ Name                                          │ Source                                                                                                      │ Instance   │ Repository                                                                                │ Status   │ Started             │ Finished            │ Info          │
╞══════╪═════════╪═══════════════════════════════════════════════╪═════════════════════════════════════════════════════════════════════════════════════════════════════════════╪════════════╪═══════════════════════════════════════════════════════════════════════════════════════════╪══════════╪═════════════════════╪═════════════════════╪═══════════════╡
│    1 │ manila  │ devstack-70e1c17f-a565-4f48-9d26-c2c550c4f12f │ /cephfs-devstack/volumes/_nogroup/8dabc902-5d08-475f-8738-adef46510c60/f7139a00-1be9-4314-a6f3-7b95cde1070e │            │ s3:https://s3-fr-prevessin-1.cern.ch/manila-devstack-70e1c17f-a565-4f48-9d26-c2c550c4f12f │ pending  │ 2024-05-07 18:05:17 │ 2024-05-07 18:05:19 │ Job completed │
╘══════╧═════════╧═══════════════════════════════════════════════╧═════════════════════════════════════════════════════════════════════════════════════════════════════════════╧════════════╧═══════════════════════════════════════════════════════════════════════════════════════════╧══════════╧═════════════════════╧═════════════════════╧═══════════════╛

[17:38][root@devstack-cback-71abd95b7d (ceph_devstack:cback/dev) cback]# cback prune status 1
╒══════╤═════════╤═══════════════════════════════════════════════╤═════════════════════════════════════════════════════════════════════════════════════════════════════════════╤════════════╤══════════╤═════════════════════╤═════════════════════╤═══════════════╕
│   Id │ Group   │ Name                                          │ Source Backup                                                                                               │ Instance   │ Status   │ Started             │ Finished            │ Info          │
╞══════╪═════════╪═══════════════════════════════════════════════╪═════════════════════════════════════════════════════════════════════════════════════════════════════════════╪════════════╪══════════╪═════════════════════╪═════════════════════╪═══════════════╡
│    1 │ manila  │ devstack-70e1c17f-a565-4f48-9d26-c2c550c4f12f │ /cephfs-devstack/volumes/_nogroup/8dabc902-5d08-475f-8738-adef46510c60/f7139a00-1be9-4314-a6f3-7b95cde1070e │            │ pending  │ 2024-05-07 17:26:25 │ 2024-05-07 17:26:28 │ Job completed │
╘══════╧═════════╧═══════════════════════════════════════════════╧═════════════════════════════════════════════════════════════════════════════════════════════════════════════╧════════════╧══════════╧═════════════════════╧═════════════════════╧═══════════════╛

[17:38][root@devstack-cback-71abd95b7d (ceph_devstack:cback/dev) cback]# cback verify status 1
╒══════╤═════════╤═══════════════════════════════════════════════╤═════════════════════════════════════════════════════════════════════════════════════════════════════════════╤════════════╤══════════╤═══════════╤════════════╤════════╕
│   Id │ Group   │ Name                                          │ Source Backup                                                                                               │ Instance   │ Status   │ Started   │ Finished   │ Info   │
╞══════╪═════════╪═══════════════════════════════════════════════╪═════════════════════════════════════════════════════════════════════════════════════════════════════════════╪════════════╪══════════╪═══════════╪════════════╪════════╡
│    1 │ manila  │ devstack-70e1c17f-a565-4f48-9d26-c2c550c4f12f │ /cephfs-devstack/volumes/_nogroup/8dabc902-5d08-475f-8738-adef46510c60/f7139a00-1be9-4314-a6f3-7b95cde1070e │            │ pending  │           │            │        │
╘══════╧═════════╧═══════════════════════════════════════════════╧═════════════════════════════════════════════════════════════════════════════════════════════════════════════╧════════════╧══════════╧═══════════╧════════════╧════════╛

[17:38][root@devstack-cback-71abd95b7d (ceph_devstack:cback/dev) cback]# cback restore status 1
╒══════════════╤═════════╤═══════════════════════════════════════════════╤═══════════╤══════════════════════════════════════════════════════════════════════════════════════════════════════════════╤═══════════╤═════════════════════╤═════════════════════╤═════════════════════╤═══════════════╕
│   Restore Id │ Group   │ Name                                          │ Pattern   │ Destination                                                                                                  │ Status    │ Created             │ Started             │ Finished            │ Info          │
╞══════════════╪═════════╪═══════════════════════════════════════════════╪═══════════╪══════════════════════════════════════════════════════════════════════════════════════════════════════════════╪═══════════╪═════════════════════╪═════════════════════╪═════════════════════╪═══════════════╡
│            1 │ manila  │ devstack-70e1c17f-a565-4f48-9d26-c2c550c4f12f │           │ /cephfs-devstack/volumes/_nogroup/8dabc902-5d08-475f-8738-adef46510c60/f7139a00-1be9-4314-a6f3-7b95cde1070e/ │ completed │ 2024-05-07 15:57:33 │ 2024-05-07 15:57:42 │ 2024-05-07 15:58:08 │ Job completed │
╘══════════════╧═════════╧═══════════════════════════════════════════════╧═══════════╧══════════════════════════════════════════════════════════════════════════════════════════════════════════════╧═══════════╧═════════════════════╧═════════════════════╧═════════════════════╧═══════════════╛

[note] cbak job id's are NOT guarenteed to be identical across different job types within a workload, therefore when making modifications to a set of jobs, allways verify the jobs are identical in regards to target s3 repository URL rather then with job-id/s.

Execution state

Status or the execution field of the job defines how agents will interact with the job in question while also providing some feedback regarding the progress and health of the job. It can have the following values:

Pending: Job is ready to be picked and ran by an available agent.
Running: The job has been picked and is being executed by an agent.
Failed: Something happened during the running of a job that caused a failure.
Completed: The job concluded successfuly and will now lay dormant until switched again.
Rescheduled: Another job had a lock on the jobs restic repo during its run. Job will be rerun.
Archived: Backup job specific, the workload is disabled, the prune agent will read this state and prune all snapshots in the repository target except the last. prune job definition so only its most recent snapshot remains, no further pruning will occur.

Job Status is nominally set and updated as a action of the different cback agents, however there exists two mechanisms to 'reset' this state to allow the job to imediately be reran:

asynchronously: cback <job-agent-subcommand> reset <job-id> sets a job back to pending state
Synchronously: cback <job-agent-subcommand> run <job-id> imediately attempt to run the job with a ephemeral agent on the worker node the command was issued.

Control state

Enabled or the control field of the job, can be either Enabled or Disabled and defines whether a job should be allowed to be picked up by a corrosponding cback agent. For example even if a job is in pending Status, if its Enabled state is Disabled, the job will not be picked and run.

Jobs default to enabled on creation but this can be set administratively via the cback-cli with cback <job-agent-sub-command> <disable/enable> <job-id> for example, bellow I disable a backup job with id 1 in the manner just described:

[17:01][root@devstack-cback-71abd95b7d (ceph_devstack:cback/dev) cback]# cback backup disable 1
╒══════╤═════════╤═══════════════════════════════════════════════╤═════════════════════════════════════════════════════════════════════════════════════════════════════════════╤════════════╤═══════════════════════════════════════════════════════════════════════════════════════════╤══════════╤═════════════════════╤═════════════════════╤═══════════════╕
│   Id │ Group   │ Name                                          │ Source                                                                                                      │ Instance   │ Repository                                                                                │ Status   │ Started             │ Finished            │ Info          │
╞══════╪═════════╪═══════════════════════════════════════════════╪═════════════════════════════════════════════════════════════════════════════════════════════════════════════╪════════════╪═══════════════════════════════════════════════════════════════════════════════════════════╪══════════╪═════════════════════╪═════════════════════╪═══════════════╡
│    1 │ manila  │ devstack-70e1c17f-a565-4f48-9d26-c2c550c4f12f │ /cephfs-devstack/volumes/_nogroup/8dabc902-5d08-475f-8738-adef46510c60/f7139a00-1be9-4314-a6f3-7b95cde1070e │            │ s3:https://s3-fr-prevessin-1.cern.ch/manila-devstack-70e1c17f-a565-4f48-9d26-c2c550c4f12f │ pending  │ 2024-05-07 18:05:17 │ 2024-05-07 18:05:19 │ Job completed │
╘══════╧═════════╧═══════════════════════════════════════════════╧═════════════════════════════════════════════════════════════════════════════════════════════════════════════╧════════════╧═══════════════════════════════════════════════════════════════════════════════════════════╧══════════╧═════════════════════╧═════════════════════╧═══════════════╛

[17:01][root@devstack-cback-71abd95b7d (ceph_devstack:cback/dev) cback]# cback backup status -l 1
╒══════╤═════════╤═══════════╤═══════════════════════════════════════════════╤════════════╤═════════════════════════════════════════════════════════════════════════════════════════════════════════════╤════════════╤═══════════════════════════════════════════════════════════════════════════════════════════╤══════════╤══════════════════════════════════════════╤═══════════════════════════════════════╤═══════════════╤════════════════════════╤═════════════════════╤═════════════════════╤═══════════════╕
│   Id │ Group   │   Enabled │ Name                                          │ Username   │ Source                                                                                                      │ Instance   │ Repository                                                                                │ Status   │ Worker                                   │ Stats                                 │ Pinned Node   │ Desidered start time   │ Started             │ Finished            │ Info          │
╞══════╪═════════╪═══════════╪═══════════════════════════════════════════════╪════════════╪═════════════════════════════════════════════════════════════════════════════════════════════════════════════╪════════════╪═══════════════════════════════════════════════════════════════════════════════════════════╪══════════╪══════════════════════════════════════════╪═══════════════════════════════════════╪═══════════════╪════════════════════════╪═════════════════════╪═════════════════════╪═══════════════╡
│    1 │ manila  │         0 │ devstack-70e1c17f-a565-4f48-9d26-c2c550c4f12f │ root       │ /cephfs-devstack/volumes/_nogroup/8dabc902-5d08-475f-8738-adef46510c60/f7139a00-1be9-4314-a6f3-7b95cde1070e │            │ s3:https://s3-fr-prevessin-1.cern.ch/manila-devstack-70e1c17f-a565-4f48-9d26-c2c550c4f12f │ pending  │ devstack-cback-71abd95b7d.cern.ch:771736 │ N[0/0] C[0/0] U[2/6] - Added: 0 bytes │               │                        │ 2024-05-07 18:05:17 │ 2024-05-07 18:05:19 │ Job completed │
╘══════╧═════════╧═══════════╧═══════════════════════════════════════════════╧════════════╧═════════════════════════════════════════════════════════════════════════════════════════════════════════════╧════════════╧═══════════════════════════════════════════════════════════════════════════════════════════╧══════════╧══════════════════════════════════════════╧═══════════════════════════════════════╧═══════════════╧════════════════════════╧═════════════════════╧═════════════════════╧═══════════════╛

Creating a workload

To create a complete workload in cback, nominally you will use the cback backup add command, it is expected, that this will create a backup, prune and verify job for the target filepath concerned.

[note] you may notice that cback also has cback <prune|verify|restore> add commands, these can be used to add individual jobs, likewise, when using cback backup add if you wish to NOT create a workload (only a backup job) you can use the --no-workload flag.

Bellow is an example of adding a workload into cback:

[13:34][root@devstack-cback-71abd95b7d (ceph_devstack:cback/dev) ~]# cback backup add example /example-path -g manila
╒══════╤═════════╤═════════╤═══════════════╤════════════╤═════════════════════════════════════════════════════╤══════════╤═══════════╤════════════╤════════╕
│   Id │ Group   │ Name    │ Source        │ Instance   │ Repository                                          │ Status   │ Started   │ Finished   │ Info   │
╞══════╪═════════╪═════════╪═══════════════╪════════════╪═════════════════════════════════════════════════════╪══════════╪═══════════╪════════════╪════════╡
│    2 │ manila  │ example │ /example-path │ default    │ s3:https://s3-fr-prevessin-1.cern.ch/manila-example │ pending  │           │            │        │
╘══════╧═════════╧═════════╧═══════════════╧════════════╧═════════════════════════════════════════════════════╧══════════╧═══════════╧════════════╧════════╛

[13:34][root@devstack-cback-71abd95b7d (ceph_devstack:cback/dev) ~]# cback prune ls
╒══════╤═════════╤═══════════════════════════════════════════════╤═════════════════════════════════════════════════════════════════════════════════════════════════════════════╤════════════╤══════════╤═════════════════════╤═════════════════════╤═══════════════╕
│   Id │ Group   │ Name                                          │ Source Backup                                                                                               │ Instance   │ Status   │ Started             │ Finished            │ Info          │
╞══════╪═════════╪═══════════════════════════════════════════════╪═════════════════════════════════════════════════════════════════════════════════════════════════════════════╪════════════╪══════════╪═════════════════════╪═════════════════════╪═══════════════╡
│    2 │ manila  │ example                                       │ /example-path                                                                                               │ default    │ pending  │                     │                     │               │
╘══════╧═════════╧═══════════════════════════════════════════════╧═════════════════════════════════════════════════════════════════════════════════════════════════════════════╧════════════╧══════════╧═════════════════════╧═════════════════════╧═══════════════╛

[13:34][root@devstack-cback-71abd95b7d (ceph_devstack:cback/dev) ~]# cback verify ls
╒══════╤═════════╤═══════════════════════════════════════════════╤═════════════════════════════════════════════════════════════════════════════════════════════════════════════╤════════════╤══════════╤═══════════╤════════════╤════════╕
│   Id │ Group   │ Name                                          │ Source Backup                                                                                               │ Instance   │ Status   │ Started   │ Finished   │ Info   │
╞══════╪═════════╪═══════════════════════════════════════════════╪═════════════════════════════════════════════════════════════════════════════════════════════════════════════╪════════════╪══════════╪═══════════╪════════════╪════════╡
│    2 │ manila  │ example                                       │ /example-path                                                                                               │ default    │ pending  │           │            │        │
╘══════╧═════════╧═══════════════════════════════════════════════╧═════════════════════════════════════════════════════════════════════════════════════════════════════════════╧════════════╧══════════╧═══════════╧════════════╧════════╛

[warning❗] the mechanism for ensuring the filepath you wish to target in your workload is accessable on the cback systems worker nodes, is left to the reader. As an example, at CERN, we ensure all worker nodes locally mount target eos and ceph clusters file systems.