Kubernetes Job: Complete Guide to Batch Processing & Short-Lived Tasks

What is a Kubernetes Job?

A Job creates one or more Pods and ensures that a specified number of them successfully terminate. As pods successfully complete, the Job tracks the successful completions. When a specified number of successful completions is reached, the Job itself completes.

Short-Lived Tasks

Workload for short-lived tasks and batch processes

Completion Tracking

Tracks successful pod completions until target is reached

Automatic Management

Automatically manages pod creation and termination

Default Behavior: By default, Jobs with more than 1 pod will create them sequentially (one after the other). To create them simultaneously, use the parallelism parameter.

kubectl Commands for Jobs

Imperative Creation

kubectl create job [jobName] --image=busybox

The imperative way to create a Job

Declarative Creation

kubectl apply -f [definition.yaml]

Create a Job using YAML configuration

List Jobs

kubectl get job

Get Detailed Info

kubectl describe job [jobName]

Delete a Job

                        kubectl delete -f [definition.yaml]

                        kubectl delete job [jobName]

Two ways to delete: using the YAML file or the Job name

Job YAML Configuration

Here's an example Job configuration that calculates Pi to 2000 decimal places:

apiVersion: batch/v1
kind: Job
metadata:
  name: pi
spec:
  activeDeadlineSeconds: 30  # Max seconds to run
  parallelism: 3             # How many pods should run in parallel
  completions: 3             # How many successful completions are needed
  template:
    spec:
      containers:
      - name: pi
        image: perl
        command: ["perl", "-Mbignum=bpi", "-wle", "print bpi(2000)"]
      restartPolicy: Never   # Default is "Always"

Key Fields Explained

apiVersion: batch/v1 (for Jobs)
kind: Job
parallelism: Number of pods to run concurrently
completions: Total number of successful completions needed
restartPolicy: Never or OnFailure (not Always)

Container Spec

name: Identifier for the container
image: Container image to use
command: Command to execute in the container
args: Arguments to pass to the command (optional)

Job Configuration Parameters

Parameter	Description	Default
completions	Number of successful pod completions required	1
parallelism	Maximum number of pods running concurrently	1
activeDeadlineSeconds	Maximum duration in seconds the job can run	No limit
backoffLimit	Number of retries before marking job as failed	6
ttlSecondsAfterFinished	Time to live after completion (auto-deletion)	Not set

Sequential Execution

When parallelism: 1 and completions: 3, pods run one after another:

Pod 1 Pod 2 Pod 3

Pod 1 ✓ Pod 2 Pod 3

Parallel Execution

When parallelism: 3 and completions: 3, pods run simultaneously:

Pod 1 Pod 2 Pod 3

Pod 1 ✓ Pod 2 ✓ Pod 3 ✓

Common Job Use Cases

Batch Processing

Data processing, calculations, or transformations

Database Operations

Database migrations, backups, or maintenance tasks

Report Generation

Generating reports, exports, or data extracts

Scheduled Tasks

Cron-like tasks using CronJobs (built on Jobs)

Testing

Running integration tests or one-off validation tasks

Data Import/Export

Importing or exporting data to/from external systems