Redis

redisq is the Redis adapter for VarMQ. It provides first-in first-out and priority queues on one Redis server. Jobs survive a restart, and several processes can share each queue. Read persistence and distribution first for what such a queue does.

Setup

redisq needs Redis 6.0 or newer, with Lua scripts allowed: it takes, adds and puts back jobs with scripts. Its go.mod asks for Go 1.26. Three steps set it up:

1. 1

   Start Redis

   varmq’s repository has examples/compose.yml, which starts Redis 7.2 on port 6375. The programs on this page use that port.

   From a cloneWithout a clone

   terminalCopy

   docker compose -f examples/compose.yml up -d

   Run it from a clone of github.com/goptics/varmq.

2. 2

   Add the two modules

   varmq’s version is the release these pages describe.

   terminalCopy

   go get github.com/goptics/varmq github.com/goptics/redisq@v1.6.0

3. 3

   Create a queue and bind it

   redisq.New takes a Redis URL and returns a Queues value. It panics when the URL does not parse. Each queue takes a key, which names it in Redis. Pick the order.

   First in, first outLowest number firstProducer only

   main.goCopy

   rdb := redisq.New("redis://localhost:6375")
   defer rdb.Close()
   
   rq := rdb.NewQueue("jobs")
   defer rq.Close()
   
   queue := worker.WithQueue(rq)

   First in, first out, on a Redis list. Jobs survive a restart. Several processes can consume one queue, and each job goes to one of them.

Queues has one constructor for each order. Each takes a key, which names the queue in Redis:

Every redisq queue can be shared by several processes. redisq’s older distributed constructors are deprecated aliases of these two. A process that only adds jobs wraps the queue with varmq.NewQueue, as below.

A queue with the key jobs keeps its waiting jobs under jobs. Every other key it uses, and its wake-up channels, start with jobs:. Give each queue its own key.

Redis keeps these in memory. Redis’s own persistence settings decide whether they survive a restart of Redis itself. Neither VarMQ nor redisq changes these settings.

A persistent queue

The program below binds a queue to a worker with concurrency 1, adds ten jobs, and waits for them.

11 collapsed lines
package main

import (
	"fmt"
	"time"

	"github.com/goptics/redisq"
	"github.com/goptics/varmq"
)

func main() {
	rdb := redisq.New("redis://localhost:6375")
	defer rdb.Close()

	pq := rdb.NewQueue("persistent_queue")
	defer pq.Close()

	worker := varmq.NewWorker(
		func(j varmq.Job[int]) {
			fmt.Println("processing", j.Data())
			time.Sleep(time.Second)
			fmt.Println("processed", j.Data())
		},
		varmq.WithAutoRun(false),
	)

	queue := worker.WithQueue(pq)
	fmt.Println("left in Redis by an earlier run:", queue.Len())

	// Start after the bind, so the worker looks at jobs already in Redis.
	worker.Start()

	for i := range 10 {
		queue.Add(i)
	}

	worker.WaitAndStop()
}

Three choices in it matter:

• The worker is created with WithAutoRun(false) and started after the bind. Binding a redisq queue already makes the worker look for jobs that a previous run left in Redis. The overview says this. Starting the worker later only delays it while the program counts the jobs.
• queue.Len() is read before Start. It asks Redis, so it counts the waiting jobs from the earlier run.
• Every job finishes before the deferred closes run. WaitAndStop lets every job run, then stops the worker. A job acknowledges when its function returns. Close on a redisq queue puts back the jobs it took and has not acknowledged. If you close it while a job runs, that job goes back to the queue to run again. The job’s acknowledgement then fails.

Run locally

1. Start Redis, as in Setup.

2. Make a module, save the program as main.go and add the dependencies:

   mkdir redis-persistent && cd redis-persistent
   go mod init redis-persistent
   go get github.com/goptics/varmq github.com/goptics/redisq@v1.6.0

   Copy

3. Run it with go run .. The first run prints left in Redis by an earlier run: 0, then a processing line and a processed line for each job, one second apart. Concurrency is 1, so they run in the order they were added.

4. Run it again, and press Ctrl+C a few seconds in. The jobs that had not started are still in Redis.

5. Run it a third time. It prints how many jobs are left, and the worker runs those first, before the ten new ones.

The job that was running when you pressed Ctrl+C is not in that count. It waits in persistent_queue:nacked until redisq treats the run that took it as dead. A restarted program gets the job back about 3 seconds after it starts, at the front of the queue. If the first run died within about a second of taking the job, it takes the heartbeat timeout instead, 30 seconds. The section on unacknowledged jobs explains the timing.

A priority queue

NewPriorityQueue gives a queue that gives out the lowest number first. WithPriorityQueue binds it, and Add on the handle takes the priority after the data.

10 collapsed lines
package main

import (
	"fmt"

	"github.com/goptics/redisq"
	"github.com/goptics/varmq"
)

func main() {
	rdb := redisq.New("redis://localhost:6375")
	defer rdb.Close()

	pq := rdb.NewPriorityQueue("persistent_priority_queue")
	defer pq.Close()

	worker := varmq.NewWorker(
		func(j varmq.Job[string]) {
			fmt.Println("processing", j.Data())
		},
		varmq.WithAutoRun(false),
	)

	queue := worker.WithPriorityQueue(pq)

	// A lower number runs first.
	queue.Add("report", 10)
	queue.Add("alert", 1)
	queue.Add("newsletter", 5)

	worker.Start()
	worker.WaitAndStop()
}

Run locally

Follow the steps for the first program, with this program as main.go. It prints:

processing alert
processing newsletter
processing report

redisq keeps a priority queue in a Redis sorted set, with the priority as the score. It stores each job with a sequence number in front. So identical jobs stay separate, and jobs with the same priority come out in the order they were added. A job that goes back to the queue keeps its priority.

A distributed queue

Any number of processes can share a queue. The producer below has no worker. It wraps the Redis queue with varmq.NewQueue and adds jobs.

10 collapsed lines
package main

import (
	"fmt"

	"github.com/goptics/redisq"
	"github.com/goptics/varmq"
)

func main() {
	rdb := redisq.New("redis://localhost:6375")
	defer rdb.Close()

	dq := rdb.NewQueue("distributed_queue")
	defer dq.Close()

	// No worker here: this process only adds jobs.
	producer := varmq.NewQueue[string](dq)

	for i := range 20 {
		if _, ok := producer.Add(fmt.Sprintf("https://example.com/%d", i)); !ok {
			fmt.Println("not added:", i)
		}
	}

	fmt.Println("waiting in Redis:", producer.Len())
}

The consumer binds the same Redis queue to a worker with concurrency 5, and runs until you stop it.

15 collapsed lines
package main

import (
	"context"
	"fmt"
	"os"
	"os/signal"
	"syscall"
	"time"

	"github.com/goptics/redisq"
	"github.com/goptics/varmq"
)

func main() {
	rdb := redisq.New("redis://localhost:6375")
	defer rdb.Close()

	dq := rdb.NewQueue("distributed_queue")
	defer dq.Close()

	worker := varmq.NewWorker(
		func(j varmq.Job[string]) {
			fmt.Println("scraping", j.Data())
			time.Sleep(2 * time.Second)
			fmt.Println("scraped", j.Data())
		},
		5,
		varmq.WithAutoRun(false),
	)

	queue := worker.WithQueue(dq)
	fmt.Println("waiting in Redis:", queue.Len())

	worker.Start()
	fmt.Println("listening")

	ctx, stop := signal.NotifyContext(context.Background(), os.Interrupt, syscall.SIGTERM)
	defer stop()
	<-ctx.Done()

	// Let running jobs finish and acknowledge before the queue closes.
	worker.StopAndWait()
}

How it fits together:

• A job reaches one consumer. The consumer takes a job from the Redis list. After that, the job is not in the list for another consumer. Two consumers on one key divide the jobs between them.
• Add wakes an idle consumer. Enqueue adds the job. In the same step, it wakes consumers that are waiting for work, at most one per waiting job. Each consumer has a channel of its own. A consumer waits for work once it finds the queue empty. A busy consumer is not woken, so the wake-up goes to a consumer that can take the job. While jobs wait, every consumer also checks once a second. So a consumer still takes a job that was added while no consumer was listening.
• The producer’s Add returns a job handle. Its Wait returns when a consumer has run the job. Jobs in other processes covers it. Len is the length of the Redis list.
• Each consumer’s counters are its own. A worker counts the jobs it ran. The queue’s own totals are in GET /queues.
• The consumer stops its worker before the deferred Close. Close puts back every job the consumer took and has not acknowledged. It also wakes another consumer for them. So if you close while a job runs, the job runs twice. StopAndWait lets the running jobs finish and acknowledge first. redisq has a Listen method that blocks until SIGINT or SIGTERM. It does not stop the worker.

Run locally

1. Start Redis, as in Setup.

2. In one module, save the producer as producer/main.go and the consumer as consumer/main.go:

   mkdir redis-distributed && cd redis-distributed
   go mod init redis-distributed
   mkdir producer consumer
   go get github.com/goptics/varmq github.com/goptics/redisq@v1.6.0

   Copy

3. In one terminal run go run ./consumer. It prints waiting in Redis: 0 and listening.

4. For a second consumer, run go run ./consumer in another terminal. Each job goes to one of the two.

5. In a third terminal run go run ./producer. It prints waiting in Redis: and a number. The number is 20 when no consumer is running, and lower when one has already taken some jobs.

6. The consumers print a scraping line and, two seconds later, a scraped line for each job. Each consumer runs up to five jobs at a time, so five lines appear together.

7. Press Ctrl+C in a consumer terminal. It finishes the jobs it is running, acknowledges them and exits.

To see a crash instead, run the producer again and kill one consumer with kill -9 while it runs jobs. The other consumer runs them a few seconds later, as the next section explains.

Jobs that were not acknowledged

The overview describes when VarMQ acknowledges a job. This is what redisq does with a job that was taken and not acknowledged, for example because the process died.

When such a job runs again, with redisq v1.6.0:

• Close gives jobs back. Close puts back at once every job its queue object took and has not acknowledged. After it, Enqueue and Acknowledge on that object return false. See shutting down.
• The timeouts are set per queue object. SetVisibilityTimeout changes the deadline for jobs taken after the call. SetHeartbeatTimeout sets how long the object may go without reaching Redis: at least five seconds, and 0 turns heartbeats off. Call them right after you create the queue.
• Keep the visibility timeout longer than your slowest job. A job still running past its deadline is put back by the next sweep and runs again.
• Keep the heartbeat timeout longer than any freeze you want to survive. A process can be frozen or cut off from Redis for longer than this timeout. Then redisq treats it as dead. Its running jobs run again elsewhere, and their later Acknowledge still returns true.
• GetNackedItemsCount counts the jobs taken and not acknowledged. The handle’s Len does not, because they have left the queue.

How redisq tracks a taken job

Job IDs

redisq checks job IDs in EnqueueOnce. An Add with WithJobId stores the job once per ID within a window, five minutes by default. That covers an Add you retry, and a push that the Redis client sent again after its reply was lost. Job IDs says what Add returns for a repeat. SetDedupWindow on the queue object changes the window, and 0 turns the check off.

A job without an ID is not checked. If the reply to its push is lost, the Redis client sends the push again, and the job can be stored twice. Give jobs IDs when that matters.

Pause, outcome, cancel and workflows

redisq v1.6.0 supports these VarMQ features. Each guide has the details.

• Queue pause: the pause is kept in Redis, so every process stops taking jobs, and it survives a restart.
• Job outcome: a producer’s job handle learns how the job ended, with a result worker’s result.
• Cancel: a cancel reaches the job in whichever process holds it.
• Workflows: tasks can run on redisq queues, with the workflow kept in Redis.

Batches

redisq implements IBatchAcknowledgeable. So a worker with only this queue takes as many jobs as it has free slots in one call. Each job starts on its own goroutine at once.

Next: SQLite.