Async jobs between Go and Ruby

This post is the first article of the #FridayTechnical serie. We’ll try to post some interesting technical content about the languages, the libraries and the tools we use to build our product.

Our technological stack is mostly based on Ruby and Go. At some point, we had to make those languages communicate with each other. This article focuses on asynchronous jobs management between them.

In any infrastructure which is not composed of a single process, background workers are necessary. Sidekiq is a fairly popular tool to achieve this in the ruby ecosystem. It stores its state in a redis database, common to the clients and to the workers. At Appsdeck, our infrastructure is more complicated than that, and it is a requirement for us to send asynchronous tasks from service to service.

Go-Workers (https://github.com/jrallison/go-workers) is a Sidekiq-compatible tool to handle jobs with Go, using almost the same DSL. We’ll see later, that with some contributions we’ve done, it does its job quite well.

Worker

The worker job is to listen for tasks. It polls regularly the queues you have configured and executes the jobs it finds.

package main

import (
"fmt"

"github.com/Scalingo/go-workers"
)

func main() {
workers.Configure(map[string]string{
"process": "worker1", "password": "secret",
"server": "localhost:6379", "namespace": "goworkers",
}) workers.Process("myqueue", MyGoWorker, 10)
workers.Run()
}

func MyGoWorker(msg *workers.Msg) {
fmt.Println("running task", msg)
}

First, the redis connection and the process identity have to be configured. Then, one or several calls to the Process method define the queues and the worker functions associated to them.

The worker method, receives a *workers.Msg as argument. It is actually a wrapper of a *simplejson.Json containing the arguments.

require "sidekiq"

Sidekiq::configure_server do |config|
config.redis = { url: "redis://:secret@localhost:6379", namespace: "goworkers" }
end

Sidekiq.options[:queues] = ["myqueue"]

class MyRubyWorker
include Sidekiq::Worker

def perform(str)
puts "Hello from Sidekiq: #{str}"
end
end

The process is pretty similar with Sidekiq, except that the Worker classes are automatically registered when including Sidekiq::Worker.

Client

The client is the process which adds jobs to the queues. The configuration setup is similar, but instead of registering workers, jobs are enqueued in queues.

package main

import "github.com/Scalingo/go-workers"

func main() {
workers.Configure(map[string]string{
"process": "client1", "password": "secret",
"server": "localhost:6379", "namespace": "goworkers", })
workers.Enqueue("myqueue", "MyRubyWorker", []string{"hello"})
}

Here is the main difference between the standard Sidekiq workflow. Instead of using MyRubyWorker.enqueue "arg1", it is required to use Sidekiq::Client.push. The reason is simple, the classes are not defined in the Ruby program, but in the Go worker. (we could define empty classes, but it’s, in my opinion, clumsy.)

require "sidekiq"

Sidekiq::configure_client do |config|
config.redis = { url: "redis://:secret@localhost:6379", namespace: "goworkers" }
end

Sidekiq::Client.push "queue" => "myqueue", "class" => "MyGoWorker", "args" => ["hello"]

Job scheduling

The package jrallison/go-workers is actually working but is lacking some features, especially, scheduled jobs enqueueing. For instance, if you want to send an email to your users at 9 in the morning, it is important to have this feature. Our fork (https://github.com/Scalingo/go-workers) of the project (waiting PR to be included upstream) has these features.

package main

import (
"time"

"github.com/Scalingo/go-workers"
)

func main() {
workers.Configure(map[string]string{
"process": "client1", "password": "secret",
"server": "localhost:6379", "namespace": "goworkers",
})
workers.EnqueueIn("myqueue", "MyRubyWorker", 2*60, []string{"in two minutes"})

now := time.Now()
hoursTo9 := (time.Duration(9 - now.Hour())) * time.Hour
at := now.AddDate(0, 0, 1).Truncate(time.Hour).Add(hoursTo9).Unix()

workers.EnqueueAt("myqueue", "MyGoWorker", at, []string{"tomorrow at 9"})
}

Instead of using workers.Enqueue, you can now use workers.EnqueueIn or workers.EnqueueAt (the same way you would use enqueue_in or enqueue_at with Sidekiq. The previous example uses those functions, and calculates the date of tomorrow 9:00am.

Conclusion

With go-workers, it’s really easy to interface your jobs between go and ruby to enjoy the best of both worlds. For instance: nice front-end gems for HTML/Email rendering for ruby and efficient concurrent tasks handling for Go.

Links

Code samples: https://github.com/Scalingo/go-workers-examples Go worker library: https://github.com/Scalingo/go-workers Sidekiq: http://sidekiq.org Homepage: https://appsdeck.eu

Credits

– Léo Unbekandt, CTO @ Appsdeck