Anton Zhiyanov

Building blocks for idiomatic Go pipelines

Mon, 11 Aug 2025 12:30:00 +0000

I've created a Go package called chans that offers generic channel operations to make it easier to build concurrent pipelines. It aims to be flexible, unopinionated, and composable, without over-abstracting or taking control away from the developer.

Here's a toy example:

// Given a channel of documents.
docs := make(chan []string, 10)
docs <- []string{"go", "is", "awesome"}
docs <- []string{"cats", "are", "cute"}
close(docs)

// Extract all words from the documents.
words := make(chan string, 10)
chans.Flatten(ctx, words, docs)
close(words)

// Calculate the total byte count of all words.
step := func(acc int, word string) int { return acc + len(word) }
count := chans.Reduce(ctx, words, 0, step)
fmt.Println("byte count =", count)

byte count = 22

Now let's go over the features.

The all-time classics

Filter sends values from the input channel to the output if a predicate returns true.

■ □ ■ □ → ■ ■

Map reads values from the input channel, applies a function, and sends the result to the output.

■ ■ ■ → ● ● ●

Reduce combines all values from the input channel into one using a function and returns the result.

■ ■ ■ ■ → ∑

Filtering and sampling

FilterOut ignores values from the input channel if a predicate returns true, otherwise sends them to the output.

■ □ ■ □ → □ □

Drop skips the first N values from the input channel and sends the rest to the output.

➊ ➋ ➌ ➍ → ➌ ➍

DropWhile skips values from the input channel as long as a predicate returns true, then sends the rest to the output.

■ ■ ▲ ● → ▲ ●

Take sends up to N values from the input channel to the output.

➊ ➋ ➌ ➍ → ➊ ➋

TakeNth sends every Nth value from the input channel to the output.

➊ ➋ ➌ ➍ → ➊ ➌

TakeWhile sends values from the input channel to the output while a predicate returns true.

■ ■ ▲ ● → ■ ■

First returns the first value from the input channel that matches a predicate.

■ ■ ▲ ● → ▲

Batching and windowing

Chunk groups values from the input channel into fixed-size slices and sends them to the output.

■ ■ ■ ■ ■ → ■ ■ │ ■ ■ │ ■

ChunkBy groups consecutive values from the input channel into slices whenever the key function's result changes.

■ ■ ● ● ▲ → ■ ■ │ ● ● │ ▲

Flatten reads slices from the input channel and sends their elements to the output in order.

■ ■ │ ■ ■ │ ■ → ■ ■ ■ ■ ■

De-duplication

Compact sends values from the input channel to the output, skipping consecutive duplicates.

■ ■ ● ● ■ → ■ ● ■

CompactBy sends values from the input channel to the output, skipping consecutive duplicates as determined by a custom equality function.

■ ■ ● ● ■ eq→ ■ ● ■

Distinct sends values from the input channel to the output, skipping all duplicates.

■ ■ ● ● ■ → ■ ●

DistinctBy sends values from the input channel to the output, skipping duplicates as determined by a key function.

■ ■ ● ● ■ key→ ■ ●

Routing

Broadcast sends every value from the input channel to all output channels.

➊ ➋ ➌ ➍
↓
➊ ➋ ➌ ➍
➊ ➋ ➌ ➍

Split sends values from the input channel to output channels in round-robin fashion.

➊ ➋ ➌ ➍
↓
➊ ➌
➋ ➍

Partition sends values from the input channel to one of two outputs based on a predicate.

■ □ ■ □
↓
■ ■
□ □

Merge concurrently sends values from multiple input channels to the output, with no guaranteed order.

■ ■ ■
● ● ●
↓
● ● ■ ■ ■ ●

Concat sends values from multiple input channels to the output, processing each input channel in order.

■ ■ ■
● ● ●
↓
■ ■ ■ ● ● ●

Drain consumes and discards all values from the input channel.

■ ■ ■ ■ → ∅

Motivation

I think third-party concurrency packages are often too opinionated and try to hide too much complexity. As a result, they end up being inflexible and don't fit a lot of use cases.

For example, here's how you use the Map function from the rill package:

// Concurrency = 3
users := rill.Map(ids, 3, func(id int) (*User, error) {
    return db.GetUser(ctx, id)
})

The code looks simple, but it makes Map pretty opinionated and not very flexible:

The function is non-blocking and spawns a goroutine. There is no way to change this.
The function doesn't exit early on error. There is no way to change this.
The function creates the output channel. There is no way to control its buffering or lifecycle.
The function can't be canceled.
The function requires the developer to use a custom Try[T] type for both input and output channels.
The "N workers" logic is baked in, so you can't use a custom concurrent group implementation.

While this approach works for many developers, I personally don't like it. With chans, my goal was to offer a fairly low-level set of composable channel operations and let developers decide how to use them.

For comparison, here's how you use the chans.Map function:

err := chans.Map(ctx, users, ids, func(id int) (*User, error) {
    return db.GetUser(ctx, id)
})

chans.Map only implements the core mapping logic:

Reads values from the input channel.
Calls the mapping function on each value.
Writes results to the output channel.
Stops if there's an error or if the context is canceled.
Does not start any additional goroutines.

You decide the rest:

Want Map to be non-blocking? Run it in a goroutine.
Don't want to exit early? Gather the errors instead of returning them.
Want to buffer the output channel or keep it open? You have full control.
Need to process input in parallel? Use errgroup.Group, or sync.WaitGroup, or any other implementation.

The same principles apply to other channel operations.

Usage example

Let's say we want to calculate the total balance of VIP user accounts:

func calcVIPsBalance(ctx context.Context, ids chan string) (float64, error) {
	// ...
}

Here's how we can do it using chans.

First, use Map to get the accounts from the database:

errs := make([]error, 2)
accounts := make(chan *Account)
go func() {
    getAcc := func(id string) (*Account, error) {
        return db.GetAccount(ctx, id)
    }
    err := chans.Map(ctx, accounts, ids, getAcc)
    errs[0] = err
    close(accounts)
}()

Next, use Filter to select only the VIP accounts:

vips := make(chan *Account)
go func() {
    onlyVIP := func(acc *Account) (bool, error) {
        return acc.VIP, nil
    }
    err := chans.Filter(ctx, vips, accounts, onlyVIP)
    errs[1] = err
    close(vips)
}()

Next, use Reduce to calculate the total balance:

sumBalance := func(total float64, acc *Account) float64 {
    return total + acc.Balance
}
total := chans.Reduce(ctx, vips, 0, sumBalance)

Finally, check for errors and return the result:

err := errors.Join(errs[0], errs[1], ctx.Err())
if err != nil {
    return 0, err
}
return total, nil

That's it!

Final thoughts

If you're building concurrent pipelines in Go, you might find chans useful.

See the nalgeon/chans repo if you are interested.

──

P.S. Want to learn more about concurrency? Check out my interactive book

Gist of Go: Signaling

Sun, 20 Jul 2025 18:00:00 +0000

This is a chapter from my book on Go concurrency, which teaches the topic from the ground up through interactive examples.

The main way goroutines communicate in Go is through channels. But channels aren't the only way for goroutines to signal each other. Let's try a different approach!

Signaling • One-time subscription • Broadcasting • Broadcasting w/channels • Publish/subscribe • Run once • OnceFunc • sync.Pool • Keep it up

Signaling

Let's say we have a goroutine that generates a random number between 1 and 100:

num := 0
go func() {
    num = 1 + rand.IntN(100)
}()

And the second one checks if the number is lucky or not:

go func() {
    if num%7 == 0 {
        fmt.Printf("Lucky number %d!\n", num)
    } else {
        fmt.Printf("Unlucky number %d...\n", num)
    }
}()

The second goroutine will only work correctly if the first one has already set the number. So, we need to find a way to synchronize them. For example, we can make num a channel:

var wg sync.WaitGroup
num := make(chan int, 1)

// Generates a random number from 1 to 100.
wg.Go(func() {
    num <- 1 + rand.IntN(100)
})

// Checks if the number is lucky.
wg.Go(func() {
    n := <-num
    if n%7 == 0 {
        fmt.Printf("Lucky number %d!\n", n)
    } else {
        fmt.Printf("Unlucky number %d...\n", n)
    }
})

wg.Wait()

Unlucky number 37...

But what if we want num to be a regular number, and channels are not an option?

We can make the generator goroutine signal when a number is ready, and have the checker goroutine wait for that signal. In Go, we can do this using a condition variable, which is implemented with the sync.Cond type.

A Cond has a mutex inside it:

cond := sync.NewCond(&sync.Mutex{})
// The mutex is available through the cond.L field.
fmt.Printf("%#v\n", cond.L)

&sync.Mutex{state:0, sema:0x0}

A Cond has two methods — Wait and Signal:

Wait unlocks the mutex and suspends the goroutine until it receives a signal.
Signal wakes the goroutine that is waiting on Wait.
When Wait wakes up, it locks the mutex again.

If there are multiple waiting goroutines when Signal is called, only one of them will be resumed. If there are no waiting goroutines, Signal does nothing.

To see why Cond needs to go through all this mutex trouble, check out this example:

cond := sync.NewCond(&sync.Mutex{})
num := 0

// Generates a random number from 1 to 100.
go func() {
    time.Sleep(10 * time.Millisecond)
    cond.L.Lock()    // (1)
    num = 1 + rand.IntN(100)
    cond.Signal()    // (2)
    cond.L.Unlock()
}()

// Checks if the number is lucky.
go func() {
    cond.L.Lock()    // (3)
    if num == 0 {
        cond.Wait()  // (4)
    }
    if num%7 == 0 {
        fmt.Printf("Lucky number %d!\n", num)
    } else {
        fmt.Printf("Unlucky number %d...\n", num)
    }
    cond.L.Unlock()
}()

Both goroutines use the shared num variable, so we need to protect it with a mutex.

The checker goroutine starts by locking the cond.L mutex ➌. If the generator hasn't run yet (meaning num is 0), the goroutine calls cond.Wait() ➍ and blocks. If Wait only blocked the goroutine, the mutex would stay locked, and the generator couldn't change num ➊. That's why Wait unlocks the mutex before blocking.

The generator goroutine also starts by locking the mutex ➊. After setting the num value, the generator calls cond.Signal() ➋ to let the checker know it's ready, and then unlocks the mutex. Now, if resumed Wait ➍ did nothing, the checker goroutine would continue running. But the mutex would stay unlocked, so working with num wouldn't be safe. That's why Wait locks the mutex again after receiving the signal.

In theory, everything should work. Here's the output:

Lucky number 77!

Everything seems fine, but there's a subtle bug. When the checker goroutine wakes up after receiving a signal, the mutex is unlocked for a brief moment before Wait locks it again. Theoretically, in that short time, another goroutine could sneak in and set num to 0. The checker goroutine wouldn't notice this and would keep running, even though it's supposed to wait if num is zero.

That's why, in practice, Wait is always called inside a for loop, not inside an if statement.

Not like this:

if num == 0 {
    cond.Wait()
}

But like this (note that the condition is the same as in the if statement):

for num == 0 {
    cond.Wait()
}

In most cases, this for loop will work just like an if statement:

The goroutine receives a signal and wakes up.
It locks the mutex.
On the next loop iteration, it checks the num value.
Since num is not 0, it exits the loop and continues.

But if another goroutine intervenes between ➊ and ➋ and sets num to zero, the goroutine will notice this at ➌ and go back to waiting. This way, it will never keep running when num is zero — which is exactly what we want.

Here's the complete example:

var wg sync.WaitGroup
cond := sync.NewCond(&sync.Mutex{})
num := 0

// Generates a random number from 1 to 100.
wg.Go(func() {
    time.Sleep(10 * time.Millisecond)
    cond.L.Lock()
    num = 1 + rand.IntN(100)
    cond.Signal()
    cond.L.Unlock()
})

// Checks if the number is lucky.
wg.Go(func() {
    cond.L.Lock()
    for num == 0 {
        cond.Wait()
    }
    if num%7 == 0 {
        fmt.Printf("Lucky number %d!\n", num)
    } else {
        fmt.Printf("Unlucky number %d...\n", num)
    }
    cond.L.Unlock()
})

wg.Wait()

Lucky number 35!

Like other synchronization primitives, a condition variable has its own internal state. So, you should only pass it as a pointer, not by value. Even better, don't pass it at all — wrap it inside a type instead. We'll do this in the next step.

✎ Exercise: Blocking queue

Practice is crucial in turning abstract knowledge into skills, making theory alone insufficient. The full version of the book contains a lot of exercises — that's why I recommend getting it.

If you are okay with just theory for now, let's continue.

One-time subscription

Let's go back to the lucky numbers example:

// Generates a random number from 1 to 100.
go func() {
    // ...
}()

// Checks if the number is lucky.
go func() {
    // ...
}()

Let's refactor the code and create a Lucky type with Guess and Wait methods:

// Guess generates a random number and notifies
// the subscriber who's waiting with Wait.
Guess()

// Wait waits for a notification about a new number,
// then calls the subscriber's callback function.
Wait(callback func(int))

Here's the implementation:

// Lucky generates a random number.
type Lucky struct {
    cond *sync.Cond
    num  int
}

// NewLucky creates a new Lucky.
func NewLucky() *Lucky {
    l := &Lucky{}
    l.cond = sync.NewCond(&sync.Mutex{})
    return l
}

// Guess generates a random number and notifies
// the subscriber who's waiting with Wait.
func (l *Lucky) Guess() {
    l.cond.L.Lock()
    defer l.cond.L.Unlock()
    l.num = 1 + rand.IntN(100)
    l.cond.Signal()
}

// Wait waits for a notification about a new number,
// then calls the subscriber's callback function.
func (l *Lucky) Wait(callback func(int)) {
    l.cond.L.Lock()
    // Wait for a signal about a new number.
    for l.num == 0 {
        l.cond.Wait()
    }

    // Make a copy of the number to avoid holding
    // the lock while calling the callback.
    num := l.num
    l.cond.L.Unlock()

    // Call the subscriber's callback function.
    callback(num)
}

Example usage:

func main() {
    var wg sync.WaitGroup
    lucky := NewLucky()

    wg.Add(1)
    go lucky.Wait(func(num int) {
        if num%7 == 0 {
            fmt.Printf("Lucky number %d!\n", num)
        } else {
            fmt.Printf("Unlucky number %d...\n", num)
        }
        wg.Done()
    })

    lucky.Guess()
    wg.Wait()
}

Lucky number 35!

Note that this is a one-time signaling, not a long-term subscription. Once a subscriber goroutine receives the generated number, it is no longer subscribed to Lucky. We'll look at an example of a long-term subscription later in the chapter.

Everything works, but there's still a problem. If you call Wait from N goroutines, you can set up N subscribers, but Guess only notifies one of them. We'll figure out how to notify all of them in the next step.

Broadcasting

Notifying all subscribers instead of just one is called broadcasting. To do this in Lucky, we only need to change one line in the Guess method:

// Guess generates a random number and notifies
// the subscribers who are waiting with Wait.
func (l *Lucky) Guess() {
    l.cond.L.Lock()
    defer l.cond.L.Unlock()
    l.num = 1 + rand.IntN(100)
    // Using Broadcast instead of Signal.
    l.cond.Broadcast()
}

The Signal method wakes up one goroutine that's waiting on Cond.Wait, while the Broadcast method wakes up all goroutines waiting on Cond.Wait. This is exactly what we need.

Here's a usage example:

// checkLucky checks if num is a lucky number.
func checkLucky(num, divisor int) {
    if num%divisor == 0 {
        fmt.Printf("mod %d: lucky number %d!\n", divisor, num)
    } else {
        fmt.Printf("mod %d: unlucky number %d...\n", divisor, num)
    }
}

func main() {
    var wg sync.WaitGroup
    wg.Add(3)

    lucky := NewLucky()
    go lucky.Wait(func(num int) {
        checkLucky(num, 3)
        wg.Done()
    })
    go lucky.Wait(func(num int) {
        checkLucky(num, 7)
        wg.Done()
    })
    go lucky.Wait(func(num int) {
        checkLucky(num, 13)
        wg.Done()
    })

    lucky.Guess()
    wg.Wait()
}

mod 3: unlucky number 98...
mod 13: unlucky number 98...
mod 7: lucky number 98!

A typical way to use a condition variable looks like this:

There is a publisher goroutine and one or more subscriber goroutines. They all use some shared state protected by a condition variable c.
The publisher goroutine changes the shared state and notifies either one subscriber (Signal) or all subscribers (Broadcast):

c.L.Lock()
// ... change the shared state
c.Signal() // or c.Broadcast()
c.L.Unlock()

The subscriber goroutine waits for an event in a loop, then works with the new shared state:

c.L.Lock()
for !condition() {
    c.Wait()
}
// ... use the changed shared state
c.L.Unlock()

Note that this is a one-time notification, not a long-term subscription. Once a subscriber goroutine receives the signal, it is no longer subscribed to the publisher. We'll look at an example of a long-term subscription later in the chapter.

✎ Exercise: Barrier using a condition variable

Practice is crucial in turning abstract knowledge into skills, making theory alone insufficient. The full version of the book contains a lot of exercises — that's why I recommend getting it.

If you are okay with just theory for now, let's continue.

Broadcasting with channels

As we discussed, it's easy to implement signaling using a channel:

// Common state.
num := make(chan int, 1)

// Sender goroutine.
go func() {
    // Send a signal.
    num <- 1 + rand.Intn(100)
}()

// Receiver goroutine.
go func() {
    // Receive a signal.
    n := <-num
    // ...
}()

This approach only works with one receiver. If we subscribe multiple goroutines to the num channel, only one of them will get the generated number.

For broadcasting, we can just close the channel:

var wg sync.WaitGroup

// Common state and mutex to protect it.
var mu sync.Mutex
var num int

// Broadcasting channel.
broadcast := make(chan struct{})

// Sender goroutine.
wg.Go(func() {
    // Generate a number.
    mu.Lock()
    num = 1 + rand.IntN(100)
    mu.Unlock()
    // Send a broadcast.
    close(broadcast)
})

// Receiver goroutine.
wg.Go(func() {
    // Receive a signal.
    <-broadcast
    mu.Lock()
    // Use the generated number.
    fmt.Println("received", num)
    mu.Unlock()
})

// Receiver goroutine.
wg.Go(func() {
    // Receive a signal.
    <-broadcast
    mu.Lock()
    // Use the generated number.
    fmt.Println("received", num)
    mu.Unlock()
})

wg.Wait()

received 6
received 6

However, we can only broadcast the fact that the state has changed (the broadcast channel is closed), not the actual state (the value of num). So, we still need to protect the state with a mutex. This goes against the idea of using channels to pass data between goroutines. Also, we can't send a second broadcast notification because a channel can only be closed once.

Publish/subscribe

We have two problems with our "broadcast by closing a channel" approach:

The broadcast only sends a signal, not the actual state.
The broadcast only works once.

Let's solve both problems and create a simple publish/subscribe system:

The Lucky type generates random numbers (publisher).
Multiple goroutines want to receive these numbers (subscribers).
Each subscriber has its own channel registered with Lucky.
After generating a number, Lucky sends it to each subscriber's channel.

Here's the Lucky type:

// Lucky generates random numbers
// and notifies subscribers.
type Lucky struct {
    // sbox stores a slice of subscription channels.
    // It lets us safely work with the slice
    // without needing a mutex.
    sbox chan []chan int
}

// NewLucky creates a new Lucky.
func NewLucky() *Lucky {
    sbox := make(chan []chan int, 1)
    sbox <- nil // no subscribers initially
    return &Lucky{sbox: sbox}
}

The Subscribe method adds a subscriber and returns the channel where random numbers will be sent:

// Subscribe adds a new subscriber and returns
// a channel to receive lucky numbers.
func (l *Lucky) Subscribe() <-chan int {
    subs := <-l.sbox
    sub := make(chan int, 1)
    subs = append(subs, sub)
    l.sbox <- subs
    return sub
}

Note that we use l.sbox as a mutex here to protect access to the shared list of subscription channels. Without it, concurrent Subscribe calls would cause a data race on subs. Alternatively, we can use a regular channel slice l.subs and protect is with a mutex l.mu:

// Lucky generates random numbers
// and notifies subscribers.
type Lucky struct {
    subs []chan int
    mu   sync.Mutex
}

// Subscribe adds a new subscriber and returns
// a channel to receive lucky numbers.
func (l *Lucky) Subscribe() <-chan int {
    l.mu.Lock()
    defer l.mu.Unlock()
    sub := make(chan int, 1)
    l.subs = append(l.subs, sub)
    return sub
}

The Guess number generates a number and sends it to each subscriber:

// Guess generates a random number
// and notifies subscribers.
func (l *Lucky) Guess() {
    subs := <-l.sbox
    num := 1 + rand.IntN(100)
    for _, sub := range subs {
        select {
        case sub <- num:
        default:
            // Subscriber is not ready
            // to receive, drop the number.
        }
    }
    l.sbox <- subs
}

Our Guess implementation drops the message if the subscriber hasn't processed the previous one. So, Guess always works quickly and doesn't block, but slower subscribers might miss some data. Alternatively, we can use a blocking sub <- num without select to make sure everyone gets all the data, but this means the whole system will only run as fast as the slowest subscriber.

The Stop method terminates all subscriptions:

// Stop unsubscribes all subscribers.
func (l *Lucky) Stop() {
    subs := <-l.sbox
    for _, sub := range subs {
        close(sub)
    }
    l.sbox <- nil
}

Here's an example with three subscribers. Each one gets three random numbers:

func main() {
    var wg sync.WaitGroup

    lucky := NewLucky()
    sub1 := lucky.Subscribe()
    sub2 := lucky.Subscribe()
    sub3 := lucky.Subscribe()

    wg.Go(func() {
        for num := range sub1 {
            checkLucky(num, 3)
        }
    })
    wg.Go(func() {
        for num := range sub2 {
            checkLucky(num, 7)
        }
    })
    wg.Go(func() {
        for num := range sub3 {
            checkLucky(num, 13)
        }
    })

    lucky.Guess()
    time.Sleep(10 * time.Millisecond)

    lucky.Guess()
    time.Sleep(10 * time.Millisecond)

    lucky.Guess()
    time.Sleep(10 * time.Millisecond)

    lucky.Stop()
    wg.Wait()
}

mod 13: unlucky number 42...
mod 3: lucky number 42!
mod 7: lucky number 42!
mod 3: lucky number 36!
mod 13: unlucky number 36...
mod 7: unlucky number 36...
mod 13: unlucky number 9...
mod 3: lucky number 9!
mod 7: unlucky number 9...

That's it for signals and broadcasting! Now let's look at a couple more tools from the sync package.

Run once

Let's say we have a currency converter:

// CurrencyConverter converts money
// amounts between currencies.
type CurrencyConverter struct {
    rates map[string]float64
}

// Converts an amount from one currency to another.
func (c *CurrencyConverter) Convert(amount float64, from, to string) float64 {
    // Skipping validation for simplicity.
    return amount * c.rates[to] / c.rates[from]
}

Exchange rates are loaded from an external API, so we decided to fetch them lazily the first time Convert is called:

// init loads exchange rates from an external source.
func (c *CurrencyConverter) init() {
    // Simulate a network delay.
    time.Sleep(100 * time.Millisecond)
    c.rates = map[string]float64{"USD": 1.0, "EUR": 0.86}
}

// Converts an amount from one currency to another.
func (c *CurrencyConverter) Convert(amount float64, from, to string) float64 {
    if c.rates == nil {
        c.init()
    }
    return amount * c.rates[to] / c.rates[from]
}

Unfortunately, this creates a data race on c.rates when used in a concurrent environment:

func main() {
    cc := new(CurrencyConverter)

    var wg sync.WaitGroup
    for i := range 4 {
        wg.Go(func() {
            usd := 100.0 * float64(i+1)
            eur := cc.Convert(usd, "USD", "EUR")
            fmt.Printf("%v USD = %v EUR\n", usd, eur)
        })
    }
    wg.Wait()
}

==================
WARNING: DATA RACE
Write at 0x00c000058028 by goroutine 9:
  ...

Previous write at 0x00c000058028 by goroutine 6:
  ...
==================

We could protect the rates field with a mutex. Or we could use the sync.Once type. It guarantees that a function called with Once.Do() runs only once:

// CurrencyConverter converts money
// amounts between currencies.
type CurrencyConverter struct {
    rates map[string]float64
    once  sync.Once
}

// init loads exchange rates from an external source.
func (c *CurrencyConverter) init() {
    time.Sleep(100 * time.Millisecond)
    c.rates = map[string]float64{"USD": 1.0, "EUR": 0.86}
}

// Converts an amount from one currency to another.
func (c *CurrencyConverter) Convert(amount float64, from, to string) float64 {
    c.once.Do(c.init)
    return amount * c.rates[to] / c.rates[from]
}

400 USD = 344 EUR
200 USD = 172 EUR
100 USD = 86 EUR
300 USD = 258 EUR

Once.Do makes sure that the given function runs only once. If multiple goroutines call Do at the same time, only one will run the function, while the others will wait until it returns. This way, all calls to Convert are guaranteed to proceed only after the rates map has been filled.

sync.Once is perfect for one-time initialization or cleanup in a concurrent environment. No need to worry about data races!

OnceFunc, OnceValue, OnceValues

Besides the Once type, the sync package also includes three convenience once-functions that you might find useful.

Let's say we have the randomN function that returns a random number:

// randomN returns a random number from 1 to 10.
func randomN() int {
    return 1 + rand.IntN(10)
}

And the initN function sets the n variable to a random number:

n := 0
initN := func() {
    if n != 0 {
        panic("n is already initialized")
    }
    n = randomN()
}

It's clear that calling initN more than once will cause a panic (I'm keeping it simple and not using goroutines here):

for range 10 {
    initN()
}
fmt.Println(n)

panic: n is already initialized

We can fix this by wrapping initN in sync.OnceFunc. It returns a function that makes sure the code runs only once:

initOnce := sync.OnceFunc(initN)

for range 10 {
    initOnce()
}
fmt.Println(n)

sync.OnceValue wraps a function that returns a single value (like our randomN). The first time you call the function, it runs and calculates a value. After that, every time you call it, it just returns the same value from the first call:

initN := sync.OnceValue(randomN)

for range 4 {
    fmt.Print(initN(), " ")
}
fmt.Println()

7 7 7 7

sync.OnceValues does the same thing for a function that returns two values:

initNM := sync.OnceValues(func() (int, int) {
    return randomN(), randomN()
})

for range 4 {
    n, m := initNM()
    fmt.Printf("(%d,%d) ", n, m)
}
fmt.Println()

(4,2) (4,2) (4,2) (4,2)

Here are the signatures of all the once-functions side by side for clarity:

// Calls f only once.
func (o *Once) Do(f func()) {}

// Returns a function that calls f only once.
func OnceFunc(f func()) func() {}

// Returns a function that calls f only once
// and returns the value from that first call.
func OnceValue[T any](f func() T) func() T {}

// Returns a function that calls f only once
// and returns the pair of values from that first call.
func OnceValues[T1, T2 any](f func() (T1, T2)) func() (T1, T2) {}

The functions OnceFunc, OnceValue, and OnceValues are shortcuts for common ways to use the Once type. You can use them if they fit your situation, or use Once directly if they don't.

✎ Exercise: Guess the average

Practice is crucial in turning abstract knowledge into skills, making theory alone insufficient. The full version of the book contains a lot of exercises — that's why I recommend getting it.

If you are okay with just theory for now, let's continue.

sync.Pool

The last tool we'll cover is sync.Pool. It helps reuse memory instead of allocating it every time, which reduces the load on the garbage collector.

Let's say we have a program that:

Allocates 1024 bytes.
Does something with that memory.
Goes back to step 1 and repeats this process many times.

It looks something like this:

func runAlloc() {
    // 4 goroutines, each allocating
    // and freeing 1000 buffers.
    var wg sync.WaitGroup
    for range 4 {
        wg.Go(func() {
            for range 1000 {
                buf := make([]byte, 1024)
                rand.Read(buf)
            }
        })
    }
    wg.Wait()
}

If we run the benchmark:

func BenchmarkAlloc(b *testing.B) {
    for b.Loop() {
        runAlloc()
    }
}

Here's what we'll see:

BenchmarkAlloc-8    219     5392291 ns/op   4096215 B/op    4005 allocs/op

Since we're allocating a new buffer on each loop iteration, we end up with 4000 memory allocations, using a total of 4 MB of memory. Even though the garbage collector eventually frees all this memory, it's quite inefficient. Ideally, we should only need 4 buffers instead of 4000 — one for each goroutine.

That's where sync.Pool comes in handy:

func runPool() {
    // Pool with 1 KB buffers.
    pool := sync.Pool{
        New: func() any {                    // (1)
            // Allocate a 1 KB buffer.
            buf := make([]byte, 1024)
            return &buf
        },
    }

    // 4 goroutines, each allocating
    // and freeing 1000 buffers.
    var wg sync.WaitGroup
    for range 4 {
        wg.Go(func() {
            for range 1000 {
                buf := pool.Get().(*[]byte)  // (2)
                rand.Read(*buf)
                pool.Put(buf)                // (3)
            }
        })
    }
    wg.Wait()
}

pool.Get ➋ takes an item from the pool. If there are no available items, it creates a new one using pool.New ➊ (which we have to define ourselves, since the pool doesn't know anything about the items it creates). pool.Put ➌ returns an item back to the pool.

When the first goroutine calls Get during the first iteration, the pool is empty, so it creates a new buffer using New. In the same way, each of the other goroutines create three more buffers. These four buffers are enough for the whole program.

Let's benchmark:

func BenchmarkPool(b *testing.B) {
    for b.Loop() {
        runPool()
    }
}

BenchmarkPool-8     206     5266199 ns/op      5770 B/op      15 allocs/op
BenchmarkAlloc-8    219     5392291 ns/op   4096215 B/op    4005 allocs/op

The difference in memory usage is clear. Thanks to the pool, the number of allocations has dropped by two orders of magnitude. As a result, the program uses less memory and puts minimal pressure on the garbage collector.

Things to keep in mind:

New should return a pointer, not a value, to reduce memory copying and avoid extra allocations.
The pool has no size limit. If you start 1000 more goroutines that all call Get at the same time, 1000 more buffers will be allocated.
After an item is returned to the pool with Put, you shouldn't use it anymore (since another goroutine might already have taken and started using it).

sync.Pool is a pretty niche tool that isn't used very often. However, if your program works with temporary objects that can be reused (like in our example), it might come in handy.

Keep it up

We've covered some of the lesser-known tools in the sync package — condition variables (sync.Cond), one-time execution (sync.Once), and pools (sync.Pool):

A condition variable notifies one or more waiting goroutines about an event. You can often use a channel instead, and that's usually the better choice.
A one-time execution guarantees that a function runs exactly once, no matter how many goroutines call it at the same time.
A pool lets you reuse temporary objects so you don't have to allocate memory every time.

Don't use these tools just because you know they exist. Rely on common sense.

In the next chapter, we'll talk about atomics (coming soon).

Pre-order for $10 or read online

Expressive tests without testify/assert

Wed, 16 Jul 2025 06:00:00 +0000

Many Go programmers prefer using if-free test assertions to make their tests shorter and easier to read. So, instead of writing if statements with t.Errorf:

func Test(t *testing.T) {
    db := getDB(t)
    db.Str().Set("name", "alice")

    age, err := db.Str().Get("age")
    if !errors.Is(err, redka.ErrNotFound) {
        t.Errorf("got: %v; want: ErrNotFound", err)
    }
    if age != nil {
        t.Errorf("got: %v; want: nil", age)
    }
}

PASS

They would use testify/assert (or its evil twin, testify/require):

func Test(t *testing.T) {
    db := getDB(t)
    db.Str().Set("name", "alice")

    age, err := db.Str().Get("age")
    assert.ErrorIs(t, err, redka.ErrNotFound)
    assert.Nil(t, age)
}

However, I don't think you need testify/assert and its 40 different assertion functions to keep your tests clean. Here's an alternative approach.

The testify package also provides mocks and test suite helpers. We won't talk about these — just about assertions.

Equality • Errors • Other assertions • Source code • Final thoughts

Asserting equality

The most common type of test assertion is checking for equality:

func Test(t *testing.T) {
    db := getDB(t)
    db.Str().Set("name", "alice")

    name, _ := db.Str().Get("name") // skip error checking for now
    if name.String() != "alice" {
        t.Errorf("got: %v; want: alice", name)
    }
}

PASS

Let's write a basic generic assertion helper:

// AssertEqual asserts that got is equal to want.
func AssertEqual[T any](tb testing.TB, got T, want T) {
    tb.Helper()

    // Check if both are nil.
    if isNil(got) && isNil(want) {
        return
    }

    // Fallback to reflective comparison.
    if reflect.DeepEqual(got, want) {
        return
    }

    // No match, report the failure.
    tb.Errorf("got: %#v; want: %#v", got, want)
}

We have to use a helper isNil function, because the compiler doesn't allow us to compare a typed T value with an untyped nil:

// isNil checks if v is nil.
func isNil(v any) bool {
    if v == nil {
        return true
    }

    // A non-nil interface can still hold a nil value,
    // so we must check the underlying value.
    rv := reflect.ValueOf(v)
    switch rv.Kind() {
    case reflect.Chan, reflect.Func, reflect.Interface,
        reflect.Map, reflect.Pointer, reflect.Slice,
        reflect.UnsafePointer:
        return rv.IsNil()
    default:
        return false
    }
}

Now let's use the assertion in our test:

func Test(t *testing.T) {
    db := getDB(t)
    db.Str().Set("name", "alice")

    name, _ := db.Str().Get("name")
    AssertEqual(t, name.String(), "alice")
}

PASS

The parameter order in AssertEqual is (got, want), not (want, got) like it is in testify. It just feels more natural — saying "her name is Alice" instead of "Alice is her name".

Also, unlike testify, our assertion doesn't support custom error messages. When a test fails, you'll end up checking the code anyway, so why bother? The default error message shows what's different, and the line number points to the rest.

AssertEqual is already good enough for all equality checks, which probably make up to 70% of your test assertions. Not bad for a 20-line testify alternative! But we can make it a little better, so let's not miss this chance.

First, types like time.Time and net.IP have an Equal method. We should use this method to make sure the comparison is accurate:

// equaler is an interface for types with an Equal method
// (like time.Time or net.IP).
type equaler[T any] interface {
    Equal(T) bool
}

// areEqual checks if a and b are equal.
func areEqual[T any](a, b T) bool {
    // Check if both are nil.
    if isNil(a) && isNil(b) {
        return true
    }

    // Try to compare using an Equal method.
    if eq, ok := any(a).(equaler[T]); ok {
        return eq.Equal(b)
    }

    // Fallback to reflective comparison.
    return reflect.DeepEqual(a, b)
}

Second, we can make comparing byte slices faster by using bytes.Equal:

// areEqual checks if a and b are equal.
func areEqual[T any](a, b T) bool {
    // ...

    // Special case for byte slices.
    if aBytes, ok := any(a).([]byte); ok {
        bBytes := any(b).([]byte)
        return bytes.Equal(aBytes, bBytes)
    }

    // ...
}

Finally, let's call areEqual from our AssertEqual function:

// AssertEqual asserts that got is equal to want.
func AssertEqual[T any](tb testing.TB, got T, want T) {
    tb.Helper()
    if areEqual(got, want) {
        return
    }
    tb.Errorf("got: %#v; want: %#v", got, want)
}

And test it on some values:

func Test(t *testing.T) {
    // date1 and date2 represent the same point in time,
    date1 := time.Date(2025, 1, 1, 0, 0, 0, 0, time.UTC)
    date2 := time.Date(2025, 1, 1, 5, 0, 0, 0, time.FixedZone("UTC+5", 5*3600))
    AssertEqual(t, date1, date2) // ok

    // b1 and b2 are equal byte slices.
    b1 := []byte("abc")
    b2 := []byte{97, 98, 99}
    AssertEqual(t, b1, b2) // ok

    // m1 and m2 are different maps.
    m1 := map[string]int{"age": 25}
    m2 := map[string]int{"age": 42} // change to 25 to pass
    AssertEqual(t, m1, m2) // fail
}

FAIL: Test (0.00s)
main_test.go:47: got: map[string]int{"age":25}; want: map[string]int{"age":42}

Works like a charm!

Asserting errors

Errors are everywhere in Go, so checking for them is an important part of testing:

func Test(t *testing.T) {
    db := getDB(t)
    db.Str().Set("name", "alice")

    _, err := db.Str().Get("name") // ignore the value for brevity
    if err != nil {
        t.Errorf("unexpected error: %v'", err)
    }
}

PASS

Error checks probably make up to 30% of your test assertions, so let's create a separate function for them.

First we cover the basic cases — expecting no error and expecting an error:

// AssertErr asserts that the got error matches the want.
func AssertErr(tb testing.TB, got error, want error) {
    tb.Helper()

    // Want nil error, but got is not nil.
    // This is a fatal error, so we fail the test immediately.
    if want == nil && got != nil {
        tb.Fatalf("unexpected error: %v", got)
        return
    }

    // Want non-nil error, got nil.
    if want != nil && got == nil {
        tb.Errorf("got: ; want: error")
        return
    }

    // Leave the rest for later.
    return
}

Usually we don't fail the test when an assertion fails, to see all the errors at once instead of hunting them one by one. The "unexpected error" case (want nil, got non-nil) is the only exception: the test terminates immediately because any following assertions probably won't make sense and could cause panics.

Let's see how the assertion works:

func Test(t *testing.T) {
    db := getDB(t)
    db.Str().Set("name", "alice")

    _, err := db.Str().Get("name")
    AssertErr(t, err, nil)
}

PASS

So far, so good. Now let's cover the rest of error checking without introducing separate functions (ErrorIs, ErrorAs, ErrorContains, etc.) like testify does.

If want is an error, we'll use errors.Is to check if the error matches the expected value:

// AssertErr asserts that the got error matches the want.
func AssertErr(tb testing.TB, got error, want any) {
    tb.Helper()

    if want != nil && got == nil {
        tb.Error("got: ; want: error")
        return
    }

    switch w := want.(type) {
    case nil:
        if got != nil {
            tb.Fatalf("unexpected error: %v", got)
        }
    case error:
        if !errors.Is(got, w) {
            tb.Errorf("got: %T(%v); want: %T(%v)", got, got, w, w)
        }
    default:
        tb.Errorf("unsupported want type: %T", want)
    }
}

Usage example:

func Test(t *testing.T) {
    err := &fs.PathError{
        Op: "open",
        Path: "file.txt",
        Err: fs.ErrNotExist,
    }
    AssertErr(t, err, fs.ErrNotExist)
}

PASS

If want is a string, we'll check that the error message contains the expected substring:

// AssertErr asserts that the got error matches the want.
func AssertErr(tb testing.TB, got error, want any) {
    // ...
    switch w := want.(type) {
    case string:
        if !strings.Contains(got.Error(), w) {
            tb.Errorf("got: %q; want: %q", got.Error(), w)
        }
    //...
    }
}

Usage example:

func Test(t *testing.T) {
    _, err := regexp.Compile("he(?o") // invalid
    AssertErr(t, err, "invalid or unsupported")
}

PASS

Finally, if want is a type, we'll use errors.As to check if the error matches the expected type:

// AssertErr asserts that the got error matches the want.
func AssertErr(tb testing.TB, got error, want any) {
    // ...
    switch w := want.(type) {
    case reflect.Type:
        target := reflect.New(w).Interface()
        if !errors.As(got, target) {
            tb.Errorf("got: %T; want: %s", got, w)
        }
    //...
    }
}

Usage example:

func Test(t *testing.T) {
    got := &fs.PathError{
        Op: "open",
        Path: "file.txt",
        Err: fs.ErrNotExist,
    }

    var want *fs.PathError
    AssertErr(t, got, reflect.TypeOf(want))

    // Same thing.
    AssertErr(t, got, reflect.TypeFor[*fs.PathError]())
}

PASS

One last thing: AssertErr doesn't make it easy to check if there was some (non-nil) error without asserting its type or value (like Error in testify). Let's fix this by making the want parameter optional:

// AssertErr asserts that the got error matches any of the wanted values.
func AssertErr(tb testing.TB, got error, wants ...any) {
    tb.Helper()

    // If no wants are given, we expect got to be a non-nil error.
    if len(wants) == 0 {
        if got == nil {
            tb.Error("got: ; want: error")
        }
        return
    }

    // Here we only match against the first want for simplicity.
    // Alternatively, we can (and probably should) check
    // if got matches *any* of the wants.
    want := wants[0]
    // ...
}

Usage example:

func Test(t *testing.T) {
    _, err := regexp.Compile("he(?o") // invalid
    AssertErr(t, err) // want non-nil error
}

PASS

Now AssertErr handles all the cases we need:

Check if there is an error.
Check if there is no error.
Check for a specific error value.
Check for an error of a certain type.
Check if the error message matches what we expect.

And it's still under 40 lines of code. Not bad, right?

Other assertions

AssertEqual and AssertErr probably handle 85-95% of test assertions in a typical Go project. But there's still that tricky 5-15% left.

We may need to check for conditions like these:

func Test(t *testing.T) {
    s := "go is awesome"
    if len(s) < 5 {
        t.Error("too short")
    }

    if !strings.Contains(s, "go") {
        t.Error("too weak")
    }
}

PASS

Technically, we can use AssertEqual. But it looks a bit ugly:

func Test(t *testing.T) {
    s := "go is awesome"
    AssertEqual(t, len(s) >= 5, true)
    AssertEqual(t, strings.Contains(s, "go"), true)
}

PASS

So let's introduce the third and final assertion function — AssertTrue. It's the simplest one of all:

// AssertTrue asserts that got is true.
func AssertTrue(tb testing.TB, got bool) {
    tb.Helper()
    if !got {
        tb.Error("got: false; want: true")
    }
}

Now these assertions look better:

func Test(t *testing.T) {
    s := "go is awesome"
    AssertTrue(t, len(s) >= 5)
    AssertTrue(t, strings.Contains(s, "go"))
}

PASS

Nice!

Source code

Here's the full annotated source code for AssertEqual, AssertErr and AssertTrue:

package testx

import (
	"bytes"
	"errors"
	"reflect"
	"strings"
	"testing"
)

// AssertEqual asserts that got is equal to want.
func AssertEqual[T any](tb testing.TB, got T, want T) {
	tb.Helper()
	if areEqual(got, want) {
		return
	}
	tb.Errorf("got: %#v; want: %#v", got, want)
}

// AssertErr asserts that the got error matches the want.
func AssertErr(tb testing.TB, got error, wants ...any) {
	tb.Helper()

	// If no wants are given, we expect got to be a non-nil error.
	if len(wants) == 0 {
		if got == nil {
			tb.Error("got: ; want: error")
		}
		return
	}

	// We'll only match against the first want for simplicity.
	want := wants[0]

	if want != nil && got == nil {
		tb.Error("got: ; want: error")
		return
	}

	switch w := want.(type) {
	case nil:
		if got != nil {
			tb.Fatalf("unexpected error: %v", got)
		}
	case string:
		if !strings.Contains(got.Error(), w) {
			tb.Errorf("got: %q; want: %q", got.Error(), w)
		}
	case error:
		if !errors.Is(got, w) {
			tb.Errorf("got: %T(%v); want: %T(%v)", got, got, w, w)
		}
	case reflect.Type:
		target := reflect.New(w).Interface()
		if !errors.As(got, target) {
			tb.Errorf("got: %T; want: %s", got, w)
		}
	default:
		tb.Errorf("unsupported want type: %T", want)
	}
}

// AssertTrue asserts that got is true.
func AssertTrue(tb testing.TB, got bool) {
	tb.Helper()
	if !got {
		tb.Error("got: false; want: true")
	}
}

// equaler is an interface for types with an Equal method
// (like time.Time or net.IP).
type equaler[T any] interface {
	Equal(T) bool
}

// areEqual checks if a and b are equal.
func areEqual[T any](a, b T) bool {
	// Check if both are nil.
	if isNil(a) && isNil(b) {
		return true
	}

	// Try to compare using an Equal method.
	if eq, ok := any(a).(equaler[T]); ok {
		return eq.Equal(b)
	}

	// Special case for byte slices.
	if aBytes, ok := any(a).([]byte); ok {
		bBytes := any(b).([]byte)
		return bytes.Equal(aBytes, bBytes)
	}

	// Fallback to reflective comparison.
	return reflect.DeepEqual(a, b)
}

// isNil checks if v is nil.
func isNil(v any) bool {
	if v == nil {
		return true
	}

	// A non-nil interface can still hold a nil value,
	// so we must check the underlying value.
	rv := reflect.ValueOf(v)
	switch rv.Kind() {
	case reflect.Chan, reflect.Func, reflect.Interface, reflect.Map,
		reflect.Pointer, reflect.Slice, reflect.UnsafePointer:
		return rv.IsNil()
	default:
		return false
	}
}

Less than 120 lines of code!

Final thoughts

I don't think we need forty assertion functions to test Go apps. Three (or even two) are enough, as long as they correctly check for equality and handle different error cases.

I find the "assertion trio" — Equal, Err, and True — quite useful in practice. That's why I extracted it into the github.com/nalgeon/be mini-package. If you like the approach described in this article, give it a try!

Redka: Redis re-implemented with SQL

Tue, 15 Jul 2025 06:00:00 +0000

I'm a big fan of Redis. It's such an amazing idea to go beyond the get-set paradigm and provide a convenient API for more complex data structures: maps, sets, lists, streams, bloom filters, etc.

I'm also a big fan of relational databases and their universal language, SQL. They've really stood the test of time and have proven to solve a wide range of problems from the 1970s to today.

So, naturally, one day I decided to combine the two and reimplement Redis using a relational backend — first SQLite, then Postgres. That's how Redka was born.

About Redka • Use cases • Usage example • Performance • Final thoughts

About Redka

Redka is a software written in Go. It comes in two flavors:

Standalone Redis-compatible server.
Go module for in-process use.

               ┌────────────┐   ┌────────────┐
Any   → RESP → │ Redka      │ → │ Postgres   │
client         │ server     │ ← │ or SQLite  │
               └────────────┘   └────────────┘
               ┌────────────┐   ┌────────────┐
Go client    → │ Redka      │ → │ Postgres   │
               │ module     │ ← │ or SQLite  │
               └────────────┘   └────────────┘

Redka currently supports five core Redis data types:

Strings are the most basic type, representing a sequence of bytes.
Lists are sequences of strings sorted by insertion order.
Sets are unordered collections of unique strings.
Sorted sets (zsets) are collections of unique strings ordered by score.
Hashes are field-value maps.

┌─────────────────────────┐  ┌─────────────────────────┐
│          Go API         │  │       RESP server       │
└─────────────────────────┘  └─────────────────────────┘
┌─────────┐ ┌─────────┐ ┌────────┐ ┌───────────────────┐
│ Strings │ │ Lists   │ │ Sets   │ │ Transaction mngmt │
└─────────┘ └─────────┘ └────────┘ └───────────────────┘
┌─────────┐ ┌─────────┐ ┌────────┐ ┌───────────────────┐
│ Keys    │ │ Hashes  │ │ ZSets  │ │ Database adapter  │
└─────────┘ └─────────┘ └────────┘ └───────────────────┘
┌──────────────────────────────────────────────────────┐
│                  Relational database                 │
└──────────────────────────────────────────────────────┘

Redka can use either SQLite or PostgreSQL as its backend. It stores data in a database with a simple schema and provides views for better introspection.

Use cases

Here are some situations where Redka might be helpful:

Embedded cache for Go applications. If your Go app already uses SQLite or just needs a built-in key-value store, Redka is a natural fit. It gives you Redis-like features without the hassle of running a separate server. You're not limited to just get/set with expiration, of course — more advanced structures like lists, maps, and sets are also available.

Lightweight testing environment. Your app uses Redis in production, but setting up a Redis server for local development or integration tests can be a hassle. Redka with an in-memory database offers a fast alternative to test containers, providing full isolation for each test run.

Postgres-first data structures. If you prefer to use PostgreSQL for everything but need Redis-like data structures, Redka can use your existing database as the backend. This way, you can manage both relational data and specialized data structures with the same tools and transactional guarantees.

Usage example

You can run the Redka server the same way you run Redis:

./redka -h localhost -p 6379

Then use redis-cli or any Redis client for your programming language, like redis-py, node-redis, go-redis, and so on:

set name alice
setex age 3600 25
get name
get age

OK
OK
alice
25

You can also use Redka as a Go package without the server:

// Open the database.
db, err := redka.Open("file:/redka.db?vfs=memdb", nil)
if err != nil {
    log.Fatal(err)
}
defer db.Close()

// Set some values.
db.Str().Set("name", "alice")
db.Str().SetExpire("age", 25, time.Hour)

// Read them back.
name, err := db.Str().Get("name")
fmt.Printf("get name = %v, err = %v\n", name, err)
age, err := db.Str().Get("age")
fmt.Printf("get age = %v, err = %v\n", age, err)

set name, err = 
set age, err = 
get name = alice, err= 
get age = 25, err=

All data is stored in the database, so you can access it using SQL views:

select * from vstring;

┌─────┬──────┬───────┬─────────────────────┬─────────────────────┐
│ kid │ key  │ value │        etime        │        mtime        │
├─────┼──────┼───────┼─────────────────────┼─────────────────────┤
│ 1   │ name │ alice │                     │ 2025-07-15 05:26:39 │
│ 2   │ age  │ 25    │ 2025-07-15 06:26:42 │ 2025-07-15 05:26:42 │
└─────┴──────┴───────┴─────────────────────┴─────────────────────┘

Performance

Redka is not about raw performance. You can't beat a specialized data store like Redis with a general-purpose relational backend like SQLite. However, Redka can still handle tens of thousands of operations per second, which should be more than enough for many apps.

Here are the redis-benchmark results for 1,000,000 GET/SET operations on 10,000 randomized keys.

Redis:

SET: 133262.25 requests per second, p50=0.055 msec
GET: 139217.59 requests per second, p50=0.055 msec

Redka (SQLite):

SET: 26773.76  requests per second, p50=0.215 msec
GET: 103092.78 requests per second, p50=0.063 msec

Redka (PostgreSQL):

SET: 11941.72 requests per second, p50=0.775 msec
GET: 25766.55 requests per second, p50=0.359 msec

Final thoughts

Redka for SQLite has been around for over a year, and I recently released a new version that also supports Postgres. If you like the idea of Redis with an SQL backend — feel free to try Redka in testing or (non-critical) production scenarios.

See the nalgeon/redka repo for more details.

Concise test assertions with Be

Tue, 08 Jul 2025 08:00:00 +0000

I appreciate Go's verbosity, but not when it comes to writing tests. Endless if statements with t.Errorf not only make tests long, but also obscure their purpose and make them harder to read. For test assertions, I'd prefer something more concise than what the standard library offers.

There's testify/assert, of course. It's a decent package, but it's quite big. There are about 40 assertion functions, and that's just the tip of the iceberg. Definitely not my thing.

There's also Mat Ryer's is. It's about as minimal as a package can get, but its focus is a bit puzzling to me. Instead of mainly helping with assertions, it seems to concentrate more on parsing comments and formatting the output.

I wanted a small package, about the size of is, with decent assertion features for testing equality and errors. So I made one.

Be — a minimal test assertions package

Be is a simple test assertions package that focuses on equality checks and flexible error checking. Here are a few highlights:

Minimal API: Equal, Err, and True assertions.
Correctly compares time.Time values and other types with an Equal method.
Flexible error assertions: check if an error exists, check its value, type, or any combination of these.

Equal asserts that two values are equal:

func Test(t *testing.T) {
    t.Run("pass", func(t *testing.T) {
        got, want := "hello", "hello"
        be.Equal(t, got, want)
        // ok
    })

    t.Run("fail", func(t *testing.T) {
        got, want := "olleh", "hello"
        be.Equal(t, got, want)
        // got: "olleh"; want: "hello"
    })
}

--- FAIL: Test (0.00s)
    --- FAIL: Test/fail (0.00s)
        main_test.go:29: got: "olleh"; want: "hello"
FAIL

Err asserts that there is an error:

func Test(t *testing.T) {
    _, err := regexp.Compile("he(?o") // invalid
    be.Err(t, err)
    // ok
}

PASS

Or that there are no errors:

func Test(t *testing.T) {
    _, err := regexp.Compile("he??o") // valid
    be.Err(t, err, nil)
    // ok
}

PASS

Or that an error message contains a substring:

func Test(t *testing.T) {
    _, err := regexp.Compile("he(?o") // invalid
    be.Err(t, err, "invalid or unsupported")
    // ok
}

PASS

Or that an error matches the expected error according to errors.Is:

func Test(t *testing.T) {
    err := &fs.PathError{
        Op: "open",
        Path: "file.txt",
        Err: fs.ErrNotExist,
    }
    be.Err(t, err, fs.ErrNotExist)
    // ok
}

PASS

Or that the error type matches the expected type according to errors.As:

func Test(t *testing.T) {
    got := &fs.PathError{
        Op: "open",
        Path: "file.txt",
        Err: fs.ErrNotExist,
    }
    be.Err(t, got, reflect.TypeFor[*fs.PathError]())
    // ok
}

PASS

Or a mix of the above:

func Test(t *testing.T) {
    err := AppError("oops")
    be.Err(t, err,
        "failed",
        AppError("oops"),
        reflect.TypeFor[AppError](),
    )
    // ok
}

PASS

True asserts that an expression is true:

func Test(t *testing.T) {
    s := "go is awesome"
    be.True(t, strings.Contains(s, "go"))
    // ok
}

PASS

That's it!

Design decisions

Be is opinionated. It only has three assert functions, which are perfectly enough to write good tests.

Unlike other testing packages, be doesn't support custom error messages. When a test fails, you'll end up checking the code anyway, so why bother? The line number shows the way.

Be has flexible error assertions. You don't need to choose between Error, ErrorIs, ErrorAs, ErrorContains, NoError, or anything like that — just use be.Err. It covers everything.

Be doesn't fail the test when an assertion fails, so you can see all the errors at once instead of hunting them one by one. The only exception is when the be.Err(err, nil) assertion fails — this means there was an unexpected error. In this case, the test terminates immediately because any following assertions probably won't make sense and could cause panics.

The parameter order is (got, want), not (want, got). It just feels more natural — like saying "account balance is 100 coins" instead of "100 coins is the account balance".

Be currently has ≈100 lines of code (+450 lines for tests). For comparison, is has ≈250 loc (+250 lines for tests).

Final thoughts

If you think the standard library's approach is too wordy, but still want to keep things simple, try using be! You can install it with go get github.com/nalgeon/be and get started right away.

I hope it serves you well.

See the nalgeon/be repo if you are interested.

Gist of Go: Semaphores

Tue, 01 Jul 2025 15:30:00 +0000

This is a chapter from my book on Go concurrency, which teaches the topic from the ground up through interactive examples.

Having the full power of multi-core hardware is great, but sometimes we prefer to limit concurrency in certain parts of a system. Semaphores are a great way to do this. Let's learn more about them!

Mutex • Semaphore • Implementing a semaphore ✎ • Rendezvous • Synchronization barrier • Keep it up

Mutex: one goroutine at a time

Let's say our program needs to call a legacy system, represented by the External type. This system is so ancient that it can handle no more than one call at a time. That's why we protect it with a mutex:

// External is a adapter for an external system.
type External struct {
    lock sync.Mutex
}

// Call calls the external system.
func (e *External) Call() {
    e.lock.Lock()
    defer e.lock.Unlock()
    // Simulate a remote call.
    time.Sleep(10 * time.Millisecond)
}

Now, no matter how many goroutines try to access the external system at the same time, they'll have to take turns:

func main() {
    const nCalls = 12
    ex := new(External)
    start := time.Now()

    var wg sync.WaitGroup
    for range nCalls {
        wg.Go(func() {
            ex.Call()
            fmt.Print(".")
        })
    }
    wg.Wait()

    fmt.Printf(
        "\n%d calls took %d ms\n",
        nCalls, time.Since(start).Milliseconds(),
    )
}

............
12 calls took 120 ms

Suppose the developers of the legacy system made some changes and now they say we can make up to four simultaneous calls. In this case, our approach with the mutex stops working, because it blocks all goroutines except the one that managed to lock the mutex.

It would be great to use a tool that allows several goroutines to run at the same time, but no more than N. Luckily for us, such a tool already exists.

Semaphore: ≤ N goroutines at a time

So, we want to make sure that no more than 4 goroutines access the external system at the same time. To do this, we'll use a semaphore. You can think of a semaphore as a container with N available slots and two operations: acquire to take a slot and release to free a slot.

Here are the semaphore rules:

Calling Acquire takes a free slot.
If there are no free slots, Acquire blocks the goroutine that called it.
Calling Release frees up a previously taken slot.
If there are any goroutines blocked on Acquire when Release is called, one of them will immediately take the freed slot and unblock.

Let's see how this works. To keep things simple, let's assume that someone has already implemented a Semaphore type for us, and we can just use it.

We add a semaphore to the external system adapter:

type External struct {
    sema Semaphore
}

func NewExternal(maxConc int) *External {
    // maxConc sets the maximum allowed
    // number of concurrent calls.
    return &External{NewSemaphore(maxConc)}
}

We acquire a spot in the semaphore before calling the external system. After the call, we release it:

func (e *External) Call() {
    e.sema.Acquire()
    defer e.sema.Release()
    // Simulate a remote call.
    time.Sleep(10 * time.Millisecond)
}

Now let's allow 4 concurrent calls and perform a total of 12 calls. The client code doesn't change:

func main() {
    const nCalls = 12
    const nConc = 4

    ex := NewExternal(nConc)
    start := time.Now()

    var wg sync.WaitGroup
    for range nCalls {
        wg.Go(func() {
            ex.Call()
            fmt.Print(".")
        })
    }
    wg.Wait()

    fmt.Printf(
        "\n%d calls took %d ms\n",
        nCalls, time.Since(start).Milliseconds(),
    )
}

............
12 calls took 30 ms

12 calls were completed in three steps (each step = 4 concurrent calls). Each step took 10 ms, so the total time was 30 ms.

You might have noticed a downside to this approach: even though only 4 goroutines (nConc) run concurrently, we actually start all 12 (nCalls) right away. With small numbers, this isn't a big deal, but if nCalls is large, the waiting goroutines will use up memory for no good reason.

We can modify the program so that there are never more than nConc goroutines at any given time. To do this, we add the Acquire and Release methods directly to External and remove them from Call:

type External struct {
    // Semaphore is embedded into External so clients
    // can call Acquire and Release directly on External.
    Semaphore
}

func NewExternal(maxConc int) *External {
    return &External{NewSemaphore(maxConc)}
}

func (e *External) Call() {
    // Simulate a remote call.
    time.Sleep(10 * time.Millisecond)
}

The client calls Acquire before starting each new goroutine in the loop, and calls Release when it's finished:

func main() {
    const nCalls = 12
    const nConc = 4

    ex := NewExternal(nConc)
    start := time.Now()

    var wg sync.WaitGroup
    for range nCalls {
        ex.Acquire()
        wg.Go(func() {
            defer ex.Release()
            ex.Call()
            fmt.Print(".")
        })
    }
    wg.Wait()

    fmt.Printf(
        "\n%d calls took %d ms\n",
        nCalls, time.Since(start).Milliseconds(),
    )
}

............
12 calls took 30 ms

Now there are never more than 4 goroutines at any time (not counting the main goroutine, of course).

In summary, the semaphore helped us solve the problem of limited concurrency:

goroutines are allowed to run concurrently,
but no more than N at the same time.

Unfortunately, the standard library doesn't include a Semaphore type. So in the next step, we'll implement it ourselves!

There is a semaphore available in the golang.org/x/sync/semaphore package. But for simple cases like ours, it's perfectly fine to use your own implementation.

✎ Implementing a semaphore

The ✎ symbol indicates exercises. They're usually only available in the paid version of the book, but this one is an exception.

Here's a Semaphore type that I'm not really proud of:

// A synchronization semaphore.
type Semaphore struct {
    n   int
    val int
}

// NewSemaphore creates a new semaphore with the given capacity.
func NewSemaphore(n int) *Semaphore {
    return &Semaphore{n: n}
}

// Acquire takes a spot in the semaphore if one is available.
// Otherwise, it blocks the calling goroutine.
func (s *Semaphore) Acquire() {
    if s.val < s.n {
        s.val += 1
        return
    }
    for s.val >= s.n {
    }
}

// Release frees up a spot in the semaphore and unblocks
// one of the blocked goroutines (if there are any).
func (s *Semaphore) Release() {
    s.val -= 1
}

There are a few problems with it:

There's a data race when changing the val counter.
The waiting in Acquire is done with an infinite loop (busy-waiting).
The semaphore doesn't work 😁

Create your own Semaphore that doesn't have these issues.

Hint

Go 1.25 interactive tour

Thu, 26 Jun 2025 11:30:00 +0000

Go 1.25 is out, so it's a good time to explore what's new. The official release notes are pretty dry, so I prepared an interactive version with lots of examples showing what has changed and what the new behavior is.

Read on and see!

synctest • json/v2 • GOMAXPROCS • New GC • Anti-CSRF • WaitGroup.Go • FlightRecorder • os.Root • reflect.TypeAssert • T.Attr • slog.GroupAttrs • hash.Cloner

This article is based on the official release notes from The Go Authors, licensed under the BSD-3-Clause license. This is not an exhaustive list; see the official release notes for that.

I provide links to the proposals (𝗣) and commits (𝗖𝗟) for the features described. Check them out for motivation and implementation details.

Error handling is often skipped to keep things simple. Don't do this in production ツ

# Synthetic time for testing

Suppose we have a function that waits for a value from a channel for one minute, then times out:

// Read reads a value from the input channel and returns it.
// Timeouts after 60 seconds.
func Read(in chan int) (int, error) {
    select {
    case v := <-in:
        return v, nil
    case <-time.After(60 * time.Second):
        return 0, errors.New("timeout")
    }
}

We use it like this:

func main() {
    ch := make(chan int)
    go func() {
        ch <- 42
    }()
    val, err := Read(ch)
    fmt.Printf("val=%v, err=%v\n", val, err)
}

val=42, err=

How do we test the timeout situation? Surely we don't want the test to actually wait 60 seconds. We could make the timeout a parameter (we probably should), but let's say we prefer not to.

The new synctest package to the rescue! The synctest.Test function executes an isolated "bubble". Within the bubble, time package functions use a fake clock, allowing our test to pass instantly:

func TestReadTimeout(t *testing.T) {
    synctest.Test(t, func(t *testing.T) {
        ch := make(chan int)
        _, err := Read(ch)
        if err == nil {
            t.Fatal("expected timeout error, got nil")
        }
    })
}

PASS

The initial time in the bubble is midnight UTC 2000-01-01. Time advances when every goroutine in the bubble is blocked. In our example, when the only goroutine is blocked on select in Read, the bubble's clock advances 60 seconds, triggering the timeout case.

Keep in mind that the t passed to the inner function isn't exactly the usual testing.T. In particular, you should never call T.Run, T.Parallel, or T.Deadline on it:

func TestSubtest(t *testing.T) {
    synctest.Test(t, func(t *testing.T) {
        t.Run("subtest", func (t *testing.T) {
            t.Log("ok")
        })
    })
}

panic: testing: t.Run called inside synctest bubble [recovered, repanicked]

So, no inner tests inside the bubble.

Another useful function is synctest.Wait. It waits for all goroutines in the bubble to block, then resumes execution:

func TestWait(t *testing.T) {
    synctest.Test(t, func(t *testing.T) {
        var innerStarted bool
        done := make(chan struct{})

        go func() {
            innerStarted = true
            time.Sleep(time.Second)
            close(done)
        }()

        // Wait for the inner goroutine to block on time.Sleep.
        synctest.Wait()
        // innerStarted is guaranteed to be true here.
        fmt.Printf("inner started: %v\n", innerStarted)

        <-done
    })
}

inner started: true

Try commenting out the Wait() call and see how the innerStarted value changes.

The testing/synctest package was first introduced as experimental in version 1.24. It's now considered stable and ready to use. Note that the Run function, which was added in 1.24, is now deprecated. You should use Test instead.

𝗣 67434, 73567 • 𝗖𝗟 629735, 629856, 671961

# JSON v2

The second version of the json package is a big update, and it has a lot of breaking changes. That's why I wrote a separate post with a detailed review of what's changed and lots of interactive examples.

Here, I'll just show one of the most impressive features.

With json/v2 you're no longer limited to just one way of marshaling a specific type. Now you can use custom marshalers and unmarshalers whenever you need, with the generic MarshalToFunc and UnmarshalFromFunc functions.

For example, we can marshal boolean values (true/false) and "boolean-like" strings (on/off) to ✓ or ✗ — all without creating a single custom type!

First we define a custom marshaler for bool values:

// Marshals boolean values to ✓ or ✗.
boolMarshaler := json.MarshalToFunc(
    func(enc *jsontext.Encoder, val bool) error {
        if val {
            return enc.WriteToken(jsontext.String("✓"))
        }
        return enc.WriteToken(jsontext.String("✗"))
    },
)

Then we define a custom marshaler for bool-like strings:

// Marshals boolean-like strings to ✓ or ✗.
strMarshaler := json.MarshalToFunc(
    func(enc *jsontext.Encoder, val string) error {
        if val == "on" || val == "true" {
            return enc.WriteToken(jsontext.String("✓"))
        }
        if val == "off" || val == "false" {
            return enc.WriteToken(jsontext.String("✗"))
        }
        // SkipFunc is a special type of error that tells Go to skip
        // the current marshaler and move on to the next one. In our case,
        // the next one will be the default marshaler for strings.
        return json.SkipFunc
    },
)

Finally, we combine marshalers with JoinMarshalers and pass them to the marshaling function using the WithMarshalers option:

// Combine custom marshalers with JoinMarshalers.
marshalers := json.JoinMarshalers(boolMarshaler, strMarshaler)

// Marshal some values.
vals := []any{true, "off", "hello"}
data, err := json.Marshal(vals, json.WithMarshalers(marshalers))
fmt.Println(string(data), err)

["✓","✗","hello"]

Isn't that cool?

There are plenty of other goodies, like support for I/O readers and writers, nested objects inlining, a plethora of options, and a huge performance boost. So, again, I encourage you to check out the post dedicated to v2 changes.

𝗣 63397, 71497

# Container-aware GOMAXPROCS

The GOMAXPROCS runtime setting controls the maximum number of operating system threads the Go scheduler can use to execute goroutines concurrently. Since Go 1.5, it defaults to the value of runtime.NumCPU, which is the number of logical CPUs on the machine (strictly speaking, this is either the total number of logical CPUs or the number allowed by the CPU affinity mask, whichever is lower).

For example, on my 8-core laptop, the default value of GOMAXPROCS is also 8:

maxProcs := runtime.GOMAXPROCS(0) // returns the current value
fmt.Println("NumCPU:", runtime.NumCPU())
fmt.Println("GOMAXPROCS:", maxProcs)

NumCPU: 8
GOMAXPROCS: 8

Go programs often run in containers, like those managed by Docker or Kubernetes. These systems let you limit the CPU resources for a container using a Linux feature called cgroups.

A cgroup (control group) in Linux lets you group processes together and control how much CPU, memory, and network I/O they can use by setting limits and priorities.

For example, here's how you can limit a Docker container to use only four CPUs:

docker run --cpus=4 golang:1.24-alpine go run /app/nproc.go

Before version 1.25, the Go runtime didn't consider the CPU quota when setting the GOMAXPROCS value. No matter how you limited CPU resources, GOMAXPROCS was always set to the number of logical CPUs on the host machine:

docker run --cpus=4 golang:1.24-alpine go run /app/nproc.go

NumCPU: 8
GOMAXPROCS: 8

Now, the Go runtime started to respect the CPU quota:

docker run --cpus=4 golang:1.25-alpine go run /app/nproc.go

NumCPU: 8
GOMAXPROCS: 4

The default value of GOMAXPROCS is now set to either the number of logical CPUs or the CPU limit enforced by cgroup settings for the process, whichever is lower.

Fractional CPU limits are rounded up:

docker run --cpus=2.3 golang:1.25-alpine go run /app/nproc.go

NumCPU: 8
GOMAXPROCS: 3

On a machine with multiple CPUs, the minimum default value for GOMAXPROCS is 2, even if the CPU limit is set lower:

docker run --cpus=1 golang:1.25-alpine go run /app/nproc.go

NumCPU: 8
GOMAXPROCS: 2

The Go runtime automatically updates GOMAXPROCS if the CPU limit changes. The current implementation updates up to once per second (less if the application is idle).

Note on CPU limits

Cgroups actually offer not just one, but two ways to limit CPU resources:

CPU quota — the maximum CPU time the cgroup may use within some period window.
CPU shares — relative CPU priorities given to the kernel scheduler.

Docker's --cpus and --cpu-period/--cpu-quota set the quota, while --cpu-shares sets the shares.

Kubernetes' CPU limit sets the quota, while CPU request sets the shares.

Go's runtime GOMAXPROCS only takes the CPU quota into account, not the shares.

You can set GOMAXPROCS manually using the runtime.GOMAXPROCS function. In this case, the runtime will use the value you provide and won't try to change it:

runtime.GOMAXPROCS(4)
fmt.Println("GOMAXPROCS:", runtime.GOMAXPROCS(0))

GOMAXPROCS: 4

You can also undo any manual changes you made — whether by setting the GOMAXPROCS environment variable or calling runtime.GOMAXPROCS() — and return to the default value chosen by the runtime. To do this, use the new runtime.SetDefaultGOMAXPROCS function:

GOMAXPROCS=2 go run nproc.go

// Using the environment variable.
fmt.Println("GOMAXPROCS:", runtime.GOMAXPROCS(0))

// Using the manual setting.
runtime.GOMAXPROCS(4)
fmt.Println("GOMAXPROCS:", runtime.GOMAXPROCS(0))

// Back to the default value.
runtime.SetDefaultGOMAXPROCS()
fmt.Println("GOMAXPROCS:", runtime.GOMAXPROCS(0))

GOMAXPROCS: 2
GOMAXPROCS: 4
GOMAXPROCS: 8

To provide backward compatibility, the new GOMAXPROCS behavior only takes effect if your program uses Go version 1.25 or higher in your go.mod. You can also turn if off manually using these GODEBUG settings:

containermaxprocs=0 to ignore the cgroups CPU quota.
updatemaxprocs=0 to prevent GOMAXPROCS from updating automatically.

𝗣 73193 • 𝗖𝗟 668638, 670497, 672277, 677037

# Green Tea garbage collector

Some of us Go folks used to joke about Java and its many garbage collectors. Now the joke's on us — there's a new experimental garbage collector coming to Go.

Green Tea is a garbage collecting algorithm optimized for programs that create lots of small objects and run on modern computers with many CPU cores.

The current GC scans memory in a way that jumps around a lot, which is slow because it causes many delays waiting for memory. The problem gets even worse on high-performance systems with many cores and non-uniform memory architectures, where each CPU or group of CPUs has its own "local" RAM.

Green Tea takes a different approach. Instead of scanning individual small objects, it scans memory in much larger, contiguous blocks — spans. Each span contains many small objects of the same size. By working with bigger blocks, the GC can scan more efficiently and make better use of the CPU's memory cache.

Benchmark results vary, but the Go team expects a 10–40% reduction in garbage collection overhead in real-world programs that heavily use the GC.

I ran an informal test by doing 1,000,000 reads and writes to Redka (my Redis clone written in Go), and observed similar GC pause times both the old and new GC algorithms. But Redka probably isn't the best example here because it relies heavily on SQLite, and the Go part is pretty minimal.

The new garbage collector is experimental and can be enabled by setting GOEXPERIMENT=greenteagc at build time. The design and implementation of the GC may change in future releases. For more information or to give feedback, see the proposal.

𝗣 73581 • Feedback

# CSRF protection

The new http.CrossOriginProtection type implements protection against cross-site request forgery (CSRF) by rejecting non-safe cross-origin browser requests.

It detects cross-origin requests in these ways:

By checking the Sec-Fetch-Site header, if it's present.
By comparing the hostname in the Origin header to the one in the Host header.

Here's an example where we enable CrossOriginProtection and explicitly allow some extra origins:

// Register some handlers.
mux := http.NewServeMux()
mux.HandleFunc("GET /get", func(w http.ResponseWriter, req *http.Request) {
    io.WriteString(w, "ok\n")
})
mux.HandleFunc("POST /post", func(w http.ResponseWriter, req *http.Request) {
    io.WriteString(w, "ok\n")
})

// Configure protection against CSRF attacks.
antiCSRF := http.NewCrossOriginProtection()
antiCSRF.AddTrustedOrigin("https://example.com")
antiCSRF.AddTrustedOrigin("https://www.example.com")

// Add CSRF protection to all handlers.
srv := http.Server{
    Addr:    ":8080",
    Handler: antiCSRF.Handler(mux),
}
log.Fatal(srv.ListenAndServe())

Now, if the browser sends a request from the same domain the server is using, the server will allow it:

curl --data "ok" -H "sec-fetch-site:same-origin" localhost:8080/post

ok

If the browser sends a cross-origin request, the server will reject it:

curl --data "ok" -H "sec-fetch-site:cross-site" localhost:8080/post

cross-origin request detected from Sec-Fetch-Site header

If the Origin header doesn't match the Host header, the server will reject the request:

curl --data "ok" \
  -H "origin:https://evil.com" \
  -H "host:antonz.org" \
  localhost:8080/post

cross-origin request detected, and/or browser is out of date:
Sec-Fetch-Site is missing, and Origin does not match Host

If the request comes from a trusted origin, the server will allow it:

curl --data "ok" \
  -H "origin:https://example.com" \
  -H "host:antonz.org" \
  localhost:8080/post

ok

Note that wildcard subdomains are not supported as trusted origins:

// Adding a trusted wildcard subdomain won't work:
antiCSRF.AddTrustedOrigin("https://*.example.com")
// You need to add each subdomain separately like this:
antiCSRF.AddTrustedOrigin("https://s1.example.com")
antiCSRF.AddTrustedOrigin("https://s2.example.com")
antiCSRF.AddTrustedOrigin("https://s2.example.com")

The server will always allow GET, HEAD, and OPTIONS methods because they are safe:

curl -H "origin:https://evil.com" localhost:8080/get

ok

The server will always allow requests without Sec-Fetch-Site or Origin headers (by design):

curl --data "ok" localhost:8080/post

ok

𝗣 73626 • 𝗖𝗟 674936, 680396

# Go wait group, go!

Everyone knows how to run a goroutine with a wait group:

var wg sync.WaitGroup

wg.Add(1)
go func() {
    defer wg.Done()
    fmt.Println("go is awesome")
}()

wg.Add(1)
go func() {
    defer wg.Done()
    fmt.Println("cats are cute")
}()

wg.Wait()
fmt.Println("done")

cats are cute
go is awesome
done

The new WaitGroup.Go method automatically increments the wait group counter, runs a function in a goroutine, and decrements the counter when it's done. This means we can rewrite the example above without using wg.Add() and wg.Done():

var wg sync.WaitGroup

wg.Go(func() {
    fmt.Println("go is awesome")
})

wg.Go(func() {
    fmt.Println("cats are cute")
})

wg.Wait()
fmt.Println("done")

cats are cute
go is awesome
done

The implementation is just what you'd expect:

// https://github.com/golang/go/blob/master/src/sync/waitgroup.go
func (wg *WaitGroup) Go(f func()) {
    wg.Add(1)
    go func() {
        defer wg.Done()
        f()
    }()
}

It's curious that it took the Go team 13 years to add a simple "Add+Done" wrapper. But hey, better late than never!

𝗣 63796 • 𝗖𝗟 662635

# Flight recording

Flight recording is a tracing technique that collects execution data, such as function calls and memory allocations, within a sliding window that's limited by size or duration. It helps to record traces of interesting program behavior, even if you don't know in advance when it will happen.

The new trace.FlightRecorder type implements a flight recorder in Go. It tracks a moving window over the execution trace produced by the runtime, always containing the most recent trace data.

Here's an example of how you might use it.

First, configure the sliding window:

// Configure the flight recorder to keep
// at least 5 seconds of trace data,
// with a maximum buffer size of 3MB.
// Both of these are hints, not strict limits.
cfg := trace.FlightRecorderConfig{
    MinAge:   5 * time.Second,
    MaxBytes: 3 << 20, // 3MB
}

Then create the recorder and start it:

// Create and start the flight recorder.
rec := trace.NewFlightRecorder(cfg)
rec.Start()
defer rec.Stop()

Continue with the application code as usual:

// Simulate some workload.
done := make(chan struct{})
go func() {
    defer close(done)
    const n = 1 << 20
    var s []int
    for range n {
        s = append(s, rand.IntN(n))
    }
    fmt.Printf("done filling slice of %d elements\n", len(s))
}()
<-done

Finally, save the trace snapshot to a file when an important event occurs:

// Save the trace snapshot to a file.
file, _ := os.Create("/tmp/trace.out")
defer file.Close()
n, _ := rec.WriteTo(file)
fmt.Printf("wrote %dB to trace file\n", n)

done filling slice of 1048576 elements
wrote 8441B to trace file

Use the go tool to view the trace in the browser:

go tool trace /tmp/trace.out

𝗣 63185 • 𝗖𝗟 673116

# More Root methods

The os.Root type, which limits filesystem operations to a specific directory, now supports several new methods similar to functions already found in the os package.

Chmod changes the mode of a file:

root, _ := os.OpenRoot("data")
root.Chmod("01.txt", 0600)

finfo, _ := root.Stat("01.txt")
fmt.Println(finfo.Mode().Perm())

-rw-------

Chown changes the numeric user ID (uid) and group ID (gid) of a file:

root, _ := os.OpenRoot("data")
root.Chown("01.txt", 1000, 1000)

finfo, _ := root.Stat("01.txt")
stat := finfo.Sys().(*syscall.Stat_t)
fmt.Printf("uid=%d, gid=%d\n", stat.Uid, stat.Gid)

uid=1000, gid=1000

Chtimes changes the access and modification times of a file:

root, _ := os.OpenRoot("data")
mtime := time.Date(2020, 1, 1, 0, 0, 0, 0, time.UTC)
atime := time.Now()
root.Chtimes("01.txt", atime, mtime)

finfo, _ := root.Stat("01.txt")
fmt.Println(finfo.ModTime())

2020-01-01 00:00:00 +0000 UTC

Link creates a hard link to a file:

root, _ := os.OpenRoot("data")
root.Link("01.txt", "hardlink.txt")

finfo, _ := root.Stat("hardlink.txt")
fmt.Println(finfo.Name())

hardlink.txt

MkdirAll creates a new directory and any parent directories that don't already exist:

const dname = "path/to/secret"

root, _ := os.OpenRoot("data")
root.MkdirAll(dname, 0750)

finfo, _ := root.Stat(dname)
fmt.Println(dname, finfo.Mode())

path/to/secret drwxr-x---

RemoveAll removes a file or a directory and any children that it contains:

root, _ := os.OpenRoot("data")
root.RemoveAll("01.txt")

finfo, err := root.Stat("01.txt")
fmt.Println(finfo, err)

 statat 01.txt: no such file or directory

Rename renames (moves) a file or a directory:

const oldname = "01.txt"
const newname = "go.txt"

root, _ := os.OpenRoot("data")
root.Rename(oldname, newname)

_, err := root.Stat(oldname)
fmt.Println(err)

finfo, _ := root.Stat(newname)
fmt.Println(finfo.Name())

statat 01.txt: no such file or directory
go.txt

Symlink creates a symbolic link to a file. Readlink returns the destination of the symbolic link:

const lname = "symlink.txt"

root, _ := os.OpenRoot("data")
root.Symlink("01.txt", lname)

lpath, _ := root.Readlink(lname)
fmt.Println(lname, "->", lpath)

symlink.txt -> 01.txt

WriteFile writes data to a file, creating it if necessary. ReadFile reads the file and returns its contents:

const fname = "go.txt"

root, _ := os.OpenRoot("data")
root.WriteFile(fname, []byte("go is awesome"), 0644)

content, _ := root.ReadFile(fname)
fmt.Printf("%s: %s\n", fname, content)

go.txt: go is awesome

Since there are now plenty of os.Root methods, you probably don't need the file-related os functions most of the time. This can make working with files much safer.

Speaking of file systems, the ones returned by os.DirFS() (a file system rooted at the given directory) and os.Root.FS() (a file system for the tree of files in the root) both implement the new fs.ReadLinkFS interface. It has two methods — ReadLink and Lstat.

ReadLink returns the destination of the symbolic link:

const lname = "symlink.txt"

root, _ := os.OpenRoot("data")
root.Symlink("01.txt", lname)

fsys := root.FS().(fs.ReadLinkFS)
lpath, _ := fsys.ReadLink(lname)
fmt.Println(lname, "->", lpath)

symlink.txt -> 01.txt

I have to say, the inconsistent naming between os.Root.Readlink and fs.ReadLinkFS.ReadLink is quite surprising.

Lstat returns the information about a file or a symbolic link:

fsys := os.DirFS("data").(fs.ReadLinkFS)
finfo, _ := fsys.Lstat("01.txt")

fmt.Printf("name:  %s\n", finfo.Name())
fmt.Printf("size:  %dB\n", finfo.Size())
fmt.Printf("mode:  %s\n", finfo.Mode())
fmt.Printf("mtime: %s\n", finfo.ModTime().Format(time.DateOnly))

name:  01.txt
size:  11B
mode:  -rw-r--r--
mtime: 2025-06-22

That's it for the os package!

𝗣 49580, 67002, 73126 • 𝗖𝗟 645718, 648295, 649515, 649536, 658995, 659416, 659757, 660635, 661595, 674116, 674315, 676135

# Reflective type assertion

To convert a reflect.Value back to a specific type, you typically use the Value.Interface() method combined with a type assertion:

alice := &Person{"Alice", 25}

// Given a reflection Value...
aliceVal := reflect.ValueOf(alice).Elem()
// ...convert it back to the Person type.
person, _ := aliceVal.Interface().(Person)

fmt.Printf("Name: %s, Age: %d\n", person.Name, person.Age)

Name: Alice, Age: 25

Now you can use the new generic reflect.TypeAssert function instead:

alice := &Person{"Alice", 25}

// Given a reflection Value...
aliceVal := reflect.ValueOf(alice).Elem()
// ...convert it back to the Person type.
person, _ := reflect.TypeAssert[Person](aliceVal)

fmt.Printf("Name: %s, Age: %d\n", person.Name, person.Age)

Name: Alice, Age: 25

It's more idiomatic and avoids unnecessary memory allocations, since the value is never boxed in an interface.

𝗣 62121 • 𝗖𝗟 648056

# Test attributes and friends

With the new T.Attr method, you can add extra test information, like a link to an issue, a test description, or anything else you need to analyze the test results:

func TestAttrs(t *testing.T) {
    t.Attr("issue", "demo-1234")
    t.Attr("description", "Testing for the impossible")

    if 21*2 != 42 {
        t.Fatal("What in the world happened to math?")
    }
}

=== RUN   TestAttrs
=== ATTR  TestAttrs issue demo-1234
=== ATTR  TestAttrs description Testing for the impossible
--- PASS: TestAttrs (0.00s)

Attributes can be especially useful in JSON format if you send the test output to a CI or other system for automatic processing:

go test -json -run=.

...
{
    "Time":"2025-06-25T20:34:16.831401+00:00",
    "Action":"attr",
    "Package":"sandbox",
    "Test":"TestAttrs",
    "Key":"issue",
    "Value":"demo-1234"
}
...
{
    "Time":"2025-06-25T20:34:16.831415+00:00",
    "Action":"attr",
    "Package":"sandbox",
    "Test":"TestAttrs",
    "Key":"description",
    "Value":"Testing for the impossible"
}
...

The output is formatted to make it easier to read.

The same Attr method is also available on testing.B and testing.F.

𝗣 43936 • 𝗖𝗟 662437

The new T.Output method lets you access the output stream (io.Writer) used by the test. This can be helpful if you want to send your application log to the test log stream, making it easier to read or analyze automatically:

func TestLog(t *testing.T) {
    t.Log("test message 1")
    t.Log("test message 2")
    appLog := slog.New(slog.NewTextHandler(t.Output(), nil))
    appLog.Info("app message")
}

=== RUN   TestLog
    main_test.go:12: test message 1
    main_test.go:13: test message 2
    time=2025-06-25T16:14:34.085Z level=INFO msg="app message"
--- PASS: TestLog (0.00s)

The same Output method is also available on testing.B and testing.F.

𝗣 59928 • 𝗖𝗟 672395, 677875

Last but not least, the testing.AllocsPerRun function will now panic if parallel tests are running.

Compare 1.24 behavior:

// go 1.24
func TestAllocs(t *testing.T) {
    t.Parallel()
    allocs := testing.AllocsPerRun(100, func() {
        var s []int
        // Do some allocations.
        for i := range 1024 {
            s = append(s, i)
        }
    })
    t.Log("Allocations per run:", allocs)
}

=== RUN   TestAllocs
=== PAUSE TestAllocs
=== CONT  TestAllocs
    main_test.go:21: Allocations per run: 12
--- PASS: TestAllocs (0.00s)

To 1.25:

// go 1.25
func TestAllocs(t *testing.T) {
    t.Parallel()
    allocs := testing.AllocsPerRun(100, func() {
        var s []int
        // Do some allocations.
        for i := range 1024 {
            s = append(s, i)
        }
    })
    t.Log("Allocations per run:", allocs)
}

=== RUN   TestAllocs
=== PAUSE TestAllocs
=== CONT  TestAllocs
--- FAIL: TestAllocs (0.00s)
panic: testing: AllocsPerRun called during parallel test [recovered, repanicked]

The thing is, the result of AllocsPerRun is inherently flaky if other tests are running in parallel. That's why there's a new panicking behavior — it should help catch these kinds of bugs.

𝗣 70464 • 𝗖𝗟 630137

# Grouped attributes for logging

With structured logging, you often group related attributes under a single key:

logger.Info("deposit",
    slog.Bool("ok", true),
    slog.Group("amount",
        slog.Int("value", 1000),
        slog.String("currency", "USD"),
    ),
)

msg=deposit ok=true amount.value=1000 amount.currency=USD

It works just fine — unless you want to gather the attributes first:

attrs := []slog.Attr{
    slog.Int("value", 1000),
    slog.String("currency", "USD"),
}
logger.Info("deposit",
	slog.Bool("ok", true),
	slog.Group("amount", attrs...),
)

cannot use attrs (variable of type []slog.Attr)
as []any value in argument to slog.Group
(exit status 1)

slog.Group expects a slice of any values, so it doesn't accept a slice of slog.Attrs.

The new slog.GroupAttrs function fixes this issue by creating a group from the given slog.Attrs:

attrs := []slog.Attr{
    slog.Int("value", 1000),
    slog.String("currency", "USD"),
}
logger.Info("deposit",
    slog.Bool("ok", true),
    slog.GroupAttrs("amount", attrs...),
)

msg=deposit ok=true amount.value=1000 amount.currency=USD

Not a big deal, but can be quite handy.

𝗣 66365 • 𝗖𝗟 672915

# Hash cloner

The new hash.Cloner interface defines a hash function that can return a copy of its current state:

// https://github.com/golang/go/blob/master/src/hash/hash.go
type Cloner interface {
    Hash
    Clone() (Cloner, error)
}

All standard library hash.Hash implementations now provide the Clone function, including MD5, SHA-1, SHA-3, FNV-1, CRC-64, and others:

h1 := sha3.New256()
h1.Write([]byte("hello"))

clone, _ := h1.Clone()
h2 := clone.(*sha3.SHA3)

// h2 has the same state as h1, so it will produce
// the same hash after writing the same data.
h1.Write([]byte("world"))
h2.Write([]byte("world"))

fmt.Printf("h1: %x\n", h1.Sum(nil))
fmt.Printf("h2: %x\n", h2.Sum(nil))
fmt.Printf("h1 == h2: %t\n", reflect.DeepEqual(h1, h2))

h1: 92dad9443e4dd6d70a7f11872101ebff87e21798e4fbb26fa4bf590eb440e71b
h2: 92dad9443e4dd6d70a7f11872101ebff87e21798e4fbb26fa4bf590eb440e71b
h1 == h2: true

𝗣 69521 • 𝗖𝗟 675197

# Final thoughts

Go 1.25 finalizes support for testing concurrent code, introduces a major experimental JSON package, and improves the runtime with a new GOMAXPROCS design and garbage collector. It also adds a flight recorder, modern CSRF protection, a long-awaited wait group shortcut, and several other improvements.

All in all, a great release!

P.S. To catch up on other Go releases, check out the Go features by version list or explore the interactive tours for Go 1.24 and 1.23.

P.P.S. Want to learn more about Go? Check out my interactive book on concurrency

JSON evolution in Go: from v1 to v2

Sun, 22 Jun 2025 13:30:00 +0000

The second version of the json package coming in Go 1.25 is a big update, and it has a lot of breaking changes. The v2 package adds new features, fixes API issues and behavioral flaws, and boosts performance. Let's take a look at what's changed!

All examples are interactive, so you can read and practice at the same time.

The basic use case with Marshal and Unmarshal stays the same. This code works in both v1 and v2:

type Person struct {
    Name string
    Age  int
}

alice := Person{Name: "Alice", Age: 25}

// Marshal Alice.
b, err := json.Marshal(alice)
fmt.Println(string(b), err)

// Unmarshal Alice.
err = json.Unmarshal(b, &alice)
fmt.Println(alice, err)

{"Name":"Alice","Age":25} 
{Alice 25}

But the rest is pretty different. Let's go over the main changes from v1.

Write/Read • Encode/Decode • Options • Tags • Custom marshaling • Default behavior • Performance • Final thoughts

MarshalWrite and UnmarshalRead

In v1, you used Encoder to marshal to the io.Writer, and Decoder to unmarshal from the io.Reader:

// Marshal Alice.
alice := Person{Name: "Alice", Age: 25}
out := new(strings.Builder) // io.Writer
enc := json.NewEncoder(out)
enc.Encode(alice)
fmt.Println(out.String())

// Unmarshal Bob.
in := strings.NewReader(`{"Name":"Bob","Age":30}`) // io.Reader
dec := json.NewDecoder(in)
var bob Person
dec.Decode(&bob)
fmt.Println(bob)

{"Name":"Alice","Age":25}

{Bob 30}

From now on, I'll leave out error handling to keep things simple.

In v2, you can use MarshalWrite and UnmarshalRead directly, without any intermediaries:

// Marshal Alice.
alice := Person{Name: "Alice", Age: 25}
out := new(strings.Builder)
json.MarshalWrite(out, alice)
fmt.Println(out.String())

// Unmarshal Bob.
in := strings.NewReader(`{"Name":"Bob","Age":30}`)
var bob Person
json.UnmarshalRead(in, &bob)
fmt.Println(bob)

{"Name":"Alice","Age":25}
{Bob 30 false}

They're not interchangeable, though:

MarshalWrite does not add a newline, unlike the old Encoder.Encode.
UnmarshalRead reads everything from the reader until it hits io.EOF, while the old Decoder.Decode only reads the next JSON value.

MarshalEncode and UnmarshalDecode

The Encoder and Decoder types have been moved to the new jsontext package, and their interfaces have changed significantly (to support low-level streaming encode/decode operations).

You can use them with json functions to read and write JSON streams, similar to how Encode and Decode worked before:

v1 Encoder.Encode → v2 json.MarshalEncode + jsontext.Encoder.
v1 Decoder.Decode → v2 json.UnmarshalDecode + jsontext.Decoder.

Streaming encoder:

people := []Person{
    {Name: "Alice", Age: 25},
    {Name: "Bob", Age: 30},
    {Name: "Cindy", Age: 15},
}
out := new(strings.Builder)
enc := jsontext.NewEncoder(out)

for _, p := range people {
    json.MarshalEncode(enc, p)
}

fmt.Print(out.String())

{"Name":"Alice","Age":25}
{"Name":"Bob","Age":30}
{"Name":"Cindy","Age":15}

Streaming decoder:

in := strings.NewReader(`
    {"Name":"Alice","Age":25}
    {"Name":"Bob","Age":30}
    {"Name":"Cindy","Age":15}
`)
dec := jsontext.NewDecoder(in)

for {
    var p Person
    // Decodes one Person object per call.
    err := json.UnmarshalDecode(dec, &p)
    if err == io.EOF {
        break
    }
    fmt.Println(p)
}

{Alice 25}
{Bob 30}
{Cindy 15}

Unlike UnmarshalRead, UnmarshalDecode works in a fully streaming way, decoding one value at a time with each call, instead of reading everything until io.EOF.

Options

Options configure marshaling and unmarshaling functions with specific features:

FormatNilMapAsNull and FormatNilSliceAsNull define how to encode nil maps and slices.
MatchCaseInsensitiveNames allows matching Name ↔ name and the like.
Multiline expands JSON objects to multiple lines.
OmitZeroStructFields omits fields with zero values from the output.
SpaceAfterColon and SpaceAfterComma add a space after each : or ,
StringifyNumbers represents numeric types as strings.
WithIndent and WithIndentPrefix indent nested properties (note that the MarshalIndent function has been removed).

Each marshaling or unmarshaling function can take any number of options:

alice := Person{Name: "Alice", Age: 25}
b, _ := json.Marshal(
    alice,
    json.OmitZeroStructFields(true),
    json.StringifyNumbers(true),
    jsontext.WithIndent("  "),
)
fmt.Println(string(b))

{
  "Name": "Alice",
  "Age": "25"
}

You can also combine options with JoinOptions:

alice := Person{Name: "Alice", Age: 25}
opts := json.JoinOptions(
    jsontext.SpaceAfterColon(true),
    jsontext.SpaceAfterComma(true),
)
b, _ := json.Marshal(alice, opts)
fmt.Println(string(b))

{"Name": "Alice", "Age": 25}

See the complete list of options in the documentation: some are in the json package, others are in the jsontext package.

Custom marshaling

The basic use case for custom marshaling using Marshaler and Unmarshaler interfaces stays the same. This code works in both v1 and v2:

// A custom boolean type represented
// as "✓" for true and "✗" for false.
type Success bool

func (s Success) MarshalJSON() ([]byte, error) {
    if s {
        return []byte(`"✓"`), nil
    }
    return []byte(`"✗"`), nil
}

func (s *Success) UnmarshalJSON(data []byte) error {
    // Data validation omitted for brevity.
    *s = string(data) == `"✓"`
    return nil
}

func main() {
    // Marshaling.
    val := Success(true)
    data, err := json.Marshal(val)
    fmt.Println(string(data), err)

    // Unmarshaling.
    src := []byte(`"✓"`)
    err = json.Unmarshal(src, &val)
    fmt.Println(val, err)
}

"✓" 
true

However, the Go standard library documentation recommends using the new MarshalerTo and UnmarshalerFrom interfaces instead (they work in a purely streaming way, which can be much faster):

// A custom boolean type represented
// as "✓" for true and "✗" for false.
type Success bool

func (s Success) MarshalJSONTo(enc *jsontext.Encoder) error {
    if s {
        return enc.WriteToken(jsontext.String("✓"))
    }
    return enc.WriteToken(jsontext.String("✗"))
}

func (s *Success) UnmarshalJSONFrom(dec *jsontext.Decoder) error {
    // Data validation omitted for brevity.
    tok, err := dec.ReadToken()
    *s = tok.String() == `✓`
    return err
}

func main() {
    // Marshaling.
    val := Success(true)
    data, err := json.Marshal(val)
    fmt.Println(string(data), err)

    // Unmarshaling.
    src := []byte(`"✓"`)
    err = json.Unmarshal(src, &val)
    fmt.Println(val, err)
}

"✓" 
true

Even better, you're no longer limited to just one way of marshaling a specific type. Now you can use custom marshalers and unmarshalers whenever you need, with the generic MarshalFunc and UnmarshalFunc functions.

func MarshalFunc[T any](fn func(T) ([]byte, error)) *Marshalers
func UnmarshalFunc[T any](fn func([]byte, T) error) *Unmarshalers

For example, you can marshal a bool value to ✓ or ✗ without creating a custom type:

// Custom marshaler for bool values.
boolMarshaler := json.MarshalFunc(
    func(val bool) ([]byte, error) {
        if val {
            return []byte(`"✓"`), nil
        }
        return []byte(`"✗"`), nil
    },
)

// Pass the custom marshaler to Marshal
// using the WithMarshalers option.
val := true
data, err := json.Marshal(val, json.WithMarshalers(boolMarshaler))
fmt.Println(string(data), err)

"✓"

And unmarshal ✓ or ✗ to bool:

// Custom unmarshaler for bool values.
boolUnmarshaler := json.UnmarshalFunc(
    func(data []byte, val *bool) error {
        *val = string(data) == `"✓"`
        return nil
    },
)

// Pass the custom unmarshaler to Unmarshal
// using the WithUnmarshalers option.
src := []byte(`"✓"`)
var val bool
err := json.Unmarshal(src, &val, json.WithUnmarshalers(boolUnmarshaler))
fmt.Println(val, err)

true

There are also MarshalToFunc and UnmarshalFromFunc functions for creating custom marshalers. They're similar to MarshalFunc and UnmarshalFunc, but they work with jsontext.Encoder and jsontext.Decoder instead of byte slices.

func MarshalToFunc[T any](fn func(*jsontext.Encoder, T) error) *Marshalers
func UnmarshalFromFunc[T any](fn func(*jsontext.Decoder, T) error) *Unmarshalers

You can combine marshalers using JoinMarshalers (and unmarshalers using JoinUnmarshalers). For example, here's how you can marshal both booleans (true/false) and "boolean-like" strings (on/off) to ✓ or ✗, while keeping the default marshaling for all other values.

First we define a custom marshaler for bool values:

// Marshals boolean values to ✓ or ✗..
boolMarshaler := json.MarshalToFunc(
    func(enc *jsontext.Encoder, val bool) error {
        if val {
            return enc.WriteToken(jsontext.String("✓"))
        }
        return enc.WriteToken(jsontext.String("✗"))
    },
)

Then we define a custom marshaler for bool-like strings:

// Marshals boolean-like strings to ✓ or ✗.
strMarshaler := json.MarshalToFunc(
    func(enc *jsontext.Encoder, val string) error {
        if val == "on" || val == "true" {
            return enc.WriteToken(jsontext.String("✓"))
        }
        if val == "off" || val == "false" {
            return enc.WriteToken(jsontext.String("✗"))
        }
        // SkipFunc is a special type of error that tells Go to skip
        // the current marshaler and move on to the next one. In our case,
        // the next one will be the default marshaler for strings.
        return json.SkipFunc
    },
)

Finally, we combine marshalers with JoinMarshalers and pass them to the marshaling function using the WithMarshalers option:

// Combine custom marshalers with JoinMarshalers.
marshalers := json.JoinMarshalers(boolMarshaler, strMarshaler)

// Marshal some values.
vals := []any{true, "off", "hello"}
data, err := json.Marshal(vals, json.WithMarshalers(marshalers))
fmt.Println(string(data), err)

["✓","✗","hello"]

Isn't that cool?

Default behavior

v2 changes not only the package interface, but the default marshaling/unmarshaling behavior as well.

Some notable marshaling differences include:

v1 marshals a nil slice to null, v2 marshals to []. You can change it with the FormatNilSliceAsNull option.
v1 marshals a nil map to null, v2 marshals to {}. You can change it with the FormatNilMapAsNull option.
v1 marshals a byte array to an array of numbers, v2 marshals to a base64-encoded string. You can change it with format:array and format:base64 tags.
v1 allows invalid UTF-8 characters within a string, v2 does not. You can change it with the AllowInvalidUTF8 option.

Here's an example of the default v2 behavior:

type Person struct {
    Name    string
    Hobbies []string
    Skills  map[string]int
    Secret  [5]byte
}

func main() {
    alice := Person{
        Name:    "Alice",
        Secret: [5]byte{1, 2, 3, 4, 5},
    }
    b, _ := json.Marshal(alice, jsontext.Multiline(true))
    fmt.Println(string(b))
}

{
    "Name": "Alice",
    "Hobbies": [],
    "Skills": {},
    "Secret": "AQIDBAU="
}

And here's how you can enforce v1 behavior:

type Person struct {
    Name    string
    Hobbies []string
    Skills  map[string]int
    Secret  [5]byte `json:",format:array"`
}

func main() {
    alice := Person{
        Name:    "Alice",
        Secret: [5]byte{1, 2, 3, 4, 5},
    }
    b, _ := json.Marshal(
        alice,
        json.FormatNilMapAsNull(true),
        json.FormatNilSliceAsNull(true),
        jsontext.Multiline(true),
    )
    fmt.Println(string(b))
}

{
    "Name": "Alice",
    "Hobbies": null,
    "Skills": null,
    "Secret": [
        1,
        2,
        3,
        4,
        5
    ]
}

Some notable unmarshaling differences include:

v1 uses case-insensitive field name matching, v2 uses an exact, case-sensitive match. You can change it with the MatchCaseInsensitiveNames option or case tag.
v1 allows duplicate fields in a object, v2 does not. You can change it with the AllowDuplicateNames option.

Here's an example of the default v2 behavior (case-sensitive):

type Person struct {
    FirstName string
    LastName  string
}

func main() {
    src := []byte(`{"firstname":"Alice","lastname":"Zakas"}`)
    var alice Person
    json.Unmarshal(src, &alice)
    fmt.Printf("%+v\n", alice)
}

{FirstName: LastName:}

And here's how you can enforce v1 behavior (case-insensitive):

type Person struct {
    FirstName string
    LastName  string
}

func main() {
    src := []byte(`{"firstname":"Alice","lastname":"Zakas"}`)
    var alice Person
    json.Unmarshal(
        src, &alice,
        json.MatchCaseInsensitiveNames(true),
    )
    fmt.Printf("%+v\n", alice)
}

{FirstName:Alice LastName:Zakas}

See the complete list of behavioral changes in the documentation.

Performance

When marshaling, v2 performs about the same as v1. It's faster with some datasets, but slower with others. However, unmarshaling is much better: v2 is 2.7x to 10.2x faster than v1.

Also, you can get significant performance benefits by switching from regular MarshalJSON and UnmarshalJSON to their streaming alternatives — MarshalJSONTo and UnmarshalJSONFrom. According to the Go team, it allows to convert certain O(n²) runtime scenarios into O(n). For example, switching from UnmarshalJSON to UnmarshalJSONFrom in the k8s OpenAPI spec made it about 40 times faster.

See the jsonbench repo for benchmark details.

Final thoughts

Phew! That's a lot to take in. The v2 package has more features and is more flexible than v1, but it's also a lot more complex, especially given the split into json/v2 and jsontext subpackages.

A couple of things to keep in mind:

As of Go 1.25, the json/v2 package is experimental and can be enabled by setting GOEXPERIMENT=jsonv2 at build time. The package API is subject to change in future releases.
Turning on GOEXPERIMENT=jsonv2 makes the v1 json package use the new JSON implementation, which is faster and supports some options for better compatibility with the old marshaling and unmarshaling behavior.

Finally, here are some links to learn more about the v2 design and implementation:

proposal p.1 • proposal p.2 • json/v2 • jsontext

P.S. Want to learn more about Go? Check out my interactive book on concurrency

Gist of Go: Race conditions

Sun, 15 Jun 2025 10:30:00 +0000

This is a chapter from my book on Go concurrency, which teaches the topic from the ground up through interactive examples.

Preventing data races with mutexes may sound easy, but dealing with race conditions is a whole other matter. Let's learn how to handle these beasts!

Race condition • Compare-and-set • Idempotence and atomicity • Locker • TryLock • Shared nothing • Keep it up

Race condition

Let's say we're keeping track of the money in users' accounts:

// Accounts - money in users' accounts.
type Accounts struct {
    bal map[string]int
    mu  sync.Mutex
}

// NewAccounts creates a new set of accounts.
func NewAccounts(bal map[string]int) *Accounts {
    return &Accounts{bal: maps.Clone(bal)}
}

We can check the balance by username or change the balance:

// Get returns the user's balance.
func (a *Accounts) Get(name string) int {
    a.mu.Lock()
    defer a.mu.Unlock()
    return a.bal[name]
}

// Set changes the user's balance.
func (a *Accounts) Set(name string, amount int) {
    a.mu.Lock()
    defer a.mu.Unlock()
    a.bal[name] = amount
}

Account operations — Get and Set — are concurrent-safe, thanks to the mutex.

There's also a store that sells Lego sets:

// A Lego set.
type LegoSet struct {
    name  string
    price int
}

Alice has 50 coins in her account. She wants to buy two sets: "Castle" for 40 coins and "Plants" for 20 coins:

func main() {
    acc := NewAccounts(map[string]int{
        "alice": 50,
    })
    castle := LegoSet{name: "Castle", price: 40}
    plants := LegoSet{name: "Plants", price: 20}

    var wg sync.WaitGroup

    // Alice buys a castle.
    wg.Go(func() {
        balance := acc.Get("alice")
        if balance < castle.price {
            return
        }
        time.Sleep(5 * time.Millisecond)
        acc.Set("alice", balance-castle.price)
        fmt.Println("Alice bought the castle")
    })

    // Alice buys plants.
    wg.Go(func() {
        balance := acc.Get("alice")
        if balance < plants.price {
            return
        }
        time.Sleep(10 * time.Millisecond)
        acc.Set("alice", balance-plants.price)
        fmt.Println("Alice bought the plants")
    })

    wg.Wait()

    balance := acc.Get("alice")
    fmt.Println("Alice's balance:", balance)
}

Alice bought the castle
Alice bought the plants
Alice's balance: 30

What a twist! Not only did Alice buy both sets for a total of 60 coins (even though she only had 50 coins), but she also ended up with 30 coins left! Great deal for Alice, not so great for us.

The problem is that checking and updating the balance is not an atomic operation:

// body of the second goroutine
balance := acc.Get("alice")             // (1)
if balance < plants.price {             // (2)
    return
}
time.Sleep(10 * time.Millisecond)
acc.Set("alice", balance-plants.price)  // (3)

At point ➊, we see a balance of 50 coins (the first goroutine hasn't done anything yet), so the check at ➋ passes. By point ➌, Alice has already bought the castle (the first goroutine has finished), so her actual balance is 10 coins. But we don't know this and still think her balance is 50 coins. So at point ➌, Alice buys the plants for 20 coins, and the balance becomes 30 coins (the "assumed" balance of 50 coins minus the 20 coins for the plants = 30 coins).

Individual actions on the balance are safe (there's no data race). However, balance reads/writes from different goroutines can get "mixed up", leading to an incorrect final balance. This situation is called a race condition.

You can't fully eliminate uncertainty in a concurrent environment. Events will happen in an unpredictable order — that's just how concurrency works. However, you can protect the system's state — in our case, the purchased sets and balance — so it stays correct no matter what order things happen in.

Let's check and update the balance in one atomic operation, protecting the entire purchase with a mutex. This way, purchases are processed strictly sequentially:

// Shared mutex.
var mu sync.Mutex

// Alice buys a castle.
wg.Go(func() {
    // Protect the entire purchase with a mutex.
    mu.Lock()
    defer mu.Unlock()

    balance := acc.Get("alice")
    if balance < castle.price {
        return
    }
    time.Sleep(5 * time.Millisecond)
    acc.Set("alice", balance-castle.price)
    fmt.Println("Alice bought the castle")
})

// Alice buys plants.
wg.Go(func() {
    // Protect the entire purchase with a mutex.
    mu.Lock()
    defer mu.Unlock()

    balance := acc.Get("alice")
    if balance < plants.price {
        return
    }
    time.Sleep(10 * time.Millisecond)
    acc.Set("alice", balance-plants.price)
    fmt.Println("Alice bought the plants")
})

Alice bought the plants
Alice's balance: 30

One of the goroutines will run first, lock the mutex, check and update the balance, then unlock the mutex. Only after that will the second goroutine be able to lock the mutex and make its purchase.

We still can't be sure which purchase will happen — it depends on the order the goroutines run. But now we are certain that Alice won't buy more than she's supposed to, and the final balance will be correct:

Alice bought the castle
Alice's balance: 10

Or:

Alice bought the plants
Alice's balance: 30

To reiterate:

A data race happens when multiple goroutines access shared data, and at least one of them modifies it. We need to protect the data from this kind of concurrent access.
A race condition happens when an unpredictable order of operations leads to an incorrect system state. In a concurrent environment, we can't control the exact order things happen. Still, we need to make sure that no matter the order, the system always ends up in the correct state.

Go's race detector can find data races, but it doesn't catch race conditions. It's always up to the programmer to prevent race conditions.

Compare-and-set

Let's go back to the situation with the race condition before we added the mutex:

// Alice's balance = 50 coins.
// Castle price = 40 coins.
// Plants price = 20 coins.

// Alice buys a castle.
wg.Go(func() {
    balance := acc.Get("alice")
    if balance < castle.price {
        return
    }
    time.Sleep(5 * time.Millisecond)
    acc.Set("alice", balance-castle.price)
    fmt.Println("Alice bought the castle")
})

// Alice buys plants.
wg.Go(func() {
    balance := acc.Get("alice")
    if balance < plants.price {
        return
    }
    time.Sleep(10 * time.Millisecond)
    acc.Set("alice", balance-plants.price)
    fmt.Println("Alice bought the plants")
})

Alice bought the castle
Alice bought the plants
Alice's balance: 30

As we discussed, the reason for the incorrect final state is that buying a set (checking and updating the balance) is not an atomic operation:

// body of the second goroutine
balance := acc.Get("alice")             // (1)
if balance < plants.price {             // (2)
    return
}
time.Sleep(10 * time.Millisecond)
acc.Set("alice", balance-plants.price)  // (3)

At point ➊, we see a balance of 50 coins, so the check at ➋ passes. By point ➌, Alice has already bought the castle, so her actual balance is 10 coins. But we don't know this and still think her balance is 50 coins. So at point ➌, Alice buys the plants for 20 coins, and the balance becomes 30 coins (the "assumed" balance of 50 coins minus the 20 coins for the plants = 30 coins).

To solve the problem, we can protect the entire purchase with a mutex, just like we did before. But there's another way to handle it.

We can keep two separate operations (checking and updating the balance), but instead of a regular update (set), use an atomic compare-and-set operation:

// Get returns the user's balance.
func (a *Accounts) Get(name string) int {
    a.mu.Lock()
    defer a.mu.Unlock()
    return a.bal[name]
}

// CompareAndSet changes the user's balance to new
// if the current value equals old. Returns false otherwise.
func (a *Accounts) CompareAndSet(name string, old, new int) bool {
    a.mu.Lock()
    defer a.mu.Unlock()
    if a.bal[name] != old {
        return false
    }
    a.bal[name] = new
    return true
}

CompareAndSet first checks if the balance has changed compared to the old value provided by the caller (the "assumed" balance). If the balance has changed (the assumed balance doesn't match the actual balance), it doesn't set the new value and returns false. If the balance hasn't changed (the assumed balance matches the actual balance), it sets the new value and returns true.

Now we can safely sell Lego:

// Alice buys a castle.
wg.Go(func() {
    balance := acc.Get("alice")
    if balance < castle.price {
        return
    }
    time.Sleep(5 * time.Millisecond)
    if acc.CompareAndSet("alice", balance, balance-castle.price) {
        fmt.Println("Alice bought the castle")
    }
})

// Alice buys plants.
wg.Go(func() {
    balance := acc.Get("alice")
    if balance < plants.price {
        return
    }
    time.Sleep(10 * time.Millisecond)
    if acc.CompareAndSet("alice", balance, balance-plants.price) {
        fmt.Println("Alice bought the plants")
    }
})

Alice bought the castle
Alice's balance: 10

We no longer use a mutex to protect the entire purchase. Individual operations from different goroutines can get mixed up, but using compare-and-set instead of a regular update protects us from the race condition. If the actual account state doesn't match what we expect (because another goroutine made a change), the update won't happen.

CAS with retry

In practice, a failed compare-and-set is often followed by a retry. In our example, we would re-read the balance with acc.Get and, if there is still enough money, retried the purchase with acc.CompareAndSet:

// Alice buys plants (with retries).
wg.Go(func() {
    for { // Start of retry loop.
        balance := acc.Get("alice")
        if balance < plants.price {
            return // Not enough money, exit loop.
        }
        // Attempt the purchase.
        if acc.CompareAndSet("alice", balance, balance-plants.price) {
            fmt.Println("Alice bought the plants")
            return // Success, exit loop.
        }
        // Wait a bit before trying again.
        time.Sleep(time.Millisecond)
        // It's also a good idea to limit the number of retries
        // (not shown here for simplicity) - you don't want
        // the program to get stuck in an infinite loop.
    }
})

This approach helps the program handle occasional goroutine conflicts over a shared resource.

Compare-and-set is often used in concurrent programming. It comes in different flavors, such as:

// CompareAndSet changes the value to new if the current value equals old.
// Returns true if the value was changed.
CompareAndSet(old, new any) bool

// CompareAndSwap changes the value to new if the current value equals old.
// Returns the old value.
CompareAndSwap(old, new any) any

// CompareAndDelete deletes the value if the current value equals old.
// Returns true if the value was deleted.
CompareAndDelete(old any) bool

// etc

The idea is always the same:

Check if the assumed (old) state matches reality.
If it does, change the state to new.
If not, do nothing.

Go's standard library provides compare-and-swap operations for basic types like bool and int64. We'll cover these in another chapter.

If compare-and-set doesn't work for your situation, you can always use a regular shared mutex instead. It can protect any sequence of operations, no matter how complex.

✎ Exercise: Concurrent map + 1 more

Practice is crucial in turning abstract knowledge into skills, making theory alone insufficient. The full version of the book contains a lot of exercises — that's why I recommend getting it.

If you are okay with just theory for now, let's continue.

Idempotence and atomicity

Idempotence means that calling an operation on an object multiple times doesn't cause any changes or errors after the first time.

Let's say we have a resource that need to be released after use:

// resource that needs to be freed after use.
type resource struct {
    // ...
}

// free releases the resource.
// Panics if the resource has already been released.
func (r *resource) free() {
    // ...
}

And there's a worker that frees up resources when closed:

// Worker does some work and
// frees resources when closed.
type Worker struct {
    res resource
}

// Work performs some work.
func (w *Worker) Work() {
    // ...
}

// Close frees resources
// and shuts down the worker.
func (w *Worker) Close() {
    w.res.free()
}

Everything works fine until we call Close twice:

func main() {
    w := new(Worker)
    w.Work()
    w.Close()
    w.Close()
    fmt.Println("worker closed")
}

panic: resource is already freed

Let's see how to make Close idempotent so we can safely release the resources.

Boolean flag

Let's add a closed flag to the worker and check it in the Close method:

// Worker does some work and
// frees resources when closed.
type Worker struct {
    res    resource
    closed bool
}

// Close frees resources
// and shuts down the worker.
func (w *Worker) Close() {
    // Ignore repeated calls.
    if w.closed {    // (1)
        return
    }

    w.res.free()     // (2)
    w.closed = true  // (3)
}

Now, calling Close multiple times works fine:

func main() {
    w := new(Worker)
    w.Close()
    w.Close()
    fmt.Println("worker closed")
}

worker closed

But what happens if we call Close simultaneously from different goroutines?

func main() {
    w := new(Worker)

    var wg sync.WaitGroup
    wg.Go(w.Close)
    wg.Go(w.Close)
    wg.Wait()

    fmt.Println("worker closed")
}
// panic: resource is already freed

panic: resource is already freed

You won't see panic here often. But it's still there, lurking in the dark.

Panic! You already know why this happens — the check for w.closed ➊ and the following resource cleanup ➋ ➌ aren't atomic. Since goroutines run concurrently, they both pass the if statement and call w.res.free(). Then, one of the goroutines panics.

We need a structure that ensures atomicity and idempotence in a concurrent environment.

select

Why don't we use a closed channel instead of a boolean flag, and use select to close the channel only once?

// Worker does some work and
// frees resources when closed.
type Worker struct {
    res    resource
    closed chan struct{}
}

// NewWorker creates a new worker.
// We need an explicit NewWorker constructor instead of
// new(Worker), because the default value for a channel
// is nil, which doesn't work for our needs.
func NewWorker() *Worker {
    return &Worker{closed: make(chan struct{})}
}

// Close frees resources
// and shuts down the worker.
func (w *Worker) Close() {
    select {
    case <-w.closed:
        // Ignore repeated calls.
        return
    default:
        w.res.free()
        close(w.closed)
    }
}

worker closed

We call Close from two goroutines and get "worker closed". Everything works fine. Then we deploy this code to production, and a week later we get a bug report saying the app sometimes crashes with the "resource is already freed" panic. What's wrong?

The thing is, select does not protect against freeing resources more than once. As we know, select is safe for concurrent use. So, if two goroutines call Close at the same time, they both enter the select and have to pick a case. Since the closed channel isn't closed yet, both goroutines choose the default case. Both call w.res.free(), and one of them panics.

The chances of this happening are pretty low. You could call Close 999 times without any issues, but on the thousandth try, the stars will align just right, both goroutines will hit the default case, and you will get a panic.

It's especially frustrating that the race detector doesn't always catch this kind of issues. In the example above, the race detector might not find anything (depends on the Go version), especially if you comment out w.res.free(). But the race condition is still there, and closing the channel more than once will eventually cause a panic.

Select is not atomic. Choosing a select case and running its body are separate actions, not a single atomic operation. So, if the code inside the case changes shared data, select can cause a race condition. It's important to keep this in mind.

Mutex

Let's go back to the closed boolean flag, but this time protect it with a mutex:

// Worker does some work and
// frees resources when closed.
type Worker struct {
    res    resource
    mu     sync.Mutex
    closed bool
}

// Close frees resources
// and shuts down the worker.
func (w *Worker) Close() {
    // Make the method atomic.
    w.mu.Lock()
    defer w.mu.Unlock()

    // Ignore repeated calls.
    if w.closed {
        return
    }

    w.res.free()
    w.closed = true
}

worker closed

The mutex stays locked for the entire scope of the Close method. This ensures that no matter how many goroutines call Close simultaneously, only one can execute its body at a time.

Checking the closed flag, freeing resources, and updating the closed state all happen as one atomic operation. This makes Close idempotent, so you won't get any panics when releasing resources.

A mutex (or something similar) is the only way to make sure a complex operation is atomic in a concurrent environment. Do not rely on select in these situations.

Speaking of "something similar", for this specific use case — making sure something happens exactly once — Go's standard library provides a handy sync.Once type. We'll cover it in another chapter.

✎ Exercise: Spot the race

Practice is crucial in turning abstract knowledge into skills, making theory alone insufficient. The full version of the book contains a lot of exercises — that's why I recommend getting it.

If you are okay with just theory for now, let's continue.

Locker

Here are the mutex methods we learned about in the last chapter:

type Mutex struct {
    // internal state
}

func (m *Mutex) Lock()
func (m *Mutex) Unlock()

type RWMutex struct {
    // internal state
}

func (rw *RWMutex) Lock()
func (rw *RWMutex) Unlock()

func (rw *RWMutex) RLock()
func (rw *RWMutex) RUnlock()

As you can see, both types have the same Lock and Unlock methods. Go's standard library provides a common interface for them:

// A Locker represents an object that
// can be locked and unlocked.
type Locker interface {
    Lock()
    Unlock()
}

If the "locking mechanism" is defined by the client, and your code just needs to lock or unlock access to shared data, use Locker instead of a specific type:

// ArrayList is a concurrent-safe dynamic array.
type ArrayList struct {
    vals []any
    lock sync.Locker
}

// NewArrayList creates a new empty array.
func NewArrayList(lock sync.Locker) *ArrayList {
    if lock == nil {
        panic("NewArrayList: lock cannot be nil")
    }
    return &ArrayList{vals: []any{}, lock: lock}
}

// Len returns the length of the array.
func (al *ArrayList) Len() int {
    al.lock.Lock()
    defer al.lock.Unlock()
    return len(al.vals)
}

// Append adds an element to the array.
func (al *ArrayList) Append(val any) {
    al.lock.Lock()
    defer al.lock.Unlock()
    al.vals = append(al.vals, val)
}

This way, the client can use Mutex, RWMutex, or any other implementation they prefer:

func main() {
    var lock sync.Mutex
    list := NewArrayList(&lock)

    // Add 400 elements to the array using 4 goroutines.
    var wg sync.WaitGroup
    for range 4 {
        wg.Go(func() {
            for range 100 {
                list.Append(rand.IntN(100))
                time.Sleep(time.Millisecond)
            }
        })
    }
    wg.Wait()

    fmt.Println("list length =", list.Len())
}

list length = 400

By using sync.Locker, you can build components that don't depend on a specific lock implementation. This lets the client decide which lock to use. While you probably won't need this very often, it's a useful feature to be aware of.

TryLock

// External is a client for an external system.
type External struct {
    lock sync.Mutex
}

// Call calls the external system.
func (e *External) Call() {
    e.lock.Lock()
    defer e.lock.Unlock()
    // Simulate a remote call.
    time.Sleep(100 * time.Millisecond)
}

Now, no matter how many goroutines try to access the external system at the same time, they'll have to take turns:

func main() {
    const nCalls = 4
    ex := new(External)
    start := time.Now()

    var wg sync.WaitGroup
    for range nCalls {
        wg.Go(func() {
            ex.Call()
            fmt.Println("success")
        })
    }
    wg.Wait()

    fmt.Printf(
        "%d calls took %d ms\n",
        nCalls, time.Since(start).Milliseconds(),
    )
}

success
success
success
success
4 calls took 400 ms

Everything looks good on paper. But in reality, if there are a lot of these goroutines, the external system will be constantly busy handling all these sequential calls. This could end up being too much for it. So, let's change the approach:

If a goroutine needs to call an external system,
and that system is already busy,
then the goroutine shouldn't wait,
but should immediately return an error.

We can use the TryLock method of a mutex to implement this logic:

// External is a client for an external system.
type External struct {
	lock sync.Mutex
}

// Call calls the external system.
func (e *External) Call() error {
	if !e.lock.TryLock() {
		return errors.New("busy")  // (1)
	}
	defer e.lock.Unlock()
	// Simulate a remote call.
	time.Sleep(100 * time.Millisecond)
	return nil
}

TryLock tries to lock the mutex, just like a regular Lock. But if it can't, it returns false right away instead of blocking the goroutine. This way, we can immediately return an error at ➊ instead of waiting for the system to become available.

Now, out of four simultaneous calls, only one will go through. The others will get a "busy" error:

func main() {
	const nCalls = 4
    ex := new(External)
	start := time.Now()

	var wg sync.WaitGroup
	for range nCalls {
		wg.Go(func() {
			err := ex.Call()
			if err != nil {
				fmt.Println(err)
			} else {
				fmt.Println("success")
			}
		})
	}
	wg.Wait()

	fmt.Printf(
		"%d calls took %d ms\n",
		nCalls, time.Since(start).Milliseconds(),
	)
}

busy
busy
busy
success
4 calls took 100 ms

According to the standard library docs, TryLock is rarely needed. In fact, using it might mean there's a problem with your program's design. For example, if you're calling TryLock in a busy-wait loop ("keep trying until the resource is free") — that's usually a bad sign:

for {
    if mutex.TryLock() {
        // Use the shared resource.
        mutex.Unlock()
        break
    }
}

This code will keep one CPU core at 100% usage until the mutex is unlocked. It's much better to use a regular Lock so the scheduler can take the blocked goroutine off the CPU.

✎ Exercise: Rate limiter

Practice is crucial in turning abstract knowledge into skills, making theory alone insufficient. The full version of the book contains a lot of exercises — that's why I recommend getting it.

If you are okay with just theory for now, let's continue.

Shared nothing

Let's go back one last time to Alice and the Lego sets we started the chapter with.

We manage user accounts:

// Accounts - money in users' accounts.
type Accounts struct {
	bal map[string]int
	mu  sync.Mutex
}

// NewAccounts creates a new set of accounts.
func NewAccounts(bal map[string]int) *Accounts {
	return &Accounts{bal: maps.Clone(bal)}
}

// Get returns the user's balance.
func (a *Accounts) Get(name string) int {
	a.mu.Lock()          // (1)
	defer a.mu.Unlock()
	return a.bal[name]
}

// Set changes the user's balance.
func (a *Accounts) Set(name string, amount int) {
	a.mu.Lock()          // (2)
	defer a.mu.Unlock()
	a.bal[name] = amount
}

And handle purchases:

acc := NewAccounts(map[string]int{
    "alice": 50,
})
castle := LegoSet{name: "Castle", price: 40}
plants := LegoSet{name: "Plants", price: 20}

// Shared mutex.
var mu sync.Mutex

// Alice buys a castle.
wg.Go(func() {
    // Protect the entire purchase with a mutex.
    mu.Lock()            // (3)
    defer mu.Unlock()

    // Check and update the balance.
})

// Alice buys plants.
wg.Go(func() {
    // Protect the entire purchase with a mutex.
    mu.Lock()            // (4)
    defer mu.Unlock()

    // Check and update the balance.
})

This isn't a very complex use case — I'm sure you've seen worse. Still, we had to put in some effort:

Protect the balance with a mutex to prevent a data race ➊ ➋.
Protect the entire purchase operation with a mutex (or use compare-and-set) to make sure the final state is correct ➌ ➍.

We were lucky to notice and prevent the race condition during a purchase. What if we had missed it?

There's another approach to achieving safe concurrency: instead of protecting shared state when working with multiple goroutines, we can avoid shared state altogether. Channels can help us do this.

Here's the idea: we'll create a Processor function that accepts purchase requests through an input channel, processes them, and sends the results back through an output channel:

// A purchase request.
type Request struct {
    buyer string
    set   LegoSet
}

// A purchase result.
type Purchase struct {
    buyer   string
    set     LegoSet
    succeed bool
    balance int // balance after purchase
}

// Processor handles purchases.
func Processor(acc map[string]int) (chan<- Request, <-chan Purchase) {
    // ...
}

Buyer goroutines will send requests to the processor's input channel and receive results (successful or failed purchases) from the output channel:

func main() {
    const buyer = "Alice"
    acc := map[string]int{buyer: 50}

    wishlist := []LegoSet{
        {name: "Castle", price: 40},
        {name: "Plants", price: 20},
    }

    reqs, purs := Processor(acc)

    // Alice buys stuff.
    var wg sync.WaitGroup
    for _, set := range wishlist {
        wg.Go(func() {
            reqs <- Request{buyer: buyer, set: set}
            pur := <-purs
            if pur.succeed {
                fmt.Printf("%s bought the %s\n", pur.buyer, pur.set.name)
                fmt.Printf("%s's balance: %d\n", buyer, pur.balance)
            }
        })
    }
    wg.Wait()
}

Alice bought the Plants
Alice's balance: 30

This approach offers several benefits:

Buyer goroutines send their requests and get results without worrying about how the purchase is done.
All the buying logic is handled inside the processor goroutine.
No need for mutexes.

All that's left is to implement the processor. How about this:

// Processor handles purchases.
func Processor(acc map[string]int) (chan<- Request, <-chan Purchase) {
    in := make(chan Request)
    out := make(chan Purchase)
    acc = maps.Clone(acc)

    go func() {
        for {
            // Receive the purchase request.
            req := <-in

            // Handle the purchase.
            balance := acc[req.buyer]
            pur := Purchase{buyer: req.buyer, set: req.set, balance: balance}
            if balance >= req.set.price {
                pur.balance -= req.set.price
                pur.succeed = true
                acc[req.buyer] = pur.balance
            } else {
                pur.succeed = false
            }

            // Send the result.
            out <- pur
        }
    }()

    return in, out
}

It would have been a good idea to add a way to stop the processor using context, but I decided not to do it to keep the code simple.

The processor clones the original account states and works with its own copy. This approach makes sure there is no concurrent access to the accounts, so there are no races. Of course, we should avoid running two processors at the same time, or we could end up with two different versions of the truth.

It's not always easy to structure a program in a way that avoids shared state. But if you can, it's a good option.

Keep it up

Now you know how to protect shared data (from data races) and sequences of operations (from race conditions) in a concurrent environment using mutexes. Be careful with them and always test your code thoroughly with the race detector enabled.

Use code reviews, because the race detector doesn't catch every data race and can't detect race conditions at all. Having someone else look over your code can be really helpful.

In the next chapter, we'll talk about semaphores.

Pre-order for $10 or read online

Gist of Go: Data races

Sun, 08 Jun 2025 17:00:00 +0000

This is a chapter from my book on Go concurrency, which teaches the topic from the ground up through interactive examples.

What happens if multiple goroutines modify the same data structure? Sadly, nothing good. Let's learn more about it.

Concurrent modification
Data race
Sequential modification
Mutex

Read-write mutex
Channel as mutex
Keep it up

Concurrent modification

So far, our goroutines haven't gotten in each other's way. They've used channels to exchange data, which is safe. But what happens if several goroutines try to access the same object at the same time? Let's find out.

Let's write a program that counts word frequencies:

func main() {
    // generate creates 100 words, each 3 letters long,
    // and sends them to the channel.
    in := generate(100, 3)

    var wg sync.WaitGroup
    wg.Add(2)

    // count reads words from the input channel
    // and counts how often each one appears.
    count := func(counter map[string]int) {
        defer wg.Done()
        for word := range in {
            counter[word]++
        }
    }

    counter := map[string]int{}
    go count(counter)
    go count(counter)
    wg.Wait()

    fmt.Println(counter)
}

fatal error: concurrent map writes

goroutine 1 [sync.WaitGroup.Wait]:
sync.runtime_SemacquireWaitGroup(0x140000021c0?)

goroutine 34 [chan send]:
main.generate.func1()

goroutine 35 [running]:
internal/runtime/maps.fatal({0x104b4039e?, 0x14000038a08?})

goroutine 36 [runnable]:
internal/runtime/maps.newTable(0x104b78340, 0x80, 0x0, 0x0)

What is generate