Anton ZhiyanovEverything about Go, SQL, and software in general.https://antonz.org/https://antonz.org/assets/favicon/favicon.pngAnton Zhiyanovhttps://antonz.org/Hugo -- gohugo.ioen-usTue, 01 Jul 2025 15:30:00 +0000Gist of Go: Semaphoreshttps://antonz.org/go-concurrency/semaphores/Tue, 01 Jul 2025 15:30:00 +0000https://antonz.org/go-concurrency/semaphores/Limiting the concurrency and waiting for the peers.This is a chapter from my book on Go concurrency, which teaches the topic from the ground up through interactive examples.

Having the full power of multi-core hardware is great, but sometimes we prefer to limit concurrency in certain parts of a system. Semaphores are a great way to do this. Let's learn more about them!

MutexSemaphoreImplementing a semaphore ✎ • RendezvousSynchronization barrierKeep it up

Mutex: one goroutine at a time

Let's say our program needs to call a legacy system, represented by the External type. This system is so ancient that it can handle no more than one call at a time. That's why we protect it with a mutex:

// External is a adapter for an external system.
type External struct {
    lock sync.Mutex
}

// Call calls the external system.
func (e *External) Call() {
    e.lock.Lock()
    defer e.lock.Unlock()
    // Simulate a remote call.
    time.Sleep(10 * time.Millisecond)
}

Now, no matter how many goroutines try to access the external system at the same time, they'll have to take turns:

func main() {
    var wg sync.WaitGroup
    start := time.Now()

    ex := new(External)

    const nCalls = 12
    for range nCalls {
        wg.Add(1)
        go func() {
            defer wg.Done()
            ex.Call()
            fmt.Print(".")
        }()
    }

    wg.Wait()
    fmt.Printf(
        "\n%d calls took %d ms\n",
        nCalls, time.Since(start).Milliseconds(),
    )
}
............
12 calls took 120 ms

Suppose the developers of the legacy system made some changes and now they say we can make up to four simultaneous calls. In this case, our approach with the mutex stops working, because it blocks all goroutines except the one that managed to lock the mutex.

It would be great to use a tool that allows several goroutines to run at the same time, but no more than N. Luckily for us, such a tool already exists.

Semaphore: ≤ N goroutines at a time

So, we want to make sure that no more than 4 goroutines access the external system at the same time. To do this, we'll use a semaphore. You can think of a semaphore as a container with N available slots and two operations: acquire to take a slot and release to free a slot.

Here are the semaphore rules:

  • Calling Acquire takes a free slot.
  • If there are no free slots, Acquire blocks the goroutine that called it.
  • Calling Release frees up a previously taken slot.
  • If there are any goroutines blocked on Acquire when Release is called, one of them will immediately take the freed slot and unblock.

Let's see how this works. To keep things simple, let's assume that someone has already implemented a Semaphore type for us, and we can just use it.

We add a semaphore to the external system adapter:

type External struct {
    sema Semaphore
}

func NewExternal(maxConc int) *External {
    // maxConc sets the maximum allowed
    // number of concurrent calls.
    return &External{NewSemaphore(maxConc)}
}

We acquire a spot in the semaphore before calling the external system. After the call, we release it:

func (e *External) Call() {
    e.sema.Acquire()
    defer e.sema.Release()
    // Simulate a remote call.
    time.Sleep(10 * time.Millisecond)
}

Now let's allow 4 concurrent calls and perform a total of 12 calls. The client code doesn't change:

func main() {
    var wg sync.WaitGroup
    start := time.Now()

    const nConc = 4
    ex := NewExternal(nConc)

    const nCalls = 12
    for range nCalls {
        wg.Add(1)
        go func() {
            defer wg.Done()
            ex.Call()
            fmt.Print(".")
        }()
    }

    wg.Wait()
    fmt.Printf(
        "\n%d calls took %d ms\n",
        nCalls, time.Since(start).Milliseconds(),
    )
}
............
12 calls took 30 ms

12 calls were completed in three steps (each step = 4 concurrent calls). Each step took 10 ms, so the total time was 30 ms.

You might have noticed a downside to this approach: even though only 4 goroutines (nConc) run concurrently, we actually start all 12 (nCalls) right away. With small numbers, this isn't a big deal, but if nCalls is large, the waiting goroutines will use up memory for no good reason.

We can modify the program so that there are never more than nConc goroutines at any given time. To do this, we add the Acquire and Release methods directly to External and remove them from Call:

type External struct {
    // Semaphore is embedded into External so clients
    // can call Acquire and Release directly on External.
    Semaphore
}

func NewExternal(maxConc int) *External {
    return &External{NewSemaphore(maxConc)}
}

func (e *External) Call() {
    // Simulate a remote call.
    time.Sleep(10 * time.Millisecond)
}

The client calls Acquire before starting each new goroutine in the loop, and calls Release when it's finished:

func main() {
    var wg sync.WaitGroup
    start := time.Now()

    const nConc = 4
    ex := NewExternal(nConc)

    const nCalls = 12
    for range nCalls {
        wg.Add(1)
        ex.Acquire()
        go func() {
            defer wg.Done()
            defer ex.Release()
            ex.Call()
            fmt.Print(".")
        }()
    }

    wg.Wait()
    fmt.Printf(
        "\n%d calls took %d ms\n",
        nCalls, time.Since(start).Milliseconds(),
    )
}
............
12 calls took 30 ms

Now there are never more than 4 goroutines at any time (not counting the main goroutine, of course).

In summary, the semaphore helped us solve the problem of limited concurrency:

  • goroutines are allowed to run concurrently,
  • but no more than N at the same time.

Unfortunately, the standard library doesn't include a Semaphore type. So in the next step, we'll implement it ourselves!

There is a semaphore available in the golang.org/x/sync/semaphore package. But for simple cases like ours, it's perfectly fine to use your own implementation.

✎ Implementing a semaphore

The ✎ symbol indicates exercises. They're usually only available in the paid version of the book, but this one is an exception.

Here's a Semaphore type that I'm not really proud of:

// A synchronization semaphore.
type Semaphore struct {
    n   int
    val int
}

// NewSemaphore creates a new semaphore with the given capacity.
func NewSemaphore(n int) *Semaphore {
    return &Semaphore{n: n}
}

// Acquire takes a spot in the semaphore if one is available.
// Otherwise, it blocks the calling goroutine.
func (s *Semaphore) Acquire() {
    if s.val < s.n {
        s.val += 1
        return
    }
    for s.val >= s.n {
    }
}

// Release frees up a spot in the semaphore and unblocks
// one of the blocked goroutines (if there are any).
func (s *Semaphore) Release() {
    s.val -= 1
}

There are a few problems with it:

  • There's a data race when changing the val counter.
  • The waiting in Acquire is done with an infinite loop (busy-waiting).
  • The semaphore doesn't work 😁

Create your own Semaphore that doesn't have these issues.

Hint

Use a channel. The solution will be very straightforward.

Submit only the code fragment marked with "solution start" and "solution end" comments. The full source code is available via the "Playground" link below.

// solution start

// A synchronization semaphore.
type Semaphore

// NewSemaphore creates a new semaphore with the given capacity.
func NewSemaphore(n int) *Semaphore {
    // ...
}

// Acquire takes a spot in the semaphore if one is available.
// Otherwise, it blocks the calling goroutine.
func (s *Semaphore) Acquire() {
    // ...
}

// Release frees up a spot in the semaphore and unblocks
// one of the blocked goroutines (if there are any).
func (s *Semaphore) Release() {
    // ...
}

// solution end

✎ Exercise: Implementing TryAcquire

Practice is crucial in turning abstract knowledge into skills, making theory alone insufficient. The full version of the book contains a lot of exercises — that's why I recommend getting it.

If you are okay with just theory for now, let's continue.

Rendezvous

Imagine you and your friend agreed to meet at a certain place before going to a cafe together. You arrive at the meeting spot, but your friend isn't there yet. As planned, you don't go anywhere; you just wait for them, and then you both proceed to drink tea (or whatever you planned) together. If your friend had arrived first, they would have waited for you the same way before heading to the cafe.

This kind of logic in concurrent programs is called a rendezvous:

  • There are two goroutines — G1 and G2 — and each one can signal that it's ready.
  • If G1 signals but G2 hasn't yet, G1 blocks and waits.
  • If G2 signals but G1 hasn't yet, G2 blocks and waits.
  • When both have signaled, they both unblock and continue running.

Let's say there are two goroutines, and each one performs two steps. After the first step, each goroutine wants to wait for the other. Here's what their execution looks like without a rendezvous:

var wg sync.WaitGroup
wg.Add(2)

go func() {
    fmt.Println("1: started")
    time.Sleep(10 * time.Millisecond)
    fmt.Println("1: reached the sync point")

    // Sync point: how do I wait for the second goroutine?

    fmt.Println("1: going further")
    time.Sleep(20 * time.Millisecond)
    fmt.Println("1: done")
    wg.Done()
}()

time.Sleep(20 * time.Millisecond)

go func() {
    fmt.Println("2: started")
    time.Sleep(20 * time.Millisecond)
    fmt.Println("2: reached the sync point")

    // Sync point: how do I wait for the first goroutine?

    fmt.Println("2: going further")
    time.Sleep(10 * time.Millisecond)
    fmt.Println("2: done")
    wg.Done()
}()

wg.Wait()
1: started
1: reached the sync point
1: going further
2: started
1: done
2: reached the sync point
2: going further
2: done

As you can see, the second goroutine is just getting started, while the first one is already finished. No one is waiting for anyone else.

Let's set up a rendezvous for them:

var wg sync.WaitGroup
wg.Add(2)

ready1 := make(chan struct{})
ready2 := make(chan struct{})

go func() {
    fmt.Println("1: started")
    time.Sleep(10 * time.Millisecond)
    fmt.Println("1: reached the sync point")

    // Sync point.
    close(ready1)
    <-ready2

    fmt.Println("1: going further")
    time.Sleep(20 * time.Millisecond)
    fmt.Println("1: done")
    wg.Done()
}()

time.Sleep(20 * time.Millisecond)

go func() {
    fmt.Println("2: started")
    time.Sleep(20 * time.Millisecond)
    fmt.Println("2: reached the sync point")

    // Sync point.
    close(ready2)
    <-ready1

    fmt.Println("2: going further")
    time.Sleep(10 * time.Millisecond)
    fmt.Println("2: done")
    wg.Done()
}()

wg.Wait()
1: started
1: reached the sync point
2: started
2: reached the sync point
2: going further
1: going further
2: done
1: done

Now everything works fine: the goroutines waited for each other at the sync point before moving on.

Here's how it works:

  • G1 closes its own channel when it's ready and then blocks on the other channel. G2 does the same thing.
  • When G1's channel is closed, it unblocks G2, and when G2's channel is closed, it unblocks G1.
  • As a result, both goroutines are unblocked at the same time.

Here, we close the channel to signal an event. We've done this before:

  • With a done channel in the Channels chapter (the goroutine signals the caller that the work is finished).
  • With a cancel channel in the Pipelines chapter (the caller signals the goroutine to stop working).

Caution: Using Print for debugging

Since printing uses a single output device (stdout), goroutines that print concurrently have to synchronize access to it. So, using print statements adds a synchronization point to your program. This can cause unexpected results that are different from what actually happens in production (where there is no printing).

In the book, I use print statements only because it's much harder to understand the material without them.

✎ Exercise: Implementing a rendezvous

Practice is crucial in turning abstract knowledge into skills, making theory alone insufficient. The full version of the book contains a lot of exercises — that's why I recommend getting it.

If you are okay with just theory for now, let's continue.

Synchronization barrier

Imagine you walk up to a crosswalk with a traffic light and see a button that's supposed to turn the light green. You press the button, but nothing happens. Another person comes up behind you and presses the button too, but still nothing changes. Two more people arrive, both press the button, but the light stays red. Now the four of you are just standing there, not sure what to do. Finally, a fifth person comes, presses the button, and the light turns green. All five of you cross the street together.

This kind of logic in concurrent programs is called a synchronization barrier:

  • The barrier has a counter (starting at 0) and a threshold N.
  • Each goroutine that reaches the barrier increases the counter by 1.
  • The barrier blocks any goroutine that reaches it.
  • Once the counter reaches N, the barrier unblocks all waiting goroutines.

Let's say there are N goroutines. Each one first does a preparation step, then the main step. Here's what their execution looks like without a barrier:

const nWorkers = 4
start := time.Now()

var wg sync.WaitGroup
wg.Add(nWorkers)

for i := range nWorkers {
    go func() {
        // Simulate the preparation step.
        dur := time.Duration((i+1)*10) * time.Millisecond
        time.Sleep(dur)
        fmt.Printf("ready to go after %d ms\n", dur.Milliseconds())

        // Simulate the main step.
        fmt.Println("go!")
        wg.Done()
    }()
}

wg.Wait()
fmt.Printf("all done in %d ms\n", time.Since(start).Milliseconds())
ready to go after 10 ms
go!
ready to go after 20 ms
go!
ready to go after 30 ms
go!
ready to go after 40 ms
go!
all done in 40 ms

Each goroutine proceeds to the main step as soon as it's ready, without waiting for the others.

Let's say we want the goroutines to wait for each other before moving on to the main step. To do this, we just need to add a barrier after the preparation step:

const nWorkers = 4
start := time.Now()

var wg sync.WaitGroup
wg.Add(nWorkers)

b := NewBarrier(nWorkers)
for i := range nWorkers {
    go func() {
        // Simulate the preparation step.
        dur := time.Duration((i+1)*10) * time.Millisecond
        time.Sleep(dur)
        fmt.Printf("ready to go after %d ms\n", dur.Milliseconds())

        // Wait for all goroutines to reach the barrier.
        b.Touch()

        // Simulate the main step.
        fmt.Println("go!")
        wg.Done()
    }()
}

wg.Wait()
fmt.Printf("all done in %d ms\n", time.Since(start).Milliseconds())
ready to go after 10 ms
ready to go after 20 ms
ready to go after 30 ms
ready to go after 40 ms
go!
go!
go!
go!
all done in 40 ms

Now the faster goroutines waited at the barrier for the slower ones, and only after that did they all move on to the main step together.

Here are some examples of when a synchronization barrier can be useful:

  • Parallel computing. If you're sorting in parallel and then merging the results, the sorting steps must finish before the merging starts. If you merge too soon, you'll get the wrong results.
  • Multiplayer applications. If a duel in the game involves N players, all resources for those players need to be fully prepared before the duel begins. Otherwise, some players might be at a disadvantage.
  • Distributed systems. To create a backup, you need to wait until all nodes in the system reach a consistent state (a checkpoint). Otherwise, the backup's integrity could be compromised.

The standard library doesn't have a barrier, so now is a great time to make one yourself!

✎ Exercise: Implementing a barrier

Practice is crucial in turning abstract knowledge into skills, making theory alone insufficient. The full version of the book contains a lot of exercises — that's why I recommend getting it.

If you are okay with just theory for now, let's continue.

Keep it up

You've learned the classic synchronization tools — mutexes, semaphores, rendezvous, and barriers. Be careful when using them. Try to avoid complicated setups, and always write tests for tricky concurrent situations.

In the next chapter, we'll talk about signaling (coming soon).

Pre-order for $10   or read online

]]>
Go 1.25 interactive tourhttps://antonz.org/go-1-25/Thu, 26 Jun 2025 11:30:00 +0000https://antonz.org/go-1-25/Fake clock, new GC, flight recorder and more.Go 1.25 is scheduled for release in August, so it's a good time to explore what's new. The official release notes are pretty dry, so I prepared an interactive version with lots of examples showing what has changed and what the new behavior is.

Read on and see!

synctestjson/v2GOMAXPROCSNew GCAnti-CSRFWaitGroup.GoFlightRecorderos.Rootreflect.TypeAssertT.Attrslog.GroupAttrshash.Cloner

This article is based on the official release notes from The Go Authors, licensed under the BSD-3-Clause license. This is not an exhaustive list; see the official release notes for that.

I provide links to the proposals (𝗣) and commits (𝗖𝗟) for the features described. Check them out for motivation and implementation details.

Error handling is often skipped to keep things simple. Don't do this in production ツ

# Synthetic time for testing

Suppose we have a function that waits for a value from a channel for one minute, then times out:

// Read reads a value from the input channel and returns it.
// Timeouts after 60 seconds.
func Read(in chan int) (int, error) {
    select {
    case v := <-in:
        return v, nil
    case <-time.After(60 * time.Second):
        return 0, errors.New("timeout")
    }
}

We use it like this:

func main() {
    ch := make(chan int)
    go func() {
        ch <- 42
    }()
    val, err := Read(ch)
    fmt.Printf("val=%v, err=%v\n", val, err)
}
val=42, err=<nil>

How do we test the timeout situation? Surely we don't want the test to actually wait 60 seconds. We could make the timeout a parameter (we probably should), but let's say we prefer not to.

The new synctest package to the rescue! The synctest.Test function executes an isolated "bubble". Within the bubble, time package functions use a fake clock, allowing our test to pass instantly:

func TestReadTimeout(t *testing.T) {
    synctest.Test(t, func(t *testing.T) {
        ch := make(chan int)
        _, err := Read(ch)
        if err == nil {
            t.Fatal("expected timeout error, got nil")
        }
    })
}
PASS

The initial time in the bubble is midnight UTC 2000-01-01. Time advances when every goroutine in the bubble is blocked. In our example, when the only goroutine is blocked on select in Read, the bubble's clock advances 60 seconds, triggering the timeout case.

Keep in mind that the t passed to the inner function isn't exactly the usual testing.T. In particular, you should never call T.Run, T.Parallel, or T.Deadline on it:

func TestSubtest(t *testing.T) {
    synctest.Test(t, func(t *testing.T) {
        t.Run("subtest", func (t *testing.T) {
            t.Log("ok")
        })
    })
}
panic: testing: t.Run called inside synctest bubble [recovered, repanicked]

So, no inner tests inside the bubble.

Another useful function is synctest.Wait. It waits for all goroutines in the bubble to block, then resumes execution:

func TestWait(t *testing.T) {
    synctest.Test(t, func(t *testing.T) {
        var innerStarted bool
        done := make(chan struct{})

        go func() {
            innerStarted = true
            time.Sleep(time.Second)
            close(done)
        }()

        // Wait for the inner goroutine to block on time.Sleep.
        synctest.Wait()
        // innerStarted is guaranteed to be true here.
        fmt.Printf("inner started: %v\n", innerStarted)

        <-done
    })
}
inner started: true

Try commenting out the Wait() call and see how the innerStarted value changes.

The testing/synctest package was first introduced as experimental in version 1.24. It's now considered stable and ready to use. Note that the Run function, which was added in 1.24, is now deprecated. You should use Test instead.

𝗣 67434, 73567 • 𝗖𝗟 629735, 629856, 671961

# JSON v2

The second version of the json package is a big update, and it has a lot of breaking changes. That's why I wrote a separate post with a detailed review of what's changed and lots of interactive examples.

Here, I'll just show one of the most impressive features.

With json/v2 you're no longer limited to just one way of marshaling a specific type. Now you can use custom marshalers and unmarshalers whenever you need, with the generic MarshalToFunc and UnmarshalFromFunc functions.

For example, we can marshal boolean values (true/false) and "boolean-like" strings (on/off) to or — all without creating a single custom type!

First we define a custom marshaler for bool values:

// Marshals boolean values to ✓ or ✗.
boolMarshaler := json.MarshalToFunc(
    func(enc *jsontext.Encoder, val bool) error {
        if val {
            return enc.WriteToken(jsontext.String("✓"))
        }
        return enc.WriteToken(jsontext.String("✗"))
    },
)

Then we define a custom marshaler for bool-like strings:

// Marshals boolean-like strings to ✓ or ✗.
strMarshaler := json.MarshalToFunc(
    func(enc *jsontext.Encoder, val string) error {
        if val == "on" || val == "true" {
            return enc.WriteToken(jsontext.String("✓"))
        }
        if val == "off" || val == "false" {
            return enc.WriteToken(jsontext.String("✗"))
        }
        // SkipFunc is a special type of error that tells Go to skip
        // the current marshaler and move on to the next one. In our case,
        // the next one will be the default marshaler for strings.
        return json.SkipFunc
    },
)

Finally, we combine marshalers with JoinMarshalers and pass them to the marshaling function using the WithMarshalers option:

// Combine custom marshalers with JoinMarshalers.
marshalers := json.JoinMarshalers(boolMarshaler, strMarshaler)

// Marshal some values.
vals := []any{true, "off", "hello"}
data, err := json.Marshal(vals, json.WithMarshalers(marshalers))
fmt.Println(string(data), err)
["✓","✗","hello"] <nil>

Isn't that cool?

There are plenty of other goodies, like support for I/O readers and writers, nested objects inlining, a plethora of options, and a huge performance boost. So, again, I encourage you to check out the post dedicated to v2 changes.

𝗣 63397, 71497

# Container-aware GOMAXPROCS

The GOMAXPROCS runtime setting controls the maximum number of operating system threads the Go scheduler can use to execute goroutines concurrently. Since Go 1.5, it defaults to the value of runtime.NumCPU, which is the number of logical CPUs on the machine (strictly speaking, this is either the total number of logical CPUs or the number allowed by the CPU affinity mask, whichever is lower).

For example, on my 8-core laptop, the default value of GOMAXPROCS is also 8:

maxProcs := runtime.GOMAXPROCS(0) // returns the current value
fmt.Println("NumCPU:", runtime.NumCPU())
fmt.Println("GOMAXPROCS:", maxProcs)
NumCPU: 8
GOMAXPROCS: 8

Go programs often run in containers, like those managed by Docker or Kubernetes. These systems let you limit the CPU resources for a container using a Linux feature called cgroups.

A cgroup (control group) in Linux lets you group processes together and control how much CPU, memory, and network I/O they can use by setting limits and priorities.

For example, here's how you can limit a Docker container to use only four CPUs:

docker run --cpus=4 golang:1.24-alpine go run /app/nproc.go

Before version 1.25, the Go runtime didn't consider the CPU quota when setting the GOMAXPROCS value. No matter how you limited CPU resources, GOMAXPROCS was always set to the number of logical CPUs on the host machine:

docker run --cpus=4 golang:1.24-alpine go run /app/nproc.go
NumCPU: 8
GOMAXPROCS: 8

Now, the Go runtime started to respect the CPU quota:

docker run --cpus=4 golang:1.25rc1-alpine go run /app/nproc.go
NumCPU: 8
GOMAXPROCS: 4

The default value of GOMAXPROCS is now set to either the number of logical CPUs or the CPU limit enforced by cgroup settings for the process, whichever is lower.

Fractional CPU limits are rounded up:

docker run --cpus=2.3 golang:1.25rc1-alpine go run /app/nproc.go
NumCPU: 8
GOMAXPROCS: 3

On a machine with multiple CPUs, the minimum default value for GOMAXPROCS is 2, even if the CPU limit is set lower:

docker run --cpus=1 golang:1.25rc1-alpine go run /app/nproc.go
NumCPU: 8
GOMAXPROCS: 2

The Go runtime automatically updates GOMAXPROCS if the CPU limit changes. The current implementation updates up to once per second (less if the application is idle).

Note on CPU limits

Cgroups actually offer not just one, but two ways to limit CPU resources:

  • CPU quota — the maximum CPU time the cgroup may use within some period window.
  • CPU shares — relative CPU priorities given to the kernel scheduler.

Docker's --cpus and --cpu-period/--cpu-quota set the quota, while --cpu-shares sets the shares.

Kubernetes' CPU limit sets the quota, while CPU request sets the shares.

Go's runtime GOMAXPROCS only takes the CPU quota into account, not the shares.

You can set GOMAXPROCS manually using the runtime.GOMAXPROCS function. In this case, the runtime will use the value you provide and won't try to change it:

runtime.GOMAXPROCS(4)
fmt.Println("GOMAXPROCS:", runtime.GOMAXPROCS(0))
GOMAXPROCS: 4

You can also undo any manual changes you made — whether by setting the GOMAXPROCS environment variable or calling runtime.GOMAXPROCS() — and return to the default value chosen by the runtime. To do this, use the new runtime.SetDefaultGOMAXPROCS function:

GOMAXPROCS=2 go1.25rc1 run nproc.go
// Using the environment variable.
fmt.Println("GOMAXPROCS:", runtime.GOMAXPROCS(0))

// Using the manual setting.
runtime.GOMAXPROCS(4)
fmt.Println("GOMAXPROCS:", runtime.GOMAXPROCS(0))

// Back to the default value.
runtime.SetDefaultGOMAXPROCS()
fmt.Println("GOMAXPROCS:", runtime.GOMAXPROCS(0))
GOMAXPROCS: 2
GOMAXPROCS: 4
GOMAXPROCS: 8

To provide backward compatibility, the new GOMAXPROCS behavior only takes effect if your program uses Go version 1.25 or higher in your go.mod. You can also turn if off manually using these GODEBUG settings:

  • containermaxprocs=0 to ignore the cgroups CPU quota.
  • updatemaxprocs=0 to prevent GOMAXPROCS from updating automatically.

𝗣 73193 • 𝗖𝗟 668638, 670497, 672277, 677037

# Green Tea garbage collector

Some of us Go folks used to joke about Java and its many garbage collectors. Now the joke's on us — there's a new experimental garbage collector coming to Go.

Green Tea is a garbage collecting algorithm optimized for programs that create lots of small objects and run on modern computers with many CPU cores.

The current GC scans memory in a way that jumps around a lot, which is slow because it causes many delays waiting for memory. The problem gets even worse on high-performance systems with many cores and non-uniform memory architectures, where each CPU or group of CPUs has its own "local" RAM.

Green Tea takes a different approach. Instead of scanning individual small objects, it scans memory in much larger, contiguous blocks — spans. Each span contains many small objects of the same size. By working with bigger blocks, the GC can scan more efficiently and make better use of the CPU's memory cache.

Benchmark results vary, but the Go team expects a 10–40% reduction in garbage collection overhead in real-world programs that heavily use the GC.

I ran an informal test by doing 1,000,000 reads and writes to Redka (my Redis clone written in Go), and observed similar GC pause times both the old and new GC algorithms. But Redka probably isn't the best example here because it relies heavily on SQLite, and the Go part is pretty minimal.

The new garbage collector is experimental and can be enabled by setting GOEXPERIMENT=greenteagc at build time. The design and implementation of the GC may change in future releases. For more information or to give feedback, see the proposal.

𝗣 73581Feedback

# CSRF protection

The new http.CrossOriginProtection type implements protection against cross-site request forgery (CSRF) by rejecting non-safe cross-origin browser requests.

It detects cross-origin requests in these ways:

  • By checking the Sec-Fetch-Site header, if it's present.
  • By comparing the hostname in the Origin header to the one in the Host header.

Here's an example where we enable CrossOriginProtection and explicitly allow some extra origins:

// Register some handlers.
mux := http.NewServeMux()
mux.HandleFunc("GET /get", func(w http.ResponseWriter, req *http.Request) {
    io.WriteString(w, "ok\n")
})
mux.HandleFunc("POST /post", func(w http.ResponseWriter, req *http.Request) {
    io.WriteString(w, "ok\n")
})

// Configure protection against CSRF attacks.
antiCSRF := http.NewCrossOriginProtection()
antiCSRF.AddTrustedOrigin("https://example.com")
antiCSRF.AddTrustedOrigin("https://*.example.com")

// Add CSRF protection to all handlers.
srv := http.Server{
    Addr:    ":8080",
    Handler: antiCSRF.Handler(mux),
}
log.Fatal(srv.ListenAndServe())

Now, if the browser sends a request from the same domain the server is using, the server will allow it:

curl --data "ok" -H "sec-fetch-site:same-origin" localhost:8080/post
ok

If the browser sends a cross-origin request, the server will reject it:

curl --data "ok" -H "sec-fetch-site:cross-site" localhost:8080/post
cross-origin request detected from Sec-Fetch-Site header

If the Origin header doesn't match the Host header, the server will reject the request:

curl --data "ok" \
  -H "origin:https://evil.com" \
  -H "host:antonz.org" \
  localhost:8080/post
cross-origin request detected, and/or browser is out of date:
Sec-Fetch-Site is missing, and Origin does not match Host

If the request comes from a trusted origin, the server will allow it:

curl --data "ok" \
  -H "origin:https://example.com" \
  -H "host:antonz.org" \
  localhost:8080/post
ok

The server will always allow GET, HEAD, and OPTIONS methods because they are safe:

curl -H "origin:https://evil.com" localhost:8080/get
ok

The server will always allow requests without Sec-Fetch-Site or Origin headers (by design):

curl --data "ok" localhost:8080/post
ok

𝗣 73626 • 𝗖𝗟 674936, 680396

# Go wait group, go!

Everyone knows how to run a goroutine with a wait group:

var wg sync.WaitGroup

wg.Add(1)
go func() {
    defer wg.Done()
    fmt.Println("go is awesome")
}()

wg.Add(1)
go func() {
    defer wg.Done()
    fmt.Println("cats are cute")
}()

wg.Wait()
fmt.Println("done")
cats are cute
go is awesome
done

The new WaitGroup.Go method automatically increments the wait group counter, runs a function in a goroutine, and decrements the counter when it's done. This means we can rewrite the example above without using wg.Add() and wg.Done():

var wg sync.WaitGroup

wg.Go(func() {
    fmt.Println("go is awesome")
})

wg.Go(func() {
    fmt.Println("cats are cute")
})

wg.Wait()
fmt.Println("done")
cats are cute
go is awesome
done

The implementation is just what you'd expect:

// https://github.com/golang/go/blob/master/src/sync/waitgroup.go
func (wg *WaitGroup) Go(f func()) {
    wg.Add(1)
    go func() {
        defer wg.Done()
        f()
    }()
}

It's curious that it took the Go team 13 years to add a simple "Add+Done" wrapper. But hey, better late than never!

𝗣 63796 • 𝗖𝗟 662635

# Flight recording

Flight recording is a tracing technique that collects execution data, such as function calls and memory allocations, within a sliding window that's limited by size or duration. It helps to record traces of interesting program behavior, even if you don't know in advance when it will happen.

The new trace.FlightRecorder type implements a flight recorder in Go. It tracks a moving window over the execution trace produced by the runtime, always containing the most recent trace data.

Here's an example of how you might use it.

First, configure the sliding window:

// Configure the flight recorder to keep
// at least 5 seconds of trace data,
// with a maximum buffer size of 3MB.
// Both of these are hints, not strict limits.
cfg := trace.FlightRecorderConfig{
    MinAge:   5 * time.Second,
    MaxBytes: 3 << 20, // 3MB
}

Then create the recorder and start it:

// Create and start the flight recorder.
rec := trace.NewFlightRecorder(cfg)
rec.Start()
defer rec.Stop()

Continue with the application code as usual:

// Simulate some workload.
done := make(chan struct{})
go func() {
    defer close(done)
    const n = 1 << 20
    var s []int
    for range n {
        s = append(s, rand.IntN(n))
    }
    fmt.Printf("done filling slice of %d elements\n", len(s))
}()
<-done

Finally, save the trace snapshot to a file when an important event occurs:

// Save the trace snapshot to a file.
file, _ := os.Create("/tmp/trace.out")
defer file.Close()
n, _ := rec.WriteTo(file)
fmt.Printf("wrote %dB to trace file\n", n)
done filling slice of 1048576 elements
wrote 8441B to trace file

Use the go tool to view the trace in the browser:

go1.25rc1 tool trace /tmp/trace.out

𝗣 63185 • 𝗖𝗟 673116

# More Root methods

The os.Root type, which limits filesystem operations to a specific directory, now supports several new methods similar to functions already found in the os package.

Chmod changes the mode of a file:

root, _ := os.OpenRoot("data")
root.Chmod("01.txt", 0600)

finfo, _ := root.Stat("01.txt")
fmt.Println(finfo.Mode().Perm())
-rw-------

Chown changes the numeric user ID (uid) and group ID (gid) of a file:

root, _ := os.OpenRoot("data")
root.Chown("01.txt", 1000, 1000)

finfo, _ := root.Stat("01.txt")
stat := finfo.Sys().(*syscall.Stat_t)
fmt.Printf("uid=%d, gid=%d\n", stat.Uid, stat.Gid)
uid=1000, gid=1000

Chtimes changes the access and modification times of a file:

root, _ := os.OpenRoot("data")
mtime := time.Date(2020, 1, 1, 0, 0, 0, 0, time.UTC)
atime := time.Now()
root.Chtimes("01.txt", atime, mtime)

finfo, _ := root.Stat("01.txt")
fmt.Println(finfo.ModTime())
2020-01-01 00:00:00 +0000 UTC

Link creates a hard link to a file:

root, _ := os.OpenRoot("data")
root.Link("01.txt", "hardlink.txt")

finfo, _ := root.Stat("hardlink.txt")
fmt.Println(finfo.Name())
hardlink.txt

MkdirAll creates a new directory and any parent directories that don't already exist:

const dname = "path/to/secret"

root, _ := os.OpenRoot("data")
root.MkdirAll(dname, 0750)

finfo, _ := root.Stat(dname)
fmt.Println(dname, finfo.Mode())
path/to/secret drwxr-x---

RemoveAll removes a file or a directory and any children that it contains:

root, _ := os.OpenRoot("data")
root.RemoveAll("01.txt")

finfo, err := root.Stat("01.txt")
fmt.Println(finfo, err)
<nil> statat 01.txt: no such file or directory

Rename renames (moves) a file or a directory:

const oldname = "01.txt"
const newname = "go.txt"

root, _ := os.OpenRoot("data")
root.Rename(oldname, newname)

_, err := root.Stat(oldname)
fmt.Println(err)

finfo, _ := root.Stat(newname)
fmt.Println(finfo.Name())
statat 01.txt: no such file or directory
go.txt

Symlink creates a symbolic link to a file. Readlink returns the destination of the symbolic link:

const lname = "symlink.txt"

root, _ := os.OpenRoot("data")
root.Symlink("01.txt", lname)

lpath, _ := root.Readlink(lname)
fmt.Println(lname, "->", lpath)
symlink.txt -> 01.txt

WriteFile writes data to a file, creating it if necessary. ReadFile reads the file and returns its contents:

const fname = "go.txt"

root, _ := os.OpenRoot("data")
root.WriteFile(fname, []byte("go is awesome"), 0644)

content, _ := root.ReadFile(fname)
fmt.Printf("%s: %s\n", fname, content)
go.txt: go is awesome

Since there are now plenty of os.Root methods, you probably don't need the file-related os functions most of the time. This can make working with files much safer.

Speaking of file systems, the ones returned by os.DirFS() (a file system rooted at the given directory) and os.Root.FS() (a file system for the tree of files in the root) both implement the new fs.ReadLinkFS interface. It has two methods — ReadLink and Lstat.

ReadLink returns the destination of the symbolic link:

const lname = "symlink.txt"

root, _ := os.OpenRoot("data")
root.Symlink("01.txt", lname)

fsys := root.FS().(fs.ReadLinkFS)
lpath, _ := fsys.ReadLink(lname)
fmt.Println(lname, "->", lpath)
symlink.txt -> 01.txt

I have to say, the inconsistent naming between os.Root.Readlink and fs.ReadLinkFS.ReadLink is quite surprising. Maybe it's not too late to fix?

Lstat returns the information about a file or a symbolic link:

fsys := os.DirFS("data").(fs.ReadLinkFS)
finfo, _ := fsys.Lstat("01.txt")

fmt.Printf("name:  %s\n", finfo.Name())
fmt.Printf("size:  %dB\n", finfo.Size())
fmt.Printf("mode:  %s\n", finfo.Mode())
fmt.Printf("mtime: %s\n", finfo.ModTime().Format(time.DateOnly))
name:  01.txt
size:  11B
mode:  -rw-r--r--
mtime: 2025-06-22

That's it for the os package!

𝗣 49580, 67002, 73126 • 𝗖𝗟 645718, 648295, 649515, 649536, 658995, 659416, 659757, 660635, 661595, 674116, 674315, 676135

# Reflective type assertion

To convert a reflect.Value back to a specific type, you typically use the Value.Interface() method combined with a type assertion:

alice := &Person{"Alice", 25}

// Given a reflection Value...
aliceVal := reflect.ValueOf(alice).Elem()
// ...convert it back to the Person type.
person, _ := aliceVal.Interface().(Person)

fmt.Printf("Name: %s, Age: %d\n", person.Name, person.Age)
Name: Alice, Age: 25

Now you can use the new generic reflect.TypeAssert function instead:

alice := &Person{"Alice", 25}

// Given a reflection Value...
aliceVal := reflect.ValueOf(alice).Elem()
// ...convert it back to the Person type.
person, _ := reflect.TypeAssert[Person](aliceVal)

fmt.Printf("Name: %s, Age: %d\n", person.Name, person.Age)
Name: Alice, Age: 25

It's more idiomatic and avoids unnecessary memory allocations, since the value is never boxed in an interface.

𝗣 62121 • 𝗖𝗟 648056

# Test attributes and friends

With the new T.Attr method, you can add extra test information, like a link to an issue, a test description, or anything else you need to analyze the test results:

func TestAttrs(t *testing.T) {
    t.Attr("issue", "demo-1234")
    t.Attr("description", "Testing for the impossible")

    if 21*2 != 42 {
        t.Fatal("What in the world happened to math?")
    }
}
=== RUN   TestAttrs
=== ATTR  TestAttrs issue demo-1234
=== ATTR  TestAttrs description Testing for the impossible
--- PASS: TestAttrs (0.00s)

Attributes can be especially useful in JSON format if you send the test output to a CI or other system for automatic processing:

go1.25rc1 test -json -run=.
...
{
    "Time":"2025-06-25T20:34:16.831401+00:00",
    "Action":"attr",
    "Package":"sandbox",
    "Test":"TestAttrs",
    "Key":"issue",
    "Value":"demo-1234"
}
...
{
    "Time":"2025-06-25T20:34:16.831415+00:00",
    "Action":"attr",
    "Package":"sandbox",
    "Test":"TestAttrs",
    "Key":"description",
    "Value":"Testing for the impossible"
}
...

The output is formatted to make it easier to read.

The same Attr method is also available on testing.B and testing.F.

𝗣 43936 • 𝗖𝗟 662437

The new T.Output method lets you access the output stream (io.Writer) used by the test. This can be helpful if you want to send your application log to the test log stream, making it easier to read or analyze automatically:

func TestLog(t *testing.T) {
    t.Log("test message 1")
    t.Log("test message 2")
    appLog := slog.New(slog.NewTextHandler(t.Output(), nil))
    appLog.Info("app message")
}
=== RUN   TestLog
    main_test.go:12: test message 1
    main_test.go:13: test message 2
    time=2025-06-25T16:14:34.085Z level=INFO msg="app message"
--- PASS: TestLog (0.00s)

The same Output method is also available on testing.B and testing.F.

𝗣 59928 • 𝗖𝗟 672395, 677875

Last but not least, the testing.AllocsPerRun function will now panic if parallel tests are running.

Compare 1.24 behavior:

// go 1.24
func TestAllocs(t *testing.T) {
    t.Parallel()
    allocs := testing.AllocsPerRun(100, func() {
        var s []int
        // Do some allocations.
        for i := range 1024 {
            s = append(s, i)
        }
    })
    t.Log("Allocations per run:", allocs)
}
=== RUN   TestAllocs
=== PAUSE TestAllocs
=== CONT  TestAllocs
    main_test.go:21: Allocations per run: 12
--- PASS: TestAllocs (0.00s)

To 1.25:

// go 1.25
func TestAllocs(t *testing.T) {
    t.Parallel()
    allocs := testing.AllocsPerRun(100, func() {
        var s []int
        // Do some allocations.
        for i := range 1024 {
            s = append(s, i)
        }
    })
    t.Log("Allocations per run:", allocs)
}
=== RUN   TestAllocs
=== PAUSE TestAllocs
=== CONT  TestAllocs
--- FAIL: TestAllocs (0.00s)
panic: testing: AllocsPerRun called during parallel test [recovered, repanicked]

The thing is, the result of AllocsPerRun is inherently flaky if other tests are running in parallel. That's why there's a new panicking behavior — it should help catch these kinds of bugs.

𝗣 70464 • 𝗖𝗟 630137

# Grouped attributes for logging

With structured logging, you often group related attributes under a single key:

logger.Info("deposit",
    slog.Bool("ok", true),
    slog.Group("amount",
        slog.Int("value", 1000),
        slog.String("currency", "USD"),
    ),
)
msg=deposit ok=true amount.value=1000 amount.currency=USD

It works just fine — unless you want to gather the attributes first:

attrs := []slog.Attr{
    slog.Int("value", 1000),
    slog.String("currency", "USD"),
}
logger.Info("deposit",
	slog.Bool("ok", true),
	slog.Group("amount", attrs...),
)
cannot use attrs (variable of type []slog.Attr)
as []any value in argument to slog.Group
(exit status 1)

slog.Group expects a slice of any values, so it doesn't accept a slice of slog.Attrs.

The new slog.GroupAttrs function fixes this issue by creating a group from the given slog.Attrs:

attrs := []slog.Attr{
    slog.Int("value", 1000),
    slog.String("currency", "USD"),
}
logger.Info("deposit",
    slog.Bool("ok", true),
    slog.GroupAttrs("amount", attrs...),
)
msg=deposit ok=true amount.value=1000 amount.currency=USD

Not a big deal, but can be quite handy.

𝗣 66365 • 𝗖𝗟 672915

# Hash cloner

The new hash.Cloner interface defines a hash function that can return a copy of its current state:

// https://github.com/golang/go/blob/master/src/hash/hash.go
type Cloner interface {
    Hash
    Clone() (Cloner, error)
}

All standard library hash.Hash implementations now provide the Clone function, including MD5, SHA-1, SHA-3, FNV-1, CRC-64, and others:

h1 := sha3.New256()
h1.Write([]byte("hello"))

clone, _ := h1.Clone()
h2 := clone.(*sha3.SHA3)

// h2 has the same state as h1, so it will produce
// the same hash after writing the same data.
h1.Write([]byte("world"))
h2.Write([]byte("world"))

fmt.Printf("h1: %x\n", h1.Sum(nil))
fmt.Printf("h2: %x\n", h2.Sum(nil))
fmt.Printf("h1 == h2: %t\n", reflect.DeepEqual(h1, h2))
h1: 92dad9443e4dd6d70a7f11872101ebff87e21798e4fbb26fa4bf590eb440e71b
h2: 92dad9443e4dd6d70a7f11872101ebff87e21798e4fbb26fa4bf590eb440e71b
h1 == h2: true

𝗣 69521 • 𝗖𝗟 675197

# Final thoughts

Go 1.25 finalizes support for testing concurrent code, introduces a major experimental JSON package, and improves the runtime with a new GOMAXPROCS design and garbage collector. It also adds a flight recorder, modern CSRF protection, a long-awaited wait group shortcut, and several other improvements.

All in all, a great release!

P.S. To catch up on other Go releases, check out the Go features by version list or explore the interactive tours for Go 1.24 and 1.23.

P.P.S. Want to learn more about Go? Check out my interactive book on concurrency

]]>
JSON evolution in Go: from v1 to v2https://antonz.org/go-json-v2/Sun, 22 Jun 2025 13:30:00 +0000https://antonz.org/go-json-v2/Reviewing the key changes in json/v2.The second version of the json package coming in Go 1.25 is a big update, and it has a lot of breaking changes. The v2 package adds new features, fixes API issues and behavioral flaws, and boosts performance. Let's take a look at what's changed!

All examples are interactive, so you can read and practice at the same time.

The basic use case with Marshal and Unmarshal stays the same. This code works in both v1 and v2:

type Person struct {
    Name string
    Age  int
}
alice := Person{Name: "Alice", Age: 25}

// Marshal Alice.
b, err := json.Marshal(alice)
fmt.Println(string(b), err)

// Unmarshal Alice.
err = json.Unmarshal(b, &alice)
fmt.Println(alice, err)
{"Name":"Alice","Age":25} <nil>
{Alice 25} <nil>

But the rest is pretty different. Let's go over the main changes from v1.

Write/ReadEncode/DecodeOptionsTagsCustom marshalingDefault behaviorPerformanceFinal thoughts

MarshalWrite and UnmarshalRead

In v1, you used Encoder to marshal to the io.Writer, and Decoder to unmarshal from the io.Reader:

// Marshal Alice.
alice := Person{Name: "Alice", Age: 25}
out := new(strings.Builder) // io.Writer
enc := json.NewEncoder(out)
enc.Encode(alice)
fmt.Println(out.String())

// Unmarshal Bob.
in := strings.NewReader(`{"Name":"Bob","Age":30}`) // io.Reader
dec := json.NewDecoder(in)
var bob Person
dec.Decode(&bob)
fmt.Println(bob)
{"Name":"Alice","Age":25}

{Bob 30}

From now on, I'll leave out error handling to keep things simple.

In v2, you can use MarshalWrite and UnmarshalRead directly, without any intermediaries:

// Marshal Alice.
alice := Person{Name: "Alice", Age: 25}
out := new(strings.Builder)
json.MarshalWrite(out, alice)
fmt.Println(out.String())

// Unmarshal Bob.
in := strings.NewReader(`{"Name":"Bob","Age":30}`)
var bob Person
json.UnmarshalRead(in, &bob)
fmt.Println(bob)
{"Name":"Alice","Age":25}
{Bob 30 false}

They're not interchangeable, though:

  • MarshalWrite does not add a newline, unlike the old Encoder.Encode.
  • UnmarshalRead reads everything from the reader until it hits io.EOF, while the old Decoder.Decode only reads the next JSON value.

MarshalEncode and UnmarshalDecode

The Encoder and Decoder types have been moved to the new jsontext package, and their interfaces have changed significantly (to support low-level streaming encode/decode operations).

You can use them with json functions to read and write JSON streams, similar to how Encode and Decode worked before:

  • v1 Encoder.Encode → v2 json.MarshalEncode + jsontext.Encoder.
  • v1 Decoder.Decode → v2 json.UnmarshalDecode + jsontext.Decoder.

Streaming encoder:

people := []Person{
    {Name: "Alice", Age: 25},
    {Name: "Bob", Age: 30},
    {Name: "Cindy", Age: 15},
}
out := new(strings.Builder)
enc := jsontext.NewEncoder(out)

for _, p := range people {
    json.MarshalEncode(enc, p)
}

fmt.Print(out.String())
{"Name":"Alice","Age":25}
{"Name":"Bob","Age":30}
{"Name":"Cindy","Age":15}

Streaming decoder:

in := strings.NewReader(`
    {"Name":"Alice","Age":25}
    {"Name":"Bob","Age":30}
    {"Name":"Cindy","Age":15}
`)
dec := jsontext.NewDecoder(in)

for {
    var p Person
    // Decodes one Person object per call.
    err := json.UnmarshalDecode(dec, &p)
    if err == io.EOF {
        break
    }
    fmt.Println(p)
}
{Alice 25}
{Bob 30}
{Cindy 15}

Unlike UnmarshalRead, UnmarshalDecode works in a fully streaming way, decoding one value at a time with each call, instead of reading everything until io.EOF.

Options

Options configure marshaling and unmarshaling functions with specific features:

  • FormatNilMapAsNull and FormatNilSliceAsNull define how to encode nil maps and slices.
  • MatchCaseInsensitiveNames allows matching Namename and the like.
  • Multiline expands JSON objects to multiple lines.
  • OmitZeroStructFields omits fields with zero values from the output.
  • SpaceAfterColon and SpaceAfterComma add a space after each : or ,
  • StringifyNumbers represents numeric types as strings.
  • WithIndent and WithIndentPrefix indent nested properties (note that the MarshalIndent function has been removed).

Each marshaling or unmarshaling function can take any number of options:

alice := Person{Name: "Alice", Age: 25}
b, _ := json.Marshal(
    alice,
    json.OmitZeroStructFields(true),
    json.StringifyNumbers(true),
    jsontext.WithIndent("  "),
)
fmt.Println(string(b))
{
  "Name": "Alice",
  "Age": "25"
}

You can also combine options with JoinOptions:

alice := Person{Name: "Alice", Age: 25}
opts := json.JoinOptions(
    jsontext.SpaceAfterColon(true),
    jsontext.SpaceAfterComma(true),
)
b, _ := json.Marshal(alice, opts)
fmt.Println(string(b))
{"Name": "Alice", "Age": 25}

See the complete list of options in the documentation: some are in the json package, others are in the jsontext package.

Tags

v2 supports field tags defined in v1:

  • omitzero and omitempty to omit empty values.
  • string to represent numeric types as strings.
  • - to ignore fields.

And adds some more:

  • case:ignore or case:strict specify how to handle case differences.
  • format:template formats the field value according to the template.
  • inline flattens the output by promoting the fields of a nested object up to the parent.
  • unknown provides a "catch-all" for the unknown fields.

Here's an example demonstrating inline and format:

type Person struct {
    Name string         `json:"name"`
    // Format date as yyyy-mm-dd.
    BirthDate time.Time `json:"birth_date,format:DateOnly"`
    // Inline address fields into the Person object.
    Address             `json:",inline"`
}

type Address struct {
    Street string `json:"street"`
    City   string `json:"city"`
}

func main() {
    alice := Person{
        Name: "Alice",
        BirthDate: time.Date(2001, 7, 15, 12, 35, 43, 0, time.UTC),
        Address: Address{
            Street: "123 Main St",
            City:   "Wonderland",
        },
    }
    b, _ := json.Marshal(alice, jsontext.WithIndent("  "))
    fmt.Println(string(b))
}
{
  "name": "Alice",
  "birth_date": "2001-07-15",
  "street": "123 Main St",
  "city": "Wonderland"
}

And unknown:

type Person struct {
    Name string         `json:"name"`
    // Collect all unknown Person fields
    // into the Data field.
    Data map[string]any `json:",unknown"`
}

func main() {
    src := `{
        "name": "Alice",
        "hobby": "adventure",
        "friends": [
            {"name": "Bob"},
            {"name": "Cindy"}
        ]
    }`
    var alice Person
    json.Unmarshal([]byte(src), &alice)
    fmt.Println(alice)
}
{Alice map[friends:[map[name:Bob] map[name:Cindy]] hobby:adventure]}

Custom marshaling

The basic use case for custom marshaling using Marshaler and Unmarshaler interfaces stays the same. This code works in both v1 and v2:

// A custom boolean type represented
// as "✓" for true and "✗" for false.
type Success bool

func (s Success) MarshalJSON() ([]byte, error) {
    if s {
        return []byte(`"✓"`), nil
    }
    return []byte(`"✗"`), nil
}

func (s *Success) UnmarshalJSON(data []byte) error {
    // Data validation omitted for brevity.
    *s = string(data) == `"✓"`
    return nil
}

func main() {
    // Marshaling.
    val := Success(true)
    data, err := json.Marshal(val)
    fmt.Println(string(data), err)

    // Unmarshaling.
    src := []byte(`"✓"`)
    err = json.Unmarshal(src, &val)
    fmt.Println(val, err)
}
"✓" <nil>
true <nil>

However, the Go standard library documentation recommends using the new MarshalerTo and UnmarshalerFrom interfaces instead (they work in a purely streaming way, which can be much faster):

// A custom boolean type represented
// as "✓" for true and "✗" for false.
type Success bool

func (s Success) MarshalJSONTo(enc *jsontext.Encoder) error {
    if s {
        return enc.WriteToken(jsontext.String("✓"))
    }
    return enc.WriteToken(jsontext.String("✗"))
}

func (s *Success) UnmarshalJSONFrom(dec *jsontext.Decoder) error {
    // Data validation omitted for brevity.
    tok, err := dec.ReadToken()
    *s = tok.String() == `"✓"`
    return err
}

func main() {
    // Marshaling.
    val := Success(true)
    data, err := json.Marshal(val)
    fmt.Println(string(data), err)

    // Unmarshaling.
    src := []byte(`"✓"`)
    err = json.Unmarshal(src, &val)
    fmt.Println(val, err)
}
"✓" <nil>
true <nil>

Even better, you're no longer limited to just one way of marshaling a specific type. Now you can use custom marshalers and unmarshalers whenever you need, with the generic MarshalFunc and UnmarshalFunc functions.

func MarshalFunc[T any](fn func(T) ([]byte, error)) *Marshalers
func UnmarshalFunc[T any](fn func([]byte, T) error) *Unmarshalers

For example, you can marshal a bool value to or without creating a custom type:

// Custom marshaler for bool values.
boolMarshaler := json.MarshalFunc(
    func(val bool) ([]byte, error) {
        if val {
            return []byte(`"✓"`), nil
        }
        return []byte(`"✗"`), nil
    },
)

// Pass the custom marshaler to Marshal
// using the WithMarshalers option.
val := true
data, err := json.Marshal(val, json.WithMarshalers(boolMarshaler))
fmt.Println(string(data), err)
"✓" <nil>

And unmarshal or to bool:

// Custom unmarshaler for bool values.
boolUnmarshaler := json.UnmarshalFunc(
    func(data []byte, val *bool) error {
        *val = string(data) == `"✓"`
        return nil
    },
)

// Pass the custom unmarshaler to Unmarshal
// using the WithUnmarshalers option.
src := []byte(`"✓"`)
var val bool
err := json.Unmarshal(src, &val, json.WithUnmarshalers(boolUnmarshaler))
fmt.Println(val, err)
true <nil>

There are also MarshalToFunc and UnmarshalFromFunc functions for creating custom marshalers. They're similar to MarshalFunc and UnmarshalFunc, but they work with jsontext.Encoder and jsontext.Decoder instead of byte slices.

func MarshalToFunc[T any](fn func(*jsontext.Encoder, T) error) *Marshalers
func UnmarshalFromFunc[T any](fn func(*jsontext.Decoder, T) error) *Unmarshalers

You can combine marshalers using JoinMarshalers (and unmarshalers using JoinUnmarshalers). For example, here's how you can marshal both booleans (true/false) and "boolean-like" strings (on/off) to or , while keeping the default marshaling for all other values.

First we define a custom marshaler for bool values:

// Marshals boolean values to ✓ or ✗..
boolMarshaler := json.MarshalToFunc(
    func(enc *jsontext.Encoder, val bool) error {
        if val {
            return enc.WriteToken(jsontext.String("✓"))
        }
        return enc.WriteToken(jsontext.String("✗"))
    },
)

Then we define a custom marshaler for bool-like strings:

// Marshals boolean-like strings to ✓ or ✗.
strMarshaler := json.MarshalToFunc(
    func(enc *jsontext.Encoder, val string) error {
        if val == "on" || val == "true" {
            return enc.WriteToken(jsontext.String("✓"))
        }
        if val == "off" || val == "false" {
            return enc.WriteToken(jsontext.String("✗"))
        }
        // SkipFunc is a special type of error that tells Go to skip
        // the current marshaler and move on to the next one. In our case,
        // the next one will be the default marshaler for strings.
        return json.SkipFunc
    },
)

Finally, we combine marshalers with JoinMarshalers and pass them to the marshaling function using the WithMarshalers option:

// Combine custom marshalers with JoinMarshalers.
marshalers := json.JoinMarshalers(boolMarshaler, strMarshaler)

// Marshal some values.
vals := []any{true, "off", "hello"}
data, err := json.Marshal(vals, json.WithMarshalers(marshalers))
fmt.Println(string(data), err)
["✓","✗","hello"] <nil>

Isn't that cool?

Default behavior

v2 changes not only the package interface, but the default marshaling/unmarshaling behavior as well.

Some notable marshaling differences include:

  • v1 marshals a nil slice to null, v2 marshals to []. You can change it with the FormatNilSliceAsNull option.
  • v1 marshals a nil map to null, v2 marshals to {}. You can change it with the FormatNilMapAsNull option.
  • v1 marshals a byte array to an array of numbers, v2 marshals to a base64-encoded string. You can change it with format:array and format:base64 tags.
  • v1 allows invalid UTF-8 characters within a string, v2 does not. You can change it with the AllowInvalidUTF8 option.

Here's an example of the default v2 behavior:

type Person struct {
    Name    string
    Hobbies []string
    Skills  map[string]int
    Secret  [5]byte
}

func main() {
    alice := Person{
        Name:    "Alice",
        Secret: [5]byte{1, 2, 3, 4, 5},
    }
    b, _ := json.Marshal(alice, jsontext.Multiline(true))
    fmt.Println(string(b))
}
{
    "Name": "Alice",
    "Hobbies": [],
    "Skills": {},
    "Secret": "AQIDBAU="
}

And here's how you can enforce v1 behavior:

type Person struct {
    Name    string
    Hobbies []string
    Skills  map[string]int
    Secret  [5]byte `json:",format:array"`
}

func main() {
    alice := Person{
        Name:    "Alice",
        Secret: [5]byte{1, 2, 3, 4, 5},
    }
    b, _ := json.Marshal(
        alice,
        json.FormatNilMapAsNull(true),
        json.FormatNilSliceAsNull(true),
        jsontext.Multiline(true),
    )
    fmt.Println(string(b))
}
{
    "Name": "Alice",
    "Hobbies": null,
    "Skills": null,
    "Secret": [
        1,
        2,
        3,
        4,
        5
    ]
}

Some notable unmarshaling differences include:

  • v1 uses case-insensitive field name matching, v2 uses an exact, case-sensitive match. You can change it with the MatchCaseInsensitiveNames option or case tag.
  • v1 allows duplicate fields in a object, v2 does not. You can change it with the AllowDuplicateNames option.

Here's an example of the default v2 behavior (case-sensitive):

type Person struct {
    FirstName string
    LastName  string
}

func main() {
    src := []byte(`{"firstname":"Alice","lastname":"Zakas"}`)
    var alice Person
    json.Unmarshal(src, &alice)
    fmt.Printf("%+v\n", alice)
}
{FirstName: LastName:}

And here's how you can enforce v1 behavior (case-insensitive):

type Person struct {
    FirstName string
    LastName  string
}

func main() {
    src := []byte(`{"firstname":"Alice","lastname":"Zakas"}`)
    var alice Person
    json.Unmarshal(
        src, &alice,
        json.MatchCaseInsensitiveNames(true),
    )
    fmt.Printf("%+v\n", alice)
}
{FirstName:Alice LastName:Zakas}

See the complete list of behavioral changes in the documentation.

Performance

When marshaling, v2 performs about the same as v1. It's faster with some datasets, but slower with others. However, unmarshaling is much better: v2 is 2.7x to 10.2x faster than v1.

Also, you can get significant performance benefits by switching from regular MarshalJSON and UnmarshalJSON to their streaming alternatives — MarshalJSONTo and UnmarshalJSONFrom. According to the Go team, it allows to convert certain O(n²) runtime scenarios into O(n). For example, switching from UnmarshalJSON to UnmarshalJSONFrom in the k8s OpenAPI spec made it about 40 times faster.

See the jsonbench repo for benchmark details.

Final thoughts

Phew! That's a lot to take in. The v2 package has more features and is more flexible than v1, but it's also a lot more complex, especially given the split into json/v2 and jsontext subpackages.

A couple of things to keep in mind:

  • As of Go 1.25, the json/v2 package is experimental and can be enabled by setting GOEXPERIMENT=jsonv2 at build time. The package API is subject to change in future releases.
  • Turning on GOEXPERIMENT=jsonv2 makes the v1 json package use the new JSON implementation, which is faster and supports some options for better compatibility with the old marshaling and unmarshaling behavior.

Finally, here are some links to learn more about the v2 design and implementation:

proposal p.1proposal p.2json/v2jsontext

P.S. Want to learn more about Go? Check out my interactive book on concurrency

]]>
Gist of Go: Race conditionshttps://antonz.org/go-concurrency/race-conditions/Sun, 15 Jun 2025 10:30:00 +0000https://antonz.org/go-concurrency/race-conditions/Keep the system state correct by any means necessary.This is a chapter from my book on Go concurrency, which teaches the topic from the ground up through interactive examples.

Preventing data races with mutexes may sound easy, but dealing with race conditions is a whole other matter. Let's learn how to handle these beasts!

Race conditionCompare-and-setIdempotence and atomicityLockerTryLockShared nothingKeep it up

Race condition

Let's say we're keeping track of the money in users' accounts:

// Accounts - money in users' accounts.
type Accounts struct {
    bal map[string]int
    mu  sync.Mutex
}

// NewAccounts creates a new set of accounts.
func NewAccounts(bal map[string]int) *Accounts {
    return &Accounts{bal: maps.Clone(bal)}
}

We can check the balance by username or change the balance:

// Get returns the user's balance.
func (a *Accounts) Get(name string) int {
    a.mu.Lock()
    defer a.mu.Unlock()
    return a.bal[name]
}

// Set changes the user's balance.
func (a *Accounts) Set(name string, amount int) {
    a.mu.Lock()
    defer a.mu.Unlock()
    a.bal[name] = amount
}

Account operations — Get and Set — are concurrent-safe, thanks to the mutex.

There's also a store that sells Lego sets:

// A Lego set.
type LegoSet struct {
    name  string
    price int
}

Alice has 50 coins in her account. She wants to buy two sets: "Castle" for 40 coins and "Plants" for 20 coins:

func main() {
    acc := NewAccounts(map[string]int{
        "alice": 50,
    })
    castle := LegoSet{name: "Castle", price: 40}
    plants := LegoSet{name: "Plants", price: 20}

    var wg sync.WaitGroup
    wg.Add(2)

    // Alice buys a castle.
    go func() {
        defer wg.Done()
        balance := acc.Get("alice")
        if balance < castle.price {
            return
        }
        time.Sleep(5 * time.Millisecond)
        acc.Set("alice", balance-castle.price)
        fmt.Println("Alice bought the castle")
    }()

    // Alice buys plants.
    go func() {
        defer wg.Done()
        balance := acc.Get("alice")
        if balance < plants.price {
            return
        }
        time.Sleep(10 * time.Millisecond)
        acc.Set("alice", balance-plants.price)
        fmt.Println("Alice bought the plants")
    }()

    wg.Wait()

    balance := acc.Get("alice")
    fmt.Println("Alice's balance:", balance)
}
Alice bought the castle
Alice bought the plants
Alice's balance: 30

What a twist! Not only did Alice buy both sets for a total of 60 coins (even though she only had 50 coins), but she also ended up with 30 coins left! Great deal for Alice, not so great for us.

The problem is that checking and updating the balance is not an atomic operation:

// body of the second goroutine
balance := acc.Get("alice")             // (1)
if balance < plants.price {             // (2)
    return
}
time.Sleep(10 * time.Millisecond)
acc.Set("alice", balance-plants.price)  // (3)

At point ➊, we see a balance of 50 coins (the first goroutine hasn't done anything yet), so the check at ➋ passes. By point ➌, Alice has already bought the castle (the first goroutine has finished), so her actual balance is 10 coins. But we don't know this and still think her balance is 50 coins. So at point ➌, Alice buys the plants for 20 coins, and the balance becomes 30 coins (the "assumed" balance of 50 coins minus the 20 coins for the plants = 30 coins).

Individual actions on the balance are safe (there's no data race). However, balance reads/writes from different goroutines can get "mixed up", leading to an incorrect final balance. This situation is called a race condition.

You can't fully eliminate uncertainty in a concurrent environment. Events will happen in an unpredictable order — that's just how concurrency works. However, you can protect the system's state — in our case, the purchased sets and balance — so it stays correct no matter what order things happen in.

Let's check and update the balance in one atomic operation, protecting the entire purchase with a mutex. This way, purchases are processed strictly sequentially:

// Shared mutex.
var mu sync.Mutex

// Alice buys a castle.
go func() {
    defer wg.Done()

    // Protect the entire purchase with a mutex.
    mu.Lock()
    defer mu.Unlock()

    balance := acc.Get("alice")
    if balance < castle.price {
        return
    }
    time.Sleep(5 * time.Millisecond)
    acc.Set("alice", balance-castle.price)
    fmt.Println("Alice bought the castle")
}()

// Alice buys plants.
go func() {
    defer wg.Done()

    // Protect the entire purchase with a mutex.
    mu.Lock()
    defer mu.Unlock()

    balance := acc.Get("alice")
    if balance < plants.price {
        return
    }
    time.Sleep(10 * time.Millisecond)
    acc.Set("alice", balance-plants.price)
    fmt.Println("Alice bought the plants")
}()
Alice bought the plants
Alice's balance: 30

One of the goroutines will run first, lock the mutex, check and update the balance, then unlock the mutex. Only after that will the second goroutine be able to lock the mutex and make its purchase.

We still can't be sure which purchase will happen — it depends on the order the goroutines run. But now we are certain that Alice won't buy more than she's supposed to, and the final balance will be correct:

Alice bought the castle
Alice's balance: 10

Or:

Alice bought the plants
Alice's balance: 30

To reiterate:

  • A data race happens when multiple goroutines access shared data, and at least one of them modifies it. We need to protect the data from this kind of concurrent access.
  • A race condition happens when an unpredictable order of operations leads to an incorrect system state. In a concurrent environment, we can't control the exact order things happen. Still, we need to make sure that no matter the order, the system always ends up in the correct state.

Go's race detector can find data races, but it doesn't catch race conditions. It's always up to the programmer to prevent race conditions.

Compare-and-set

Let's go back to the situation with the race condition before we added the mutex:

// Alice's balance = 50 coins.
// Castle price = 40 coins.
// Plants price = 20 coins.

// Alice buys a castle.
go func() {
    defer wg.Done()
    balance := acc.Get("alice")
    if balance < castle.price {
        return
    }
    time.Sleep(5 * time.Millisecond)
    acc.Set("alice", balance-castle.price)
    fmt.Println("Alice bought the castle")
}()

// Alice buys plants.
go func() {
    defer wg.Done()
    balance := acc.Get("alice")
    if balance < plants.price {
        return
    }
    time.Sleep(10 * time.Millisecond)
    acc.Set("alice", balance-plants.price)
    fmt.Println("Alice bought the plants")
}()
Alice bought the castle
Alice bought the plants
Alice's balance: 30

As we discussed, the reason for the incorrect final state is that buying a set (checking and updating the balance) is not an atomic operation:

// body of the second goroutine
balance := acc.Get("alice")             // (1)
if balance < plants.price {             // (2)
    return
}
time.Sleep(10 * time.Millisecond)
acc.Set("alice", balance-plants.price)  // (3)

At point ➊, we see a balance of 50 coins, so the check at ➋ passes. By point ➌, Alice has already bought the castle, so her actual balance is 10 coins. But we don't know this and still think her balance is 50 coins. So at point ➌, Alice buys the plants for 20 coins, and the balance becomes 30 coins (the "assumed" balance of 50 coins minus the 20 coins for the plants = 30 coins).

To solve the problem, we can protect the entire purchase with a mutex, just like we did before. But there's another way to handle it.

We can keep two separate operations (checking and updating the balance), but instead of a regular update (set), use an atomic compare-and-set operation:

// Get returns the user's balance.
func (a *Accounts) Get(name string) int {
    a.mu.Lock()
    defer a.mu.Unlock()
    return a.bal[name]
}

// CompareAndSet changes the user's balance to new
// if the current value equals old. Returns false otherwise.
func (a *Accounts) CompareAndSet(name string, old, new int) bool {
    a.mu.Lock()
    defer a.mu.Unlock()
    if a.bal[name] != old {
        return false
    }
    a.bal[name] = new
    return true
}

CompareAndSet first checks if the balance has changed compared to the old value provided by the caller (the "assumed" balance). If the balance has changed (the assumed balance doesn't match the actual balance), it doesn't set the new value and returns false. If the balance hasn't changed (the assumed balance matches the actual balance), it sets the new value and returns true.

Now we can safely sell Lego:

// Alice buys a castle.
go func() {
    defer wg.Done()
    balance := acc.Get("alice")
    if balance < castle.price {
        return
    }
    time.Sleep(5 * time.Millisecond)
    if acc.CompareAndSet("alice", balance, balance-castle.price) {
        fmt.Println("Alice bought the castle")
    }
}()

// Alice buys plants.
go func() {
    defer wg.Done()
    balance := acc.Get("alice")
    if balance < plants.price {
        return
    }
    time.Sleep(10 * time.Millisecond)
    if acc.CompareAndSet("alice", balance, balance-plants.price) {
        fmt.Println("Alice bought the plants")
    }
}()
Alice bought the castle
Alice's balance: 10

We no longer use a mutex to protect the entire purchase. Individual operations from different goroutines can get mixed up, but using compare-and-set instead of a regular update protects us from the race condition. If the actual account state doesn't match what we expect (because another goroutine made a change), the update won't happen.

CAS with retry

In practice, a failed compare-and-set is often followed by a retry. In our example, we would re-read the balance with acc.Get and, if there is still enough money, retried the purchase with acc.CompareAndSet:

// Alice buys plants (with retries).
go func() {
    defer wg.Done()
    for { // Start of retry loop.
        balance := acc.Get("alice")
        if balance < plants.price {
            return // Not enough money, exit loop.
        }
        // Attempt the purchase.
        if acc.CompareAndSet("alice", balance, balance-plants.price) {
            fmt.Println("Alice bought the plants")
            return // Success, exit loop.
        }
        // Wait a bit before trying again.
        time.Sleep(time.Millisecond)
        // It's also a good idea to limit the number of retries
        // (not shown here for simplicity) - you don't want
        // the program to get stuck in an infinite loop.
    }
}()

This approach helps the program handle occasional goroutine conflicts over a shared resource.

Compare-and-set is often used in concurrent programming. It comes in different flavors, such as:

// CompareAndSet changes the value to new if the current value equals old.
// Returns true if the value was changed.
CompareAndSet(old, new any) bool

// CompareAndSwap changes the value to new if the current value equals old.
// Returns the old value.
CompareAndSwap(old, new any) any

// CompareAndDelete deletes the value if the current value equals old.
// Returns true if the value was deleted.
CompareAndDelete(old any) bool

// etc

The idea is always the same:

  • Check if the assumed (old) state matches reality.
  • If it does, change the state to new.
  • If not, do nothing.

Go's standard library provides compare-and-swap operations for basic types like bool and int64. We'll cover these in another chapter.

If compare-and-set doesn't work for your situation, you can always use a regular shared mutex instead. It can protect any sequence of operations, no matter how complex.

✎ Exercise: Concurrent map + 1 more

Practice is crucial in turning abstract knowledge into skills, making theory alone insufficient. The full version of the book contains a lot of exercises — that's why I recommend getting it.

If you are okay with just theory for now, let's continue.

Idempotence and atomicity

Idempotence means that calling an operation on an object multiple times doesn't cause any changes or errors after the first time.

Let's say we have a resource that need to be released after use:

// resource that needs to be freed after use.
type resource struct {
    // ...
}

// free releases the resource.
// Panics if the resource has already been released.
func (r *resource) free() {
    // ...
}

And there's a worker that frees up resources when closed:

// Worker does some work and
// frees resources when closed.
type Worker struct {
    res resource
}

// Work performs some work.
func (w *Worker) Work() {
    // ...
}

// Close frees resources
// and shuts down the worker.
func (w *Worker) Close() {
    w.res.free()
}

Everything works fine until we call Close twice:

func main() {
    w := new(Worker)
    w.Work()
    w.Close()
    w.Close()
    fmt.Println("worker closed")
}
panic: resource is already freed

Let's see how to make Close idempotent so we can safely release the resources.

Boolean flag

Let's add a closed flag to the worker and check it in the Close method:

// Worker does some work and
// frees resources when closed.
type Worker struct {
    res    resource
    closed bool
}

// Close frees resources
// and shuts down the worker.
func (w *Worker) Close() {
    // Ignore repeated calls.
    if w.closed {    // (1)
        return
    }

    w.res.free()     // (2)
    w.closed = true  // (3)
}

Now, calling Close multiple times works fine:

func main() {
    w := new(Worker)
    w.Close()
    w.Close()
    fmt.Println("worker closed")
}
worker closed

But what happens if we call Close simultaneously from different goroutines?

func main() {
    var wg sync.WaitGroup
    wg.Add(2)

    w := new(Worker)
    go func() {
        defer wg.Done()
        w.Close()
    }()

    go func() {
        defer wg.Done()
        w.Close()
    }()

    wg.Wait()
    fmt.Println("worker closed")
}
// panic: resource is already freed
panic: resource is already freed

You won't see panic here often. But it's still there, lurking in the dark.

Panic! You already know why this happens — the check for w.closed ➊ and the following resource cleanup ➋ ➌ aren't atomic. Since goroutines run concurrently, they both pass the if statement and call w.res.free(). Then, one of the goroutines panics.

We need a structure that ensures atomicity and idempotence in a concurrent environment.

select

Why don't we use a closed channel instead of a boolean flag, and use select to close the channel only once?

// Worker does some work and
// frees resources when closed.
type Worker struct {
    res    resource
    closed chan struct{}
}

// NewWorker creates a new worker.
// We need an explicit NewWorker constructor instead of
// new(Worker), because the default value for a channel
// is nil, which doesn't work for our needs.
func NewWorker() *Worker {
    return &Worker{closed: make(chan struct{})}
}

// Close frees resources
// and shuts down the worker.
func (w *Worker) Close() {
    select {
    case <-w.closed:
        // Ignore repeated calls.
        return
    default:
        w.res.free()
        close(w.closed)
    }
}
worker closed

We call Close from two goroutines and get "worker closed". Everything works fine. Then we deploy this code to production, and a week later we get a bug report saying the app sometimes crashes with the "resource is already freed" panic. What's wrong?

The thing is, select does not protect against freeing resources more than once. As we know, select is safe for concurrent use. So, if two goroutines call Close at the same time, they both enter the select and have to pick a case. Since the closed channel isn't closed yet, both goroutines choose the default case. Both call w.res.free(), and one of them panics.

The chances of this happening are pretty low. You could call Close 999 times without any issues, but on the thousandth try, the stars will align just right, both goroutines will hit the default case, and you will get a panic.

It's especially frustrating that the race detector doesn't always catch this kind of issues. In the example above, the race detector might not find anything (depends on the Go version), especially if you comment out w.res.free(). But the race condition is still there, and closing the channel more than once will eventually cause a panic.

Select is not atomic. Choosing a select case and running its body are separate actions, not a single atomic operation. So, if the code inside the case changes shared data, select can cause a race condition. It's important to keep this in mind.

Mutex

Let's go back to the closed boolean flag, but this time protect it with a mutex:

// Worker does some work and
// frees resources when closed.
type Worker struct {
    res    resource
    mu     sync.Mutex
    closed bool
}

// Close frees resources
// and shuts down the worker.
func (w *Worker) Close() {
    // Make the method atomic.
    w.mu.Lock()
    defer w.mu.Unlock()

    // Ignore repeated calls.
    if w.closed {
        return
    }

    w.res.free()
    w.closed = true
}
worker closed

The mutex stays locked for the entire scope of the Close method. This ensures that no matter how many goroutines call Close simultaneously, only one can execute its body at a time.

Checking the closed flag, freeing resources, and updating the closed state all happen as one atomic operation. This makes Close idempotent, so you won't get any panics when releasing resources.

A mutex (or something similar) is the only way to make sure a complex operation is atomic in a concurrent environment. Do not rely on select in these situations.

Speaking of "something similar", for this specific use case — making sure something happens exactly once — Go's standard library provides a handy sync.Once type. We'll cover it in another chapter.

✎ Exercise: Spot the race

Practice is crucial in turning abstract knowledge into skills, making theory alone insufficient. The full version of the book contains a lot of exercises — that's why I recommend getting it.

If you are okay with just theory for now, let's continue.

Locker

Here are the mutex methods we learned about in the last chapter:

type Mutex struct {
    // internal state
}

func (m *Mutex) Lock()
func (m *Mutex) Unlock()
type RWMutex struct {
    // internal state
}

func (rw *RWMutex) Lock()
func (rw *RWMutex) Unlock()

func (rw *RWMutex) RLock()
func (rw *RWMutex) RUnlock()

As you can see, both types have the same Lock and Unlock methods. Go's standard library provides a common interface for them:

// A Locker represents an object that
// can be locked and unlocked.
type Locker interface {
    Lock()
    Unlock()
}

If the "locking mechanism" is defined by the client, and your code just needs to lock or unlock access to shared data, use Locker instead of a specific type:

// ArrayList is a concurrent-safe dynamic array.
type ArrayList struct {
    vals []any
    lock sync.Locker
}

// NewArrayList creates a new empty array.
func NewArrayList(lock sync.Locker) *ArrayList {
    if lock == nil {
        panic("NewArrayList: lock cannot be nil")
    }
    return &ArrayList{vals: []any{}, lock: lock}
}

// Len returns the length of the array.
func (al *ArrayList) Len() int {
    al.lock.Lock()
    defer al.lock.Unlock()
    return len(al.vals)
}

// Append adds an element to the array.
func (al *ArrayList) Append(val any) {
    al.lock.Lock()
    defer al.lock.Unlock()
    al.vals = append(al.vals, val)
}

This way, the client can use Mutex, RWMutex, or any other implementation they prefer:

func main() {
    var wg sync.WaitGroup

    var lock sync.Mutex
    list := NewArrayList(&lock)

    // Add 400 elements to the array using 4 goroutines.
    for i := 0; i < 4; i++ {
        wg.Add(1)
        go func() {
            defer wg.Done()
            for range 100 {
                list.Append(rand.IntN(100))
                time.Sleep(time.Millisecond)
            }
        }()
    }

    wg.Wait()
    fmt.Println("list length =", list.Len())
}
list length = 400

By using sync.Locker, you can build components that don't depend on a specific lock implementation. This lets the client decide which lock to use. While you probably won't need this very often, it's a useful feature to be aware of.

TryLock

Let's say our program needs to call a legacy system, represented by the External type. This system is so ancient that it can handle no more than one call at a time. That's why we protect it with a mutex:

// External is a client for an external system.
type External struct {
    lock sync.Mutex
}

// Call calls the external system.
func (e *External) Call() {
    e.lock.Lock()
    defer e.lock.Unlock()
    // Simulate a remote call.
    time.Sleep(100 * time.Millisecond)
}

Now, no matter how many goroutines try to access the external system at the same time, they'll have to take turns:

func main() {
    var wg sync.WaitGroup

    ex := new(External)
    start := time.Now()

    const nCalls = 4
    for range nCalls {
        wg.Add(1)
        go func() {
            defer wg.Done()
            ex.Call()
            fmt.Println("success")
        }()
    }

    wg.Wait()
    fmt.Printf(
        "%d calls took %d ms\n",
        nCalls, time.Since(start).Milliseconds(),
    )
}
success
success
success
success
4 calls took 400 ms

Everything looks good on paper. But in reality, if there are a lot of these goroutines, the external system will be constantly busy handling all these sequential calls. This could end up being too much for it. So, let's change the approach:

  • If a goroutine needs to call an external system,
  • and that system is already busy,
  • then the goroutine shouldn't wait,
  • but should immediately return an error.

We can use the TryLock method of a mutex to implement this logic:

// External is a client for an external system.
type External struct {
	lock sync.Mutex
}

// Call calls the external system.
func (e *External) Call() error {
	if !e.lock.TryLock() {
		return errors.New("busy")  // (1)
	}
	defer e.lock.Unlock()
	// Simulate a remote call.
	time.Sleep(100 * time.Millisecond)
	return nil
}

TryLock tries to lock the mutex, just like a regular Lock. But if it can't, it returns false right away instead of blocking the goroutine. This way, we can immediately return an error at ➊ instead of waiting for the system to become available.

Now, out of four simultaneous calls, only one will go through. The others will get a "busy" error:

func main() {
	var wg sync.WaitGroup

	ex := new(External)
	start := time.Now()

	const nCalls = 4
	for range nCalls {
		wg.Add(1)
		go func() {
			defer wg.Done()
			err := ex.Call()
			if err != nil {
				fmt.Println(err)
			} else {
				fmt.Println("success")
			}
		}()
	}

	wg.Wait()
	fmt.Printf(
		"%d calls took %d ms\n",
		nCalls, time.Since(start).Milliseconds(),
	)
}
busy
busy
busy
success
4 calls took 100 ms

According to the standard library docs, TryLock is rarely needed. In fact, using it might mean there's a problem with your program's design. For example, if you're calling TryLock in a busy-wait loop ("keep trying until the resource is free") — that's usually a bad sign:

for {
    if mutex.TryLock() {
        // Use the shared resource.
        mutex.Unlock()
        break
    }
}

This code will keep one CPU core at 100% usage until the mutex is unlocked. It's much better to use a regular Lock so the scheduler can take the blocked goroutine off the CPU.

✎ Exercise: Rate limiter

Practice is crucial in turning abstract knowledge into skills, making theory alone insufficient. The full version of the book contains a lot of exercises — that's why I recommend getting it.

If you are okay with just theory for now, let's continue.

Shared nothing

Let's go back one last time to Alice and the Lego sets we started the chapter with.

We manage user accounts:

// Accounts - money in users' accounts.
type Accounts struct {
	bal map[string]int
	mu  sync.Mutex
}

// NewAccounts creates a new set of accounts.
func NewAccounts(bal map[string]int) *Accounts {
	return &Accounts{bal: maps.Clone(bal)}
}

// Get returns the user's balance.
func (a *Accounts) Get(name string) int {
	a.mu.Lock()          // (1)
	defer a.mu.Unlock()
	return a.bal[name]
}

// Set changes the user's balance.
func (a *Accounts) Set(name string, amount int) {
	a.mu.Lock()          // (2)
	defer a.mu.Unlock()
	a.bal[name] = amount
}

And handle purchases:

acc := NewAccounts(map[string]int{
    "alice": 50,
})
castle := LegoSet{name: "Castle", price: 40}
plants := LegoSet{name: "Plants", price: 20}

// Shared mutex.
var mu sync.Mutex

// Alice buys a castle.
go func() {
    defer wg.Done()

    // Protect the entire purchase with a mutex.
    mu.Lock()            // (3)
    defer mu.Unlock()

    // Check and update the balance.
}()

// Alice buys plants.
go func() {
    defer wg.Done()

    // Protect the entire purchase with a mutex.
    mu.Lock()            // (4)
    defer mu.Unlock()

    // Check and update the balance.
}()

This isn't a very complex use case — I'm sure you've seen worse. Still, we had to put in some effort:

  • Protect the balance with a mutex to prevent a data race ➊ ➋.
  • Protect the entire purchase operation with a mutex (or use compare-and-set) to make sure the final state is correct ➌ ➍.

We were lucky to notice and prevent the race condition during a purchase. What if we had missed it?

There's another approach to achieving safe concurrency: instead of protecting shared state when working with multiple goroutines, we can avoid shared state altogether. Channels can help us do this.

Here's the idea: we'll create a Processor function that accepts purchase requests through an input channel, processes them, and sends the results back through an output channel:

// A purchase request.
type Request struct {
    buyer string
    set   LegoSet
}

// A purchase result.
type Purchase struct {
    buyer   string
    set     LegoSet
    succeed bool
    balance int // balance after purchase
}

// Processor handles purchases.
func Processor(acc map[string]int) (chan<- Request, <-chan Purchase) {
    // ...
}

Buyer goroutines will send requests to the processor's input channel and receive results (successful or failed purchases) from the output channel:

func main() {
    const buyer = "Alice"
    acc := map[string]int{buyer: 50}

    wishlist := []LegoSet{
        {name: "Castle", price: 40},
        {name: "Plants", price: 20},
    }

    reqs, purs := Processor(acc)

    // Alice buys stuff.
    var wg sync.WaitGroup
    for _, set := range wishlist {
        wg.Add(1)
        go func() {
            defer wg.Done()
            reqs <- Request{buyer: buyer, set: set}
            pur := <-purs
            if pur.succeed {
                fmt.Printf("%s bought the %s\n", pur.buyer, pur.set.name)
                fmt.Printf("%s's balance: %d\n", buyer, pur.balance)
            }
        }()
    }
    wg.Wait()
}
Alice bought the Plants
Alice's balance: 30

This approach offers several benefits:

  • Buyer goroutines send their requests and get results without worrying about how the purchase is done.
  • All the buying logic is handled inside the processor goroutine.
  • No need for mutexes.

All that's left is to implement the processor. How about this:

// Processor handles purchases.
func Processor(acc map[string]int) (chan<- Request, <-chan Purchase) {
    in := make(chan Request)
    out := make(chan Purchase)
    acc = maps.Clone(acc)

    go func() {
        for {
            // Receive the purchase request.
            req := <-in

            // Handle the purchase.
            balance := acc[req.buyer]
            pur := Purchase{buyer: req.buyer, set: req.set, balance: balance}
            if balance >= req.set.price {
                pur.balance -= req.set.price
                pur.succeed = true
                acc[req.buyer] = pur.balance
            } else {
                pur.succeed = false
            }

            // Send the result.
            out <- pur
        }
    }()

    return in, out
}

It would have been a good idea to add a way to stop the processor using context, but I decided not to do it to keep the code simple.

The processor clones the original account states and works with its own copy. This approach makes sure there is no concurrent access to the accounts, so there are no races. Of course, we should avoid running two processors at the same time, or we could end up with two different versions of the truth.

It's not always easy to structure a program in a way that avoids shared state. But if you can, it's a good option.

Keep it up

Now you know how to protect shared data (from data races) and sequences of operations (from race conditions) in a concurrent environment using mutexes. Be careful with them and always test your code thoroughly with the race detector enabled.

Use code reviews, because the race detector doesn't catch every data race and can't detect race conditions at all. Having someone else look over your code can be really helpful.

In the next chapter, we'll talk about semaphores.

Pre-order for $10   or read online

]]>
Gist of Go: Data raceshttps://antonz.org/go-concurrency/data-races/Sun, 08 Jun 2025 17:00:00 +0000https://antonz.org/go-concurrency/data-races/Two goroutines racing for the same data is a recipe for disaster.This is a chapter from my book on Go concurrency, which teaches the topic from the ground up through interactive examples.

What happens if multiple goroutines modify the same data structure? Sadly, nothing good. Let's learn more about it.

Concurrent modification

So far, our goroutines haven't gotten in each other's way. They've used channels to exchange data, which is safe. But what happens if several goroutines try to access the same object at the same time? Let's find out.

Let's write a program that counts word frequencies:

func main() {
    // generate creates 100 words, each 3 letters long,
    // and sends them to the channel.
    in := generate(100, 3)

    var wg sync.WaitGroup
    wg.Add(2)

    // count reads words from the input channel
    // and counts how often each one appears.
    count := func(counter map[string]int) {
        defer wg.Done()
        for word := range in {
            counter[word]++
        }
    }

    counter := map[string]int{}
    go count(counter)
    go count(counter)
    wg.Wait()

    fmt.Println(counter)
}
fatal error: concurrent map writes

goroutine 1 [sync.WaitGroup.Wait]:
sync.runtime_SemacquireWaitGroup(0x140000021c0?)

goroutine 34 [chan send]:
main.generate.func1()

goroutine 35 [running]:
internal/runtime/maps.fatal({0x104b4039e?, 0x14000038a08?})

goroutine 36 [runnable]:
internal/runtime/maps.newTable(0x104b78340, 0x80, 0x0, 0x0)
What is generate
// generate creates nWords words, each wordLen letters long,
// and sends them to the channel.
func generate(nWords, wordLen int) <-chan string {
    out := make(chan string)
    go func() {
        defer close(out)
        for ; nWords > 0; nWords-- {
            out <- randomWord(wordLen)
        }
    }()
    return out
}

// randomWord returns a random word with n letters.
func randomWord(n int) string {
    const vowels = "eaiou"
    const consonants = "rtnslcdpm"
    chars := make([]byte, n)
    for i := 0; i < n; i += 2 {
        chars[i] = consonants[rand.IntN(len(consonants))]
    }
    for i := 1; i < n; i += 2 {
        chars[i] = vowels[rand.IntN(len(vowels))]
    }
    return string(chars)
}

generate() generates words and sends them to the in channel. main() creates an empty map called counter and passes it to two count() goroutines. count() reads from the in channel and fills the map with word counts. In the end, counter should contain the frequency of each word.

Let's run it:

map[cec:1 ... nol:2 not:3 ... tut:1]

And once again, just in case:

fatal error: concurrent map writes

goroutine 1 [sync.WaitGroup.Wait]:
sync.runtime_SemacquireWaitGroup(0x140000021c0?)

goroutine 34 [chan send]:
main.generate.func1()

goroutine 35 [running]:
internal/runtime/maps.fatal({0x104b4039e?, 0x14000038a08?})

goroutine 36 [runnable]:
internal/runtime/maps.newTable(0x104b78340, 0x80, 0x0, 0x0)

Panic!

Go doesn't let multiple goroutines write to a map at the same time. At first, this might seem odd. Here's the only operation that the count() goroutine does with the map:

counter[word]++

Looks like an atomic action. Why not perform it from multiple goroutines?

The problem is that the action only seems atomic. The operation "increase the key value in the map" actually involves several smaller steps. If one goroutine does some of these steps and another goroutine does the rest, the map can get messed up. That's what the runtime is warning us about.

Data race

When multiple goroutines access the same variable at the same time, and at least one of them changes it, it's called a data race. Concurrent map modification in the previous section is an example of a data race.

A data race doesn't always cause a runtime panic (the map example in the previous section is a nice exception: Go's map implementation has built-in runtime checks that can catch some data races). That's why Go provides a special tool called the race detector. You can turn it on with the race flag, which works with the test, run, build, and install commands.

To use Go's race detector, you'll need to install gcc, the C compiler.

For example, take this program:

func main() {
    var total int

    var wg sync.WaitGroup
    wg.Add(2)

    go func() {
        defer wg.Done()
        total++
    }()

    go func() {
        defer wg.Done()
        total++
    }()

    wg.Wait()
    fmt.Println(total)
}
2

At first glance, it seems to work correctly. But actually, it has a data race:

go run -race race.go
==================
WARNING: DATA RACE
Read at 0x00c000112038 by goroutine 6:
  main.main.func1()
      race.go:16 +0x74

Previous write at 0x00c000112038 by goroutine 7:
  main.main.func2()
      race.go:21 +0x84

Goroutine 6 (running) created at:
  main.main()
      race.go:14 +0x104

Goroutine 7 (finished) created at:
  main.main()
      race.go:19 +0x1a4
==================
2
Found 1 data race(s)

If you're wondering why a data race is a problem for a simple operation like total++ — we'll cover it later in the chapter on atomic operations.

Channels, on the other hand, are safe for concurrent reading and writing, and they don't cause data races:

func main() {
    ch := make(chan int, 2)

    var wg sync.WaitGroup
    wg.Add(2)

    go func() {
        defer wg.Done()
        ch <- 1
    }()

    go func() {
        defer wg.Done()
        ch <- 1
    }()

    wg.Wait()
    fmt.Println(<-ch + <-ch)
}
2

Data races are dangerous because they're hard to spot. Your program might work fine a hundred times, but on the hundred and first try, it could give the wrong result. Always check your code with a race detector.

✎ Exercise: Spot the race

Practice is crucial in turning abstract knowledge into skills, making theory alone insufficient. The full version of the book contains a lot of exercises — that's why I recommend getting it.

If you are okay with just theory for now, let's continue.

Sequential modification

You can often rewrite a program to avoid concurrent modifications. Here is a possible approach for our word frequency program:

  • Each count() goroutine counts frequencies in its own map.
  • A separate merge() function goes through the frequency maps and builds the final map.
func main() {
    // generate creates 100 words, each 3 letters long,
    // and sends them to the channel.
    in := generate(100, 3)

    var wg sync.WaitGroup
    wg.Add(2)

    // count reads words from the input channel
    // and counts how often each one appears.
    count := func(counters []map[string]int, idx int) {
        defer wg.Done()
        counter := map[string]int{}
        for word := range in {
            counter[word]++
        }
        counters[idx] = counter
    }

    counters := make([]map[string]int, 2)
    go count(counters, 0)
    go count(counters, 1)
    wg.Wait()

    // merge combines frequency maps.
    counter := merge(counters...)
    fmt.Println(counter)
}

// merge combines frequency maps into a single map.
func merge(counters ...map[string]int) map[string]int {
    merged := map[string]int{}
    for _, counter := range counters {
        for word, freq := range counter {
            merged[word] += freq
        }
    }
    return merged
}
map[cec:1 ... nol:2 not:3 ... tut:1]

Technically, we're still using shared data — the counters slice in count(). But the idx parameter makes sure the first count() only works with the first element of the slice, and the second one only works with the second element. Go allows this kind of concurrent slice access.

It's important that multiple goroutines don't try to change the same element of the slice or call append().

Even if Go didn't allow concurrent access to slices, we could still solve the problem. We would just use a channel of maps instead of a slice:

  • Each count() goroutine counts frequencies in its own map and sends it to a shared channel.
  • A separate merge() function in the main goroutine reads frequency maps from the shared channel and builds the final map.

Either way, using a separate merge() step works, but it's not always convenient. Sometimes, we want to modify the same data from multiple goroutines. As is our right.

Let's see how we can do it.

Mutex

The sync package has a special tool called a mutex. It protects shared data and parts of your code (critical sections) from being accessed concurrently:

func main() {
    // generate creates 100 words, each 3 letters long,
    // and sends them to the channel.
    in := generate(100, 3)

    var wg sync.WaitGroup
    wg.Add(2)

    // count reads words from the input channel
    // and counts how often each one appears.
    count := func(lock *sync.Mutex, counter map[string]int) {
        defer wg.Done()
        for word := range in {
            lock.Lock()       // (2)
            counter[word]++
            lock.Unlock()     // (3)
        }
    }

    var lock sync.Mutex       // (1)
    counter := map[string]int{}
    go count(&lock, counter)
    go count(&lock, counter)
    wg.Wait()

    fmt.Println(counter)
}
map[cec:1 ... nol:2 not:3 ... tut:1]

The mutex guarantees that only one goroutine can run the code between Lock() and Unlock() at a time. Here's how it works:

  1. In ➊, we create a mutex and pass it to both count() goroutines.
  2. In ➋, the first goroutine locks the mutex, then runs counter[word]++.
  3. If the second goroutine reaches ➋ at this time, it will block because the mutex is locked.
  4. In ➌, the first goroutine unlocks the mutex.
  5. Now the second goroutine is unblocked. It then locks the mutex and runs counter[word]++.

This way, counter[word]++ can't be run by multiple goroutines at the same time. Now the map won't get corrupted:

$ go run -race counter.go
map[cec:1 ... nol:2 not:3 ... tut:1]

A mutex is used in these situations:

  • When multiple goroutines are changing the same data.
  • When one goroutine is changing data and others are reading it.

If all goroutines are only reading the data, you don't need a mutex.

Unlike some other languages, a mutex in Go is not reentrant. If a goroutine calls Lock() on a mutex it already holds, it will block itself:

func main() {
    var lock sync.Mutex

    lock.Lock()
    // ok

    lock.Lock()
    // fatal error: all goroutines are asleep - deadlock!
}
fatal error: all goroutines are asleep - deadlock!

goroutine 1 [sync.Mutex.Lock]:
internal/sync.runtime_SemacquireMutex(0x4641f9?, 0x80?, 0xc00006cf40?)
    /usr/local/go/src/runtime/sema.go:95 +0x25
internal/sync.(*Mutex).lockSlow(0xc000010060)
    /usr/local/go/src/internal/sync/mutex.go:149 +0x15d
internal/sync.(*Mutex).Lock(...)
    /usr/local/go/src/internal/sync/mutex.go:70
sync.(*Mutex).Lock(...)
    /usr/local/go/src/sync/mutex.go:46
main.main()
    /sandbox/src/main.go:45 +0x5f (exit status 2)

This makes things harder for people who like to use mutexes in recursive functions (which isn't a great idea anyway).

Like a wait group, a mutex has internal state, so you should only pass it as a pointer.

✎ Exercise: Concurrent-safe counter

Practice is crucial in turning abstract knowledge into skills, making theory alone insufficient. The full version of the book contains a lot of exercises — that's why I recommend getting it.

If you are okay with just theory for now, let's continue.

Read-write mutex

A regular mutex doesn't distinguish between read and write access: if one goroutine locks the mutex, others can't access the protected code. This isn't always necessary.

Here's the situation:

  • One writer goroutine writes data.
  • Four reader goroutines read that same data.
var wg sync.WaitGroup
wg.Add(5)

var lock sync.Mutex

// writer fills in the word frequency map.
writer := func(counter map[string]int, nWrites int) {
    defer wg.Done()
    for ; nWrites > 0; nWrites-- {
        word := randomWord(3)
        lock.Lock()
        counter[word]++
        time.Sleep(time.Millisecond)
        lock.Unlock()
    }
}

// reader looks up random words in the frequency map.
reader := func(counter map[string]int, nReads int) {
    defer wg.Done()
    for ; nReads > 0; nReads-- {
        word := randomWord(3)
        lock.Lock()
        _ = counter[word]
        time.Sleep(time.Millisecond)
        lock.Unlock()
    }
}

start := time.Now()

counter := map[string]int{}
go writer(counter, 100)
go reader(counter, 100)
go reader(counter, 100)
go reader(counter, 100)
go reader(counter, 100)
wg.Wait()

fmt.Println("Took", time.Since(start))
Took 500ms

Even though we started 4 reader goroutines, they run sequentially because of the mutex. This isn't really necessary. It makes sense for readers to wait while the writer is updating the map. But why can't the readers run in parallel? They're not changing any data.

The sync package includes a sync.RWMutex that separates readers and writers. It provides two sets of methods:

  • Lock / Unlock lock and unlock the mutex for both reading and writing.
  • RLock / RUnlock lock and unlock the mutex for reading only.

Here's how it works:

  • If a goroutine locks the mutex with Lock(), other goroutines will be blocked if they try to use Lock() or RLock().
  • If a goroutine locks the mutex with RLock(), other goroutines can also lock it with RLock() without being blocked.
  • If at least one goroutine has locked the mutex with RLock(), other goroutines will be blocked if they try to use Lock().

This creates a "single writer, multiple readers" setup. Let's verify it:

var wg sync.WaitGroup
wg.Add(5)

var lock sync.RWMutex          // (1)

// writer fills in the word frequency map.
writer := func(counter map[string]int, nWrites int) {
    // Not changed.
    defer wg.Done()
    for ; nWrites > 0; nWrites-- {
        word := randomWord(3)
        lock.Lock()
        counter[word]++
        time.Sleep(time.Millisecond)
        lock.Unlock()
    }
}

// reader looks up random words in the frequency map.
reader := func(counter map[string]int, nReads int) {
    defer wg.Done()
    for ; nReads > 0; nReads-- {
        word := randomWord(3)
        lock.RLock()           // (2)
        _ = counter[word]
        time.Sleep(time.Millisecond)
        lock.RUnlock()         // (3)
    }
}

start := time.Now()

counter := map[string]int{}
go writer(counter, 100)
go reader(counter, 100)
go reader(counter, 100)
go reader(counter, 100)
go reader(counter, 100)
wg.Wait()

fmt.Println("Took", time.Since(start))
Took 200ms

The mutex type ➊ has changed, so have the locking ➋ and unlocking ➌ methods in the reader. Now, readers run concurrently, but they always wait while the writer updates the map. That's exactly what we need!

✎ Exercise: Counter with RWMutex

Practice is crucial in turning abstract knowledge into skills, making theory alone insufficient. The full version of the book contains a lot of exercises — that's why I recommend getting it.

If you are okay with just theory for now, let's continue.

Channel as mutex

Let's go back to the program that counts word frequencies:

func main() {
    // generate creates 100 words, each 3 letters long,
    // and sends them to the channel.
    in := generate(100, 3)

    var wg sync.WaitGroup
    wg.Add(2)

    // count reads words from the input channel
    // and counts how often each one appears.
    count := func(lock *sync.Mutex, counter map[string]int) {
        defer wg.Done()
        for word := range in {
            lock.Lock()       // (2)
            counter[word]++
            lock.Unlock()     // (3)
        }
    }

    var lock sync.Mutex       // (1)
    counter := map[string]int{}
    go count(&lock, counter)
    go count(&lock, counter)
    wg.Wait()

    fmt.Println(counter)
}

We created the lock mutex ➊ and used it to protect access to the shared counter map ➋ ➌. This way, the count() goroutines don't cause data races, and the final counter[word] value is correct.

We can also use a channel instead of a mutex to protect shared data:

type token struct{}

func main() {
    // generate creates 100 words, each 3 letters long,
    // and sends them to the channel.
    in := generate(100, 3)

    var wg sync.WaitGroup
    wg.Add(2)

    // count reads words from the input channel
    // and counts how often each one appears.
    count := func(lock chan token, counter map[string]int) {
        defer wg.Done()
        for word := range in {
            lock <- token{}     // (2)
            counter[word]++
            <-lock              // (3)
        }
    }

    lock := make(chan token, 1) // (1)

    counter := map[string]int{}
    go count(lock, counter)
    go count(lock, counter)
    wg.Wait()

    fmt.Println(counter)
}
map[cec:1 ... nol:2 not:3 ... tut:1]

We created a lock channel with a one-element buffer ➊ and used it to protect access to the shared counter map ➋ ➌.

Two count() goroutines run concurrently. However, in each loop iteration, only one of them can put a token into the lock channel (like locking a mutex), update the counter, and take the token back out (like unlocking a mutex). So, even though the goroutines run in parallel, changes to the map happen sequentially.

As a result, the count() goroutines don't cause data races, and the final counter[word] value is correct — just like when we used a mutex.

Go's channels are a versatile concurrency tool. Often, you can use a channel instead of lower-level synchronization primitives. Sometimes, using a channel is unnecessary, as in the example above. Other times, however, it makes your code simpler and helps prevent mistakes. You'll see this idea come up again throughout the book.

Keep it up

Now you know how to safely change shared data from multiple goroutines using mutexes. Be careful not to overuse them — it's easy to make mistakes and cause data races or deadlocks.

In the next chapter, we'll talk about race conditions.

Pre-order for $10   or read online

]]>
AI-free writinghttps://antonz.org/ai-free/Sat, 07 Jun 2025 10:00:00 +0000https://antonz.org/ai-free/I never use AI-generated content in my writing, and I never will.I've been following a rule for a while now, and I want to share it. I never use AI-generated content in my writing, and I never will.

I'm not a Luddite. I use AI extensively for tasks like generating unit tests, reviewing code, and discussing system design. I recently started using coding agents and plan to use them even more once they improve.

Still, I don't want AI to write prose for me. It might be silly, but I prefer to write my blog posts and books entirely by hand. Sure, that means I'll never be as productive as authors who publish AI-generated content. Or cover as many topics. So be it.

We're quickly moving into the era of AI-generated content for marketing, education, and perhaps even fiction. I don't think AI writing is inherently bad, nor do I think that people who create AI-heavy content are wrong. But I want to stay out of it.

Will readers still be interested in AI-free writing five years from now? I really hope so. But if they aren't, that's okay.

My rule of thumb: writing is an AI-free zone.

]]>
Sad story of http.DefaultTransporthttps://antonz.org/default-transport/Sat, 24 May 2025 05:00:00 +0000https://antonz.org/default-transport/A tale of false flexibility and leaking abstractions.Even if you're an experienced Go developer, this expression might make you raise an eyebrow for a moment:

http.DefaultTransport.(*http.Transport)

Interestingly, it shows a couple of not-so-common features in Go's standard library. Let's review them.

http.DefaultTransport

Packages in Go mainly consist of:

  • Functions, like http.StatusText(), which gives the text description for a numeric HTTP status code.
  • Types, such as http.Client, which is used to make HTTP requests.
  • Constants, like http.StatusOK, which equals 200 and means the request was successful.

Less commonly, packages export variables. Typically, these are errors, like http.ErrNotSupported. But sometimes, they are something else.

http.DefaultTransport is one of those rare package variables that isn't an error. Basically, it’s an implementation of the HTTP protocol (called "transport") with default settings.

Since the transport in the http package is represented by the Transport type, it makes sense to define DefaultTransport like this:

var DefaultTransport = &Transport{
    // default settings
}

But in reality, it looks like this:

var DefaultTransport RoundTripper = &Transport{
    // default settings
}

Although DefaultTransport is in fact a pointer to a Transport value, it's declared as a RoundTripper interface. We'll look into this mystery a bit later.

http.RoundTripper

RoundTripper is an interface with a single RoundTrip method:

type RoundTripper interface {
    // RoundTrip executes a single HTTP transaction,
    // returning a Response for the provided Request.
    RoundTrip(*Request) (*Response, error)
}

The http.Client type delegates the actual request execution to its Transport property, which is of type RoundTripper:

type Client struct {
    // Transport specifies the mechanism by which
    // individual HTTP requests are made.
    // If nil, DefaultTransport is used.
    Transport RoundTripper
    // ...
}

This is flexible and convenient. To change how the transport works, we just need to implement the RoundTrip method in a custom type:

type DummyTransport struct{}

func (t *DummyTransport) RoundTrip(*http.Request) (*http.Response, error) {
    return nil, errors.New("not implemented")
}

Then the Client will use it:

func main() {
    c := &http.Client{}
    c.Transport = &DummyTransport{}
    resp, err := c.Get("http://example.com")
    fmt.Println(resp, err)
}
<nil> Get "http://example.com": not implemented

So, why is DefaultTransport declared as a RoundTripper? We could have just declared it as *Transport and used it wherever a RoundTripper is needed — since *Transport implements the RoundTripper interface, that's okay in Go.

Why RoundTripper?

Why is DefaultTransport declared as a RoundTripper interface instead of a specific *Transport type, even though it actually holds a *Transport value?

var DefaultTransport RoundTripper = &Transport{...}

You won't find an answer in the official documentation, so here's my take.

First, it clearly shows that DefaultTransport is what the Client expects in its Transport field — a RoundTripper. But, since *Transport implements the RoundTripper interface, this isn't really necessary — it just makes the intention clearer.

Second, it serves as a safety check. If *Transport no longer meets the RoundTripper interface, the DefaultTransport declaration won't compile. It's also not strictly necessary — the client already uses DefaultTransport as RoundTripper in its transport method:

func (c *Client) transport() RoundTripper {
	if c.Transport != nil {
		return c.Transport
	}
	return DefaultTransport
}

If at any point *Transport stops matching the RoundTripper interface, the code won't compile, no matter how DefaultTransport is declared. But, of course, it's more reliable to check the type in the DefaultTransport declaration — who knows what might happen to the transport method in the future.

Finally, by making DefaultTransport an interface, the stdlib developers left themselves the option to replace the implementation in the future. For example, they could create a new MagicTransport instead of Transport without breaking backward compatibility.

But in reality, this will never happen.

Transport leak

DefaultTransport is a classic example of a leaked abstraction.

In many codebases, DefaultTransport is cast to *Transport to set up the transport options. This is often done with an unsafe type assertion (there's even an example of this in the stdlib documentation):

t := http.DefaultTransport.(*http.Transport).Clone()
// Now you can change transport properties.
t.TLSHandshakeTimeout = time.Second
t.DisableKeepAlives = true

If the stdlib developers replace the implementation of DefaultTransport with a different type, this code will panic. So, in practice, such changes won't happen.

I think it was not a great idea to declare DefaultTransport as a RoundTripper. It would be better if it were *http.Transport, so we wouldn't have to do type assertions all the time. If the stdlib developers wanted an interface guard for *Transport, they could just state it separately:

var _ RoundTripper = (*Transport)(nil)

Final thoughts

To improve the situation, it would be helpful to have a http.NewDefaultTransport function that returns a specific type:

func NewDefaultTransport() *Transport {
    return &Transport{...}
}

var DefaultTransport RoundTripper = NewDefaultTransport()

Then we can just call NewDefaultTransport() instead of the messy http.DefaultTransport.(*http.Transport).Clone().

However, the Go team shows no intention of implementing this proposal.

Oh well.

──

P.S. Want to learn more about Go? Check out my book on concurrency

]]>
Am I online?https://antonz.org/is-online/Thu, 15 May 2025 05:00:00 +0000https://antonz.org/is-online/Checking internet connectivity with 'generate 204' endpoints.Recently, I was working on an application that needed to know if it was connected to the internet. A common way to do this is to ping DNS servers like 8.8.8.8 (Google) or 1.1.1.1 (Cloudflare). However, this uses the ICMP protocol (which only checks for basic network connectivity), while I wanted to exercise the full stack used by real HTTP clients: DNS, TCP, and HTTP.

Generate 204

After some research, I found this URL that Google itself seems to use in Chrome to check for connectivity:

http://google.com/generate_204
https://google.com/generate_204

The URL returns a 204 No Content HTTP status (a successful response without a body). It's super fast, relies only on the core Google infrastructure (so it's unlikely to fail), and supports both HTTP and HTTPS. So I went with it, and it turned out to be sufficient for my needs.

There are also http://www.gstatic.com/generate_204 and http://clients3.google.com/generate_204. As far as I can tell, they are served by the same backend as the one on google.com.

Other companies provide similar URLs to check for connectivity:

  • http://cp.cloudflare.com/generate_204 (Cloudflare)
  • http://edge-http.microsoft.com/captiveportal/generate_204 (Microsoft)
  • http://connectivity-check.ubuntu.com (Ubuntu)
  • http://connect.rom.miui.com/generate_204 (Xiaomi)

200 OK

Some companies provide 200 OK endpoints instead of 204 No Content:

  • http://spectrum.s3.amazonaws.com/kindle-wifi/wifistub.html (Amazon)
  • http://captive.apple.com/hotspot-detect.html (Apple)
  • http://network-test.debian.org/nm (Debian)
  • http://nmcheck.gnome.org/check_network_status.txt (Gnome)
  • http://www.msftncsi.com/ncsi.txt (Microsoft)
  • http://detectportal.firefox.com/success.txt (Mozilla)

They are all reasonably fast and return compact responses.

Implementation

Finally, here's a simple internet connectivity check implemented in several programming languages. It uses Google's URL, but you can replace it with any of the others listed above.

Python:

import datetime as dt
import http.client

def is_online(timeout: dt.timedelta = dt.timedelta(seconds=1)) -> bool:
    """Checks if there is an internet connection."""
    try:
        conn = http.client.HTTPConnection(
            "google.com",
            timeout=timeout.total_seconds(),
        )
        conn.request("GET", "/generate_204")
        response = conn.getresponse()
        return response.status in (200, 204)
    except Exception:
        return False
    finally:
        conn.close()

JavaScript:

// isOnline checks if there is an internet connection.
async function isOnline(timeoutMs) {
    try {
        const url = "http://google.com/generate_204";
        const response = await fetch(url, {
            signal: AbortSignal.timeout(timeoutMs ?? 1000),
        });
        return response.status === 200 || response.status === 204;
    } catch (error) {
        return false;
    }
}

Shell:

#!/usr/bin/env sh

# Checks if there is an internet connection.
is_online() {
    local url="http://google.com/generate_204"
    local timeout=${1:-1}
    local response=$(
        curl \
        --output /dev/null \
        --write-out "%{http_code}" \
        --max-time "$timeout" \
        --silent \
        "$url"
    )
    if [ "$response" = "200" ] || [ "$response" = "204" ]; then
        return 0
    else
        return 1
    fi
}

Go:

package main

import (
    "context"
    "net/http"
)

// isOnline checks if there is an internet connection.
func isOnline(ctx context.Context) bool {
    const url = "http://google.com/generate_204"

    req, err := http.NewRequestWithContext(ctx, http.MethodGet, url, nil)
    if err != nil {
        return false
    }

    resp, err := http.DefaultClient.Do(req)
    if err != nil {
        return false
    }

    defer resp.Body.Close()
    return resp.StatusCode == http.StatusOK || resp.StatusCode == http.StatusNoContent
}

Final thoughts

I'm not a big fan of Google, but I think it's nice of them to provide a publicly available endpoint to check internet connectivity. The same goes for Cloudflare and the other companies mentioned in this post.

Do you know of other similar endpoints? Let me know! @ohmypy (Twitter/X) or @antonz.org (Bluesky)

]]>
Sandboxeshttps://antonz.org/sandboxes/Mon, 21 Apr 2025 14:30:00 +0000https://antonz.org/sandboxes/From programming languages to databases to networking and CLI tools.I'm a big fan of interactive playgrounds. So today I'm open sourcing 30+ sandboxes — from programming languages to databases to networking and CLI tools. You can use them to experiment locally, or embed interactive examples in your writing like this:

// Java 23

void greet(String name) {
    System.out.println("Hello, %s!".formatted(name));
}

void main() {
    greet("World");
}
Hello, World!

Just look at this Java code, where did all that public-static-class stuff go? Is this even legal? :)

Or like this:

-- ClickHouse 24.8

create table data (id UInt32, x UInt32)
engine MergeTree order by id sample by id
as

select number+1 as id, randUniform(1, 100) as x
from numbers(10000);

select
  avg(x) as "avg",
  round(quantile(0.95)(x), 2) as p95
from data
sample 0.1;
┌─────avg─┬─p95─┐
│ 49.9294 │  95 │
└─────────┴─────┘

Having a sampling feature in your SQL is really nice, isn't it?

What is a sandbox

Sandbox = image + box + commands:

  • Image as a Docker image containing specific software, like a compiler or a database engine.
  • Box is a configuration for running a container: cpu and memory restrictions, network, mounts, etc.
  • Commands are predefined actions you can run in a container.

For example, here is a MariaDB sandbox:

mariadb.tar.gz
├── server
│   ├── Dockerfile
│   ├── database-create.sh
│   ├── database-drop.sh
│   └── init.sql
├── client
│   └── Dockerfile
├── box.json
├── commands.json
├── build.sh
└── setup.sh

You run sandboxes with Codapi (a lightweight sandbox server, essentially a thin layer on top of Docker/Podman).

To add a sandbox, run ./codapi-cli sandbox add <name> like this:

./codapi-cli sandbox add lua
./codapi-cli sandbox add go
./codapi-cli sandbox add mariadb

...and you are good to go.

See github.com/nalgeon/sandboxes for details.

Supported languages and software

Programming languages:

Bash        JavaScript      Raku
C           Kotlin          Ruby
C#          Lua             Rust
C++         Odin            TypeScript
Elixir      PHP             V
Go          Python          Zig
Java        R

Databases:

chDB            MySQL
ClickHouse      PostgreSQL
DuckDB          SQL Server
MariaDB         SQLite
MongoDB

Networking and tools:

Caddy       Grep        ...and a ton of others
Curl        Hurl
Git         Ripgrep

Feel free to contribute new sandboxes or ask me to do so.

Final thoughts

Personally, I use these sandboxes all the time to write blog posts and interactive books. I hope you'll find them useful too!

See the nalgeon/sandboxes repo if you are interested.

]]>
Howto: Humble command-line assistanthttps://antonz.org/howto/Sat, 22 Feb 2025 12:00:00 +0000https://antonz.org/howto/You ask, and howto answers. That's the deal.After many years of working with CLI tools (and even writing books on the subject), my skills are far from perfect. So I created Howto — a simple AI command line assistant to help with all kinds of tasks.

Describe the task, and Howto will suggest a solution:

$ howto curl example.org but print only the headers
curl -I example.org

The `curl` command is used to transfer data from or to a server.
The `-I` option tells `curl` to fetch the HTTP headers only, without the body
content.

Howto does not interfere with your workflow. It's not an alternate shell or "intelligent terminal" or anything. You ask, and Howto answers. That's the deal.

Usage: howto [-h] [-v] [-run] [question]

A humble command-line assistant.

Options:
  -h, --help      Show this help message and exit
  -v, --version   Show version information and exit
  -run            Run the last suggested command
  question        Describe the task to get a command suggestion
                  Use '+' to ask a follow up question

However, Howto has some convenience features that you may find useful. Read on for details.

Follow-ups

If you're not satisfied with an answer, refine it or ask a follow-up question by starting with +:

$ howto a command that works kinda like diff but compares differently
comm file1 file2

The `comm` command compares two sorted files line by line and outputs three
columns: lines unique to the first file, lines unique to the second file, and
lines common to both files.
$ howto + yeah right i need only the intersection
comm -12 file1 file2

The `comm` command compares two sorted files line by line.
The `-12` option suppresses the first and second columns, showing only lines
common to both files (the intersection).

If you don't use +, Howto will forget the previous conversation and treat your question as new.

This feature was a game changer for me. Sometimes the AI doesn't get the answer quite right the first time, but with follow-ups it works very well.

You are not limited to a single follow-up, of course. The exchange can be quite lengthy:

$ howto git revert commit
git revert <commit>

Reverts the changes introduced by the specified commit by creating a new commit
with the inverse changes. This is a safe way to undo changes as it doesn't
rewrite history.
$ howto +no, rollback it
git reset --hard <commit>

Resets the current branch to the specified commit, discarding all changes after
that commit. This rewrites history and should be used with caution, especially
on shared branches.
$ howto +keep the changes
git reset --soft <commit>

Resets the current branch to the specified commit, but keeps the changes as
staged changes. This allows you to modify the changes before creating a new
commit.
$ howto +last commit i mean
git reset --soft HEAD^

Resets the current branch to the commit before the last commit (HEAD^), keeping
the changes from the last commit as staged changes.

It's never easy with Git, I guess.

Run command

When satisfied with the suggested command, run howto -run to execute it without manually copying and pasting:

$ howto curl example.org but print only the headers
curl -I example.org

The `curl` command is used to transfer data from or to a server.
The `-I` option tells `curl` to fetch the HTTP headers only, without the body
content.
$ howto -run
curl -I example.org

HTTP/1.1 200 OK
Content-Type: text/html
ETag: "84238dfc8092e5d9c0dac8ef93371a07:1736799080.121134"
Last-Modified: Mon, 13 Jan 2025 20:11:20 GMT
Cache-Control: max-age=2804
Date: Sun, 09 Feb 2025 12:54:51 GMT
Connection: keep-alive

Dead simple, but works reasonably well.

Configure AI provider

Howto works with any OpenAI-compatible provider (like OpenAI itself, OpenRouter, Nebius, Gemini or Grok) and local Ollama models.

To change the provider, set the HOWTO_AI_URL, HOWTO_AI_TOKEN and HOWTO_AI_MODEL environment variables, and you are good to go.

You can also customize the prompt (HOWTO_PROMPT), although the default works pretty well for me.

Final thoughts

I find Howto quite handy in my daily terminal work. If you are interested, check it out:

brew tap nalgeon/howto https://github.com/nalgeon/howto
brew install howto

Or:

go install github.com/nalgeon/howto@latest

See github.com/nalgeon/howto for installation, configuration and usage details.

]]>