• Go developers who want low-level control without having to learn another language.
• C developers who like Go's style.

At the end of the v0.2 post, I said the obvious goal for the next release was concurrency, along with the stdlib packages that support it. That's what v0.3 is about. So now has threads, channels, worker pools, mutexes, and atomics — enough tools for parallel data processing or handling network connections.

This release also adds a streaming JSON package, a bunch of safety checks (escape analysis, leak checking, nil-pointer panics, stack traces), and proper so test and so bench commands.

Threads • Channels • Worker pools • Sharing state • JSON • Safety net • Tooling • Wrapping up

Threads

The new conc package is the foundation. It provides real OS threads, backed by pthreads. If you're familiar with Go's goroutines, the code will look similar — but there are some important differences.

// greet prints a label three times.
func greet(arg any) any {
    from := arg.(string)
    for i := range 3 {
        println(from, "->", i)
    }
    return nil
}

func main() {
    // Run greet on a separate OS thread, concurrently with main.
    name := "thread"
    th := conc.Go(greet, name)

    // Wait blocks until the thread finishes.
    th.Wait()
    println("done")
}

thread -> 0
thread -> 1
thread -> 2
done

Solod doesn't support closures, so conc.Go takes a function and an any argument, instead of just a func() like you'd expect in Go.

Other important differences: starting an OS thread isn't free, and you always have to Wait on it (or Detach it), or it will leak. That makes conc.Go a good fit for a small, fixed number of long-lived threads — but not for thousands of short-lived tasks. For those cases, it's better to use a pool (shown below).

Channels

Threads in Solod communicate with each other through channels, like goroutines in Go. A channel carries values of a specific type. By default, sending or receiving on a channel blocks until both sides are ready, so a channel also works as a synchronization point.

// ping sends a single message on the given channel.
func ping(arg any) any {
    messages := arg.(*conc.Chan[string])
    messages.Send("ping")
    return nil
}

func main() {
    // An unbuffered channel (buffer size 0): each send blocks
    // until a receiver is ready to take the value.
    messages := conc.NewChan[string](mem.System, 0)
    defer messages.Free()

    // Launch a thread that sends "ping" into the channel.
    th := conc.Go(ping, &messages)
    defer th.Wait()

    // Receive the message and print it.
    var msg string
    messages.Recv(&msg)
    println(msg)
}

ping

A couple of So-specific moments here. When you create a channel, you give it an allocator (mem.System in this case), and you call Free when you're done with it. Also, Recv writes to a pointer you pass in, instead of returning the value directly. It returns a bool, which is false when the channel is closed and empty. So, a typical for msg := range ch loop in Go becomes for ch.Recv(&msg) { ... } in Solod.

Allocators are a key concept in Solod. The language doesn't allow hidden heap allocations, so any function that needs to allocate memory must take an allocator (the mem.Allocator interface) as its first argument.

Buffered channels can hold a limited number of values without having a receiver ready — just pass a non-zero size with NewChan. If you don't want to block forever, use RecvTimeout or SendTimeout with a duration. They return conc.Ok or conc.Timeout instead of getting stuck.

Worker pools

Threads are expensive, so spawning one per task doesn't scale. For handling many short-lived tasks, use conc.Pool: it uses a fixed number of worker threads that take tasks from a queue.

// job holds input and the result.
type job struct {
    id     int
    result int
}

// process handles one job.
func process(arg any) {
    j := arg.(*job)
    time.Sleep(100*time.Millisecond)
    j.result = j.id * 2
}

func main() {
    // A pool of 4 worker threads. Each submitted job is handled
    // by the next available worker.
    pool := conc.NewPool(mem.System, conc.PoolOptions{NumThreads: 4})
    defer pool.Free()

    // Submit 8 jobs. Each writes into its own struct, so keep
    // the structs alive in a slice until the jobs finish.
    start := time.Now()
    jobs := make([]job, 8)
    for i := range jobs {
        jobs[i].id = i + 1
        pool.Go(process, &jobs[i])
    }

    // Wait until all submitted jobs have finished.
    pool.Wait()

    for i := range jobs {
        println("job", jobs[i].id, "->", jobs[i].result)
    }
    elapsed := time.Since(start) / 1_000_000
    println("took", elapsed, "ms")
}

job 1 -> 2
job 2 -> 4
job 3 -> 6
job 4 -> 8
job 5 -> 10
job 6 -> 12
job 7 -> 14
job 8 -> 16
took 200 ms

pool.Wait() works similar to Go's WaitGroup.Wait — it blocks until all submitted jobs are finished. This program takes about 200 ms to run (even though there's 800 ms of total work), because 4 workers run concurrently.

You might think OS threads are much slower than Go's goroutines, but for pools, that's not the case. On realistic workloads, conc.Pool is usually only about 10% slower than Go, whether the tasks are CPU-bound or waiting on I/O. Channels are a different story: handing off work between threads requires a kernel wakeup, while Go does this in user space, so it can be several times slower. Check out Go-flavored concurrency in C for more details.

Sharing state

One way to share state in Solod is by using channels to communicate it. However, sometimes you just need a shared counter or a lock. For that, the new release introduces the sync and sync/atomic packages.

Here's an example of an atomic counter being updated by 50 tasks running on 4 threads:

// increment atomically increases the shared counter 1000 times.
func increment(arg any) {
    ops := arg.(*atomic.Uint64)
    for range 1000 {
        ops.Add(1)
    }
}

func main() {
    // An atomic value is safe for concurrent reads and writes.
    var ops atomic.Uint64

    pool := conc.NewPool(mem.System, conc.PoolOptions{NumThreads: 4})
    defer pool.Free()

    // 50 tasks, each incrementing the counter 1000 times.
    for range 50 {
        pool.Go(increment, &ops)
    }
    pool.Wait()

    println("ops:", ops.Load())
}

ops: 50000

A regular int incremented with ops++ would cause a data race and give a different result each time. Here, the result is exactly 50,000 on every run, thanks to the atomic Uint64 counter. The atomic package provides Int64, Uint64, Bool, and Pointer[T] types, all of which are lock-free and safe for concurrent use.

For anything more complex than a counter, use sync, which provides Mutex, Cond (a condition variable), and Once (runs a function exactly once). One thing to watch out for: unlike Go, a So mutex's zero value isn't ready to use — you need to Init it before locking and Free it when done.

var mu sync.Mutex
mu.Init()
defer mu.Free()

mu.Lock()
defer mu.Unlock()
// ... critical section ...

JSON

Go's encoding/json relies on reflection to marshal arbitrary structs. Solod has no reflection, and uses a different approach: a token-level API. You read and write one JSON token at a time, and the Encoder and Decoder types take care of the syntax — adding commas and colons, checking UTF-8, and rejecting bad input.

Encoding is done through a series of calls that match the structure of your document:

out := make([]byte, 256)
sb := strings.FixedBuilder(out)
enc := json.NewEncoder(&sb)

enc.BeginObject()
enc.Str("name")
enc.Str("Alice")
enc.Str("age")
enc.Int(25)
enc.EndObject()
enc.Flush()

println(sb.String())

{"name":"Alice","age":25}

Decoding pulls one validated token at a time with Next. You can check each token with Kind and read its value using typed getters like Str, Int, or Bool:

src := `{"name":"Alice","age":25}`
dec := json.NewDecoder(mem.System, []byte(src))
defer dec.Free()

var name string
var age int64

dec.Next() // the opening {
for dec.Next() && dec.Kind() == json.KindString {
    switch dec.Str() {
    case "name":
        dec.Next()
        name = dec.Str()
    case "age":
        dec.Next()
        age = dec.Int()
    default:
        dec.Next()
        dec.Skip()
    }
}

println(name, age)

Alice 25

This is a simplified example that works only because the decoder doesn't allocate any memory. In a real-world situation, you'd need to use an allocator.

The decoder works the same way whether you're using an in-memory document (NewDecoder) or reading from a stream with an io.Reader (NewReader). This means you can decode data directly from a source without having to buffer the entire message first. Both the encoder and decoder use minimal memory and will reject invalid JSON or non-UTF-8 strings.

As you can see, API is low-level and not nearly as ergonomic as it is in Go, especially when it comes to decoding. But on the bright side, it's 10 times faster and almost doesn't allocate, unlike in Go.

Safety net

Solod compiles to plain C, which is fast but not very forgiving: if you use an out-of-bounds index, dereference nil, or divide by zero, you get undefined behavior that could crash the program or silently give wrong results. The new release addresses some of these issues.

Escape analysis. Returning a pointer to a stack-allocated value is a classic C footgun. So now catches the common cases at compile time:

type Point struct{ x, y int }

func newPoint(x, y int) *Point {
    return &Point{x: x, y: y}
    //     ^ compile-time error: stack-allocated
    //       value escapes function frame
}

func main() {
    p := newPoint(3, 4)
    println(p.x, p.y)
}

so run: /tmp/sandbox/main.go:26:12: stack-allocated value escapes function frame
    return &Point{x: x, y: y}
           ^here (exit status 1)

While the escape analyzer doesn't catch every case, it's still quite useful in practice. I actually found a couple of dangling pointers in the standard library code with it, even though I was sure there weren't any.

Leak detection. Solod has no garbage collector, so a forgotten Free is a real memory leak. mem.Tracker helps catch these leaks: it wraps an allocator and keeps track of every allocation and free that goes through it. This way, you can monitor the program's memory usage in real time instead of guessing.

Wrap mem.System once, allocate memory through the tracker, and have a background thread log the stats at regular intervals:

// monitor periodically logs live allocation stats.
func monitor(arg any) any {
    t := arg.(*mem.Tracker)
    for {
        time.Sleep(100 * time.Millisecond)
        s := t.Stats()
        println("live:", s.Mallocs-s.Frees, "allocations,", s.Alloc, "bytes")
    }
    return nil
}

func main() {
    // Wrap the system allocator to count every allocation and free.
    heap := &mem.Tracker{Allocator: mem.System}

    // Watch memory from a background thread.
    conc.Go(monitor, heap).Detach()

    // Allocate through heap so the monitor sees it.
    for i := range 10 {
        v := mem.Alloc[int](heap) // intentionally not freeing it
        *v = i
        time.Sleep(50*time.Millisecond)
    }
    // ...
}

live: 2 allocations, 16 bytes
live: 4 allocations, 32 bytes
live: 6 allocations, 48 bytes
live: 8 allocations, 64 bytes
live: 10 allocations, 80 bytes

The tracker is lock-free and only uses a few atomic operations for each allocation, so it's cheap enough to keep enabled in production.

Nil-pointer panics. If you try to dereference a nil pointer, it will cause a panic at runtime instead of a raw segmentation fault:

type Rect struct{ width, height int }

func (r *Rect) area() int {
    return r.width * r.height
    //     ^ runtime error: nil pointer dereference
}

func main() {
    var r *Rect
    println(r.area())
}

panic: nil pointer dereference

Stack traces. When a program panics, the -panic flag controls what happens next:

so run -panic=trace .  # print a stack trace, then exit(1) - the default
so run -panic=exit  .  # just exit(1) after the message
so run -panic=abort .  # raise SIGABRT for a debugger or core dump

Stack trace frames represent each function in the call chain:

func main() {
    Work()
}

func Work() {
    res := Calc(42)
    println(res)
}

func Calc(x int) int {
    if x == 42 {
        panic("can't handle 42")
    }
    return x * 2
}

panic: can't handle 42
/tmp/solod_build764532392/main.c:27 (func main_Calc)
/tmp/solod_run1106942066(main_Calc+0x51)
/tmp/solod_run1106942066(main_Work+0x12)
/tmp/solod_run1106942066(main+0x9)

The same system handles assertions like slice bounds, index-out-of-range, c.Assert, and similar checks. Instead of calling C's assert, they panic in a way that respects the -panic flag.

There's also a new -sanitize flag that enables C sanitizers (address and undefined by default) to help you catch more issues during development:

so run -sanitize -panic=abort example/play

Tooling

'so test' and 'so bench'. Solod now has built-in test and benchmark runners. so test finds TestXxx(t *testing.T) functions in a package's test subdirectory, creates a runner, transpiles it, and runs it. so bench does the same for BenchmarkXxx(b *testing.B).

A typical package layout with tests and benchmarks looks like this:

so/uuid
├── bench
│   ├── main.go
│   └── uuid.go
├── test
│   ├── main.go
│   └── uuid.go
└── uuid.go

There's also a quick check for memory leaks: t.Allocator() gives you a tracking allocator (described in the 'Safety net' section above), and the test will fail if anything allocated with it isn't freed by the end of the test.

=== RUN   TestAlloc
    memory leak: 1 unfreed allocation(s), 16 byte(s)
--- FAIL: TestAlloc

Fuzzing. Since Solod is a strict subset of Go, any So package is also a valid Go package. This means you get Go's built-in fuzzer for free, making fuzz testing pretty easy. So's encoding/json package takes advantage of this by using Go's own encoding/json as an oracle, making sure that every JSON document accepted by So is also accepted by Go.

Automatic linking. The new so:link directive lets a package specify which C library it needs, and so build gathers these libraries and passes them to the C compiler. The standard packages already use the new directive, so importing so/math links with -lm, and so/sync or so/conc links with -lpthread — you no longer have to set LDFLAGS manually.

Wrapping up

With v0.3, Solod reaches an important milestone: a program can now do multiple things at once. The JSON package gives programs a standard way to communicate, and the safety checks help prevent silent failures — both during development and in production.

There's still a lot to do, of course. In the next release, the standard library will keep growing, and the language and tooling will get better to make programming in So more convenient and safe.

If you're interested, take a look at So's readme — it has everything you need to get started. Or try So online without installing anything.

I know some of you have read them, and I've received a lot of kind words from you (even some core Go team members reached out) — thank you so much for that!

Tour history: Go 1.22 • 1.23 • 1.24 • 1.25 • 1.26 + Go features by version

Unfortunately, at some point, writing these tours stopped being fun and started to feel like a part-time job. I'm not really excited about that, so I've decided to stop.

I still like Go (well, most of it). I read a lot of Go code, I write some Go code, and I write Solod code, which is also Go 🙂 (Solod is a systems language with Go syntax and a Go-like stdlib).

I'm still pretty close to the language and will probably continue to write about it.

But the interactive tours story is over.

None of that machinery exists in C. Which made me wonder: how close can you get to Go's concurrency model using only POSIX threads? Obviously, native OS threads can't match the efficiency of lightweight goroutines, but what is the actual cost, when does it become a problem, and is there any way to at least partially avoid it?

I ran into these questions while adding concurrency to Solod (So), a strict subset of Go that translates to plain C, with no runtime and no garbage collector. In the end, I came to the conclusion that you can do quite a lot with pthreads — as long as you're honest about the tradeoffs.

This post is about the POSIX threads-based concurrency model I chose, the benefits it offers, and its limitations.

Mutex/Cond • Atomics • Pool • Channel • Performance • Design • Wrapping up

Mutex/Cond

Everything in So's concurrency stack is built on two basic POSIX primitives: the mutex and the condition variable. sync.Mutex is a thin wrapper around pthread_mutex_t:

// Extracted from So's stdlib source code.
type Mutex struct {
    mu pthread_mutex_t
}

func (m *Mutex) Lock() {
    rc := pthread_mutex_lock(&m.mu)
    if rc != 0 {
        panic("sync: Mutex.Lock failed")
    }
}

Since So translates to C, this is basically a struct that holds a pthread_mutex_t and a function that calls pthread_mutex_lock. Here's the transpiler output:

// The translated C code.
typedef struct sync_Mutex {
    pthread_mutex_t mu;
} sync_Mutex;

void sync_Mutex_Lock(sync_Mutex* m) {
    int rc = pthread_mutex_lock(&m->mu);
    if (rc != 0) {
        so_panic("sync: Mutex.Lock failed");
    }
}

That is the whole translation — the generated C is a near-mechanical mirror of the So code, only noisier. From here on, I'll mainly show the So version, but I'll also provide the C code for those who are interested.

There's nothing exciting here: sync.Mutex is a pthread mutex wrapper that panics if something goes wrong (which is rare).

The companion primitive is sync.Cond, a wrapper around pthread_cond_t. It's the standard "wait until a condition holds" tool, associated with a mutex:

type Cond struct           // wraps pthread_cond_t + pthread_mutex_t
func (c *Cond) Wait()      // wraps pthread_cond_wait
func (c *Cond) Signal()    // wraps pthread_cond_signal
func (c *Cond) Broadcast() // wraps pthread_cond_broadcast

typedef struct sync_Cond {
    pthread_cond_t cond;
    sync_Mutex*    mu;
} sync_Cond;

void sync_Cond_Wait(sync_Cond* c);      // wraps pthread_cond_wait
void sync_Cond_Signal(sync_Cond* c);    // wraps pthread_cond_signal
void sync_Cond_Broadcast(sync_Cond* c); // wraps pthread_cond_broadcast

These two types — Mutex and Cond — are the foundation. Other concurrency tools — Once, the thread pool, channels — are built using a mutex and one or more condition variables. This has several effects on performance, as we'll see later.

Atomics

Not everything needs a lock. So's sync/atomic mirrors Go's: Bool, Int32, Int64, Uint32, Uint64, and a generic Pointer[T], all with Load, Store, Swap, and CompareAndSwap methods.

The nice thing is that these don't need pthreads at all. They map directly to the C compiler's __atomic builtins — the same hardware instructions that Go's compiler emits. So there's no reason for them to be any slower, and they're not:

Each number is the cost of one operation on a single thread.

sync.Once is a good example of using atomics effectively. Its fast path only needs a single atomic load — after the given function runs, every future call to Do checks a flag and returns:

type Once struct {
    mu   Mutex
    done atomic.Bool
}

// Do calls f if and only if Do is being called
// for the first time for this o.
func (o *Once) Do(f func()) {
    if o.done.Load() { // lock-free fast path
        return
    }
    // slow path...
}

typedef struct sync_Once {
    sync_Mutex mu;
    atomic_Bool done;
} sync_Once;

// Do calls f if and only if Do is being called
// for the first time for this o.
void sync_Once_Do(sync_Once* o, void (*f)()) {
    if (atomic_Bool_Load(&o->done)) { // lock-free fast path
        return;
    }
    // slow path...
}

Worker pool

To actually run code concurrently, you need threads. The conc.Thread type wraps pthread_t and its related functions:

type Thread struct          // wraps pthread_t
func (th Thread) Wait() any // wraps pthread_join
func (th Thread) Detach()   // wraps pthread_detach

typedef struct conc_Thread {
    pthread_t t;
} conc_Thread;

void* conc_Thread_Wait(conc_Thread th);   // wraps pthread_join
void  conc_Thread_Detach(conc_Thread th); // wraps pthread_detach

Consider this conc.Go function:

// Go launches an OS thread that runs fn(arg) and returns a handle to it.
func Go(entry func(any) any, arg any) Thread {
    var th Thread
    rc := pthread_create(&th.t, nil, entry, arg)
    // ...
}

// Go launches an OS thread that runs fn(arg) and returns a handle to it.
// `any` in So translates to `void*` in C.
conc_Thread conc_Go(void* (*entry)(void*), void* arg) {
    conc_Thread th = {0};
    int rc = pthread_create(&th.t, NULL, entry, arg);
    // ...
}

Usage example:

func work(arg any) any {
    acc := arg.(*Account)
    // ...
}

func main() {
    var acc Account
    th := conc.Go(work, &acc)
    // ... do other work concurrently ...
    th.Wait() // work is complete once Wait returns
}

void* work(void* arg) {
    main_Account* acc = (main_Account*)arg;
    // ...
}

int main(void) {
    main_Account acc = {0};
    conc_Thread th = conc_Go(work, &acc);
    // ... do other work concurrently ...
    conc_Thread_Wait(th); // work is complete once Wait returns
}

It might look like go work(&acc), but that's just on the surface. conc.Go starts an actual OS thread, not a goroutine. You have to eventually call Wait to join or Detach it, or else its resources will leak. Also, OS threads are expensive to create — they're nothing like Go's goroutines, which only need a few kilobytes of stack and start up in nanoseconds.

That's exactly why you usually don't want to call Go inside a loop. For tasks that are short-lived or happen often, it's better to use a pool of long-lived worker threads and send tasks to them.

conc.Pool to the rescue:

     Worker thread pool in So
┌────────┐ ┌────────┐   ┌────────┐
│ Task 1 │ │ Task 2 │...│ Task M │  M tasks
└────────┘ └────────┘   └────────┘
┌────────────────────────────────┐
│           conc.Pool            │  coordinator
└────────────────────────────────┘
┌────────┐ ┌────────┐   ┌────────┐
│ Thrd 1 │ │ Thrd 2 │...│ Thrd N │  N threads, N << M
└────────┘ └────────┘   └────────┘
┌────────────────────────────────┐
│          OS scheduler          │
└────────────────────────────────┘

Usage example:

type Task struct {
    in  int
    out int
}

func square(arg any) {
    task := arg.(*Task)
    task.out = task.in * task.in
}

func main() {
    tasks := make([]Task, 10)

    opts := conc.PoolOpts{NumThreads: 2}
    pool := conc.NewPool(mem.System, opts)
    defer pool.Free()

    for i := range tasks {
        tasks[i].in = i
        pool.Go(square, &tasks[i])
    }
    pool.Wait()
}

typedef struct main_Task {
    so_int in;
    so_int out;
} main_Task;

void square(void* arg) {
    main_Task* task = (main_Task*)arg;
    task->out = task->in * task->in;
}

int main(void) {
    so_Slice tasks = so_make_slice(main_Task, 10, 10);

    conc_PoolOpts opts = (conc_PoolOpts){.NumThreads = 2};
    conc_Pool* pool = conc_NewPool(mem_System, opts);

    for (so_int i = 0; i < so_len(tasks); i++) {
        // so_at is a generic macro to get the i-th element of a
        // specific type (main_Task here) from a type-erased slice.
        // Here we're getting the i-th task from the tasks slice.
        so_at(main_Task, tasks, i).in = i;
        conc_Pool_Go(pool, square, &so_at(main_Task, tasks, i));
    }
    conc_Pool_Wait(pool);
    conc_Pool_Free(pool);
}

The first argument to NewPool, mem.System, is a memory allocator. Solod avoids hidden allocations, so anything that needs memory takes an allocator explicitly — here it backs the pool's task queue.

Under the hood, a Pool is a fixed group of worker threads that pull tasks from a shared queue (a ring buffer). It uses one mutex and a few condition variables:

// Pool is a bounded pool of worker threads with a wait queue
// which execute tasks of the form func(any).
type Pool struct {
    alloc mem.Allocator

    mu       sync.Mutex
    notEmpty sync.Cond // signaled when a task is enqueued
    notFull  sync.Cond // signaled when a slot frees
    allDone  sync.Cond // broadcast when no task is in flight

    workers []Thread
    queue   []task // ring buffer of submitted tasks
    active  int    // tasks submitted but not yet finished
    stopped bool   // set by Free to drain and exit
}

// NewPool creates a pool with a given number
// of worker threads and starts them.
func NewPool(alloc mem.Allocator, opts PoolOpts) *Pool

// Go submits a task for execution, blocking while the queue is full.
func (p *Pool) Go(fn func(any), arg any)

// Wait blocks until all submitted tasks finish.
func (p *Pool) Wait()

// Pool is a bounded pool of worker threads with a wait queue
// which execute tasks of the form func(any).
typedef struct conc_Pool {
    mem_Allocator alloc;

    sync_Mutex mu;
    sync_Cond  notEmpty; // signaled when a task is enqueued
    sync_Cond  notFull;  // signaled when a slot frees
    sync_Cond  allDone;  // broadcast when no task is in flight

    so_Slice workers;
    so_Slice queue;      // ring buffer of submitted tasks
    so_int   active;     // tasks submitted but not yet finished
    bool     stopped;    // set by Free to drain and exit
} conc_Pool;

conc_Pool* conc_NewPool(mem_Allocator alloc, conc_PoolOpts opts);
void       conc_Pool_Go(conc_Pool* p, void (*fn)(void*), void* arg);
void       conc_Pool_Wait(conc_Pool* p);

notEmpty wakes up a worker when there are tasks to do, notFull applies back-pressure when the queue is full, and allDone lets Wait know when everything is finished. It's a classic producer-consumer setup, about 200 lines of code, and there's nothing fancy about it.

The heart of the pool is the worker loop. Each thread blocks until a task appears, runs it outside the lock so workers execute in parallel, then records that it finished:

// workerMain runs on every pool thread: pull a task, run it, repeat.
func workerMain(arg any) any {
    p := arg.(*Pool)
    for {
        p.mu.Lock()
        for p.qempty() && !p.stopped {
            p.notEmpty.Wait() // sleep until a task is enqueued
        }
        if p.qempty() && p.stopped {
            p.mu.Unlock()
            break // queue drained and pool shutting down
        }
        t := p.qpop()
        p.notFull.Signal() // a slot freed for a waiting submitter
        p.mu.Unlock()

        t.fn(t.arg) // run the task with the lock released

        p.mu.Lock()
        p.active--
        if p.active == 0 {
            p.allDone.Broadcast() // wake anyone parked in Wait
        }
        p.mu.Unlock()
    }
    return nil
}

// workerMain runs on every pool thread: pull a task, run it, repeat.
static void* workerMain(void* arg) {
    conc_Pool* p = (conc_Pool*)arg;
    for (;;) {
        sync_Mutex_Lock(&p->mu);
        for (; conc_Pool_qempty(p) && !p->stopped;) {
            sync_Cond_Wait(&p->notEmpty); // sleep until a task is enqueued
        }
        if (conc_Pool_qempty(p) && p->stopped) {
            sync_Mutex_Unlock(&p->mu);
            break; // queue drained and pool shutting down
        }
        task t = conc_Pool_qpop(p);
        sync_Cond_Signal(&p->notFull); // a slot freed for a waiting submitter
        sync_Mutex_Unlock(&p->mu);

        t.fn(t.arg); // run the task with the lock released

        sync_Mutex_Lock(&p->mu);
        p->active--;
        if (p->active == 0) {
            sync_Cond_Broadcast(&p->allDone); // wake anyone parked in Wait
        }
        sync_Mutex_Unlock(&p->mu);
    }
    return NULL;
}

This is what separates a pool from a plain queue. Pool.Go bumps active as it enqueues; each worker decrements it after running a task, and the last one out broadcasts allDone.

Pool.Wait sleeps until the count hits zero:

// Wait blocks until every submitted task has finished.
func (p *Pool) Wait() {
    p.mu.Lock()
    for p.active != 0 {
        p.allDone.Wait()
    }
    p.mu.Unlock()
}

// Wait blocks until every submitted task has finished.
void conc_Pool_Wait(conc_Pool* p) {
    sync_Mutex_Lock(&p->mu);
    for (; p->active != 0;) {
        sync_Cond_Wait(&p->allDone);
    }
    sync_Mutex_Unlock(&p->mu);
}

The tradeoff is that the number of worker threads is fixed. In Go, a program can handle thousands of concurrent I/O waits because blocked goroutines use very little memory. A So pool can't do this — if all N workers are parked on a blocking syscall, the pool is stalled until one returns. You have to set the pool size based on the workload, instead of letting the runtime manage it for you.

Channel

Channels are an important part of Go's concurrency model, and So's conc.Chan[T] gives you something quite similar. Just like in Go, it passes values by copy and comes in buffered and unbuffered flavors:

ch := conc.NewChan[int](mem.System, 2) // buffered, capacity 2
defer ch.Free()

// Producer on its own thread.
prod := producer{ch: &ch, n: 5}
thr := conc.Go(produce, &prod)
defer thr.Wait()

// Consume until the channel is closed and drained.
var v int
for ch.Recv(&v) {
    fmt.Printf("received %d\n", v)
}

// conc_NewChan, conc_Chan_Recv, and friends are generic macros:
// the element type (so_int here) is passed as the first argument.
conc_Chan ch = conc_NewChan(so_int, mem_System, 2); // buffered, capacity 2

// Producer on its own thread.
producer prod = (producer){.ch = &ch, .n = 5};
conc_Thread thr = conc_Go(produce, &prod);

// Consume until the channel is closed and drained.
so_int v = 0;
for (; conc_Chan_Recv(so_int, &ch, &v);) {
    fmt_Printf("received %d\n", v);
}

conc_Thread_Wait(thr);
conc_Chan_Free(so_int, &ch);

Chan[T] is a thin generic shell over one of two engines, picked at creation time:

Buffered (n > 0) is a mutex-guarded ring buffer with notEmpty and notFull condition variables — like the Pool queue. Senders block when it's full, receivers block when it's empty.

type Buffer struct {
    alloc mem.Allocator

    mu       sync.Mutex
    notEmpty sync.Cond // signaled when an item becomes available
    notFull  sync.Cond // signaled when a slot frees

    buf    mem.Array   // ring buffer
    closed bool        // true after Close
}

// Send copies v into the ring, blocking while it is full.
func (ch *Buffer) Send(v any) {
    ch.mu.Lock()
    for ch.bfull() {
        ch.notFull.Wait() // back-pressure until a slot frees
    }
    ch.bpush(v)
    ch.notEmpty.Signal() // wake one waiting receiver
    ch.mu.Unlock()
}

typedef struct conc_Buffer {
    mem_Allocator alloc;

    sync_Mutex mu;
    sync_Cond  notEmpty; // signaled when an item becomes available
    sync_Cond  notFull;  // signaled when a slot frees

    mem_Array buf;       // ring buffer
    bool closed;         // true after Close
} conc_Buffer;

// Send copies v into the ring, blocking while it is full.
void conc_Buffer_Send(conc_Buffer* ch, void* v) {
    sync_Mutex_Lock(&ch->mu);
    for (; conc_Buffer_bfull(ch);) {
        sync_Cond_Wait(&ch->notFull); // back-pressure until a slot frees
    }
    conc_Buffer_bpush(ch, v);
    sync_Cond_Signal(&ch->notEmpty); // wake one waiting receiver
    sync_Mutex_Unlock(&ch->mu);
}

The full implementation also checks for closed, but I left it out for brevity.

Recv is the mirror method: block while empty, pop the next value, signal notFull to wake a sender. It also handles the closed channel, returning false once the buffer is closed and drained. The rest is this lock-wait-signal core.

Buffer source code

Unbuffered (n == 0) is a rendezvous: each send blocks until a receiver takes the value, copying vsize bytes directly from the sender's stack to the receiver's destination without using an intermediate buffer.

type Rendezvous struct {
    alloc mem.Allocator
    vsize int // size in bytes of a handed-off value

    mu   sync.Mutex
    cond sync.Cond // broadcast on every slot state change

    src     any  // the sender's published value (valid while full)
    full    bool // a value is published and not yet freed
    claimed bool // the published value has been taken by a receiver
    closed  bool // true after Close
}

// Send publishes v and waits for a receiver to take it.
func (ch *Rendezvous) Send(v any) {
    ch.mu.Lock()
    for ch.full {
        ch.cond.Wait()  // wait for the previous hand-off to finish
    }
    ch.src, ch.full, ch.claimed = v, true, false // publish
    ch.cond.Broadcast() // wakeup #1: wake a receiver
    for !ch.claimed {
        ch.cond.Wait()  // wait until the value is taken
    }
    ch.src, ch.full = nil, false // free the slot
    ch.cond.Broadcast()
    ch.mu.Unlock()
}

typedef struct conc_Rendezvous {
    mem_Allocator alloc;
    so_int vsize; // size in bytes of a handed-off value

    sync_Mutex mu;
    sync_Cond  cond; // broadcast on every slot state change

    void* src;     // the sender's published value (valid while full)
    bool  full;    // a value is published and not yet freed
    bool  claimed; // the published value has been taken by a receiver
    bool  closed;  // true after Close
} conc_Rendezvous;

// Send publishes v and waits for a receiver to take it.
void conc_Rendezvous_Send(conc_Rendezvous* ch, void* v) {
    sync_Mutex_Lock(&ch->mu);
    for (; ch->full;) {
        sync_Cond_Wait(&ch->cond);  // wait for the previous hand-off to finish
    }
    ch->src = v;                    // publish
    ch->full = true;
    ch->claimed = false;
    sync_Cond_Broadcast(&ch->cond); // wakeup #1: wake a receiver
    for (; !ch->claimed;) {
        sync_Cond_Wait(&ch->cond);  // wait until the value is taken
    }
    ch->full = false;               // free the slot
    ch->src = NULL;
    sync_Cond_Broadcast(&ch->cond);
    sync_Mutex_Unlock(&ch->mu);
}

Recv is the other half: it waits for a published, unclaimed value, copies vsize bytes straight from the sender's stack into dst (no intermediate buffer), marks it as claimed, and broadcasts to wake the sender back, creating wakeup #2. One hand-off, two wakeups.

Copying directly from the sender's stack is safe because of that second wakeup. src is a pointer to v, which lives on the sender's stack. While the receiver is reading it, the sender is parked in for !ch.claimed { ch.cond.Wait() }, so its stack frame stays alive. The sender only returns (and reclaims that memory) after the receiver sets claimed and wakes it up. There's no need to copy into a shared buffer because the source is guaranteed to outlive the read.

Rendezvous source code

As you can see, the API is pretty similar to Go. Now let's look at the numbers.

Performance

Here's the main tradeoff: pthread-based concurrency primitives are fast when no one has to block, but they get slow when someone does. And it's always for the same reason.

Go schedules goroutines in userspace. When one goroutine blocks on a channel and another wakes it up, the runtime moves them between its own queues — no kernel involved. POSIX threads, on the other hand, don't provide a userland scheduler. When a thread blocks on a condition variable, it parks in the kernel, and waking it up requires a syscall. Every hand-off between threads that actually parks pays the cost of a syscall on both ends.

You can clearly see the difference in the mutex benchmarks. With 8 competing threads, it all comes down to whether the waiting threads have to park or not:

Each number is the average time for a single Lock/Unlock pair. The uncontended benchmark runs on one thread, while the contended benchmarks have multiple threads fighting over the same mutex.

Notice that So actually wins the first two benchmarks, and for good reason. So's Lock is a plain pthread_mutex_lock call with nothing extra, while Go's sync.Mutex adds more overhead — like starvation-mode tracking and a runtime that stays involved because a goroutine can be preempted in the middle of a critical section.

When nobody parks, that overhead is the main cost, and the thinner wrapper is closer to the hardware. With an empty critical section (the spin benchmark), a waiting thread grabs the lock while still spinning and almost never parks — So wins by 2.8x. The uncontended benchmark (a single thread, no contention) shows the same thing: less code between the call and the lock, so 9ns versus 14ns.

The picture flips the moment threads have to park. Give the critical section about a microsecond of real work (the work benchmark) and waiters exhaust their spin budget and park. Now every hand-off costs a wakeup syscall, and So drops to half of Go's throughput. The work is identical in both cases — the difference comes from the parking cost.

Condition variables demonstrate this clearly because they always park:

Each number is the cost of one rendezvous round: a single broadcast that wakes every waiter and hands control back, with N waiters plus one broadcaster.

Pthread-based condition variable is consistently 7-10 times slower. There's no trick to close this gap — it's just the cost of waking up a real OS thread instead of a goroutine.

Channels have the same issue because they're built using mutexes and condition variables:

Each number is the cost of moving one value through the channel (send plus its matching receive). The number in parentheses is the buffer capacity.

The uncontended case fills and drains a buffer from a single thread, so nothing ever blocks — it's just a lock plus a copy, which gives So a slight advantage. But the moment a producer and consumer actually start handing off work, So has to wake up a thread for every transfer that gets parked. It's worst for the unbuffered channel, where every value is a rendezvous with two wakeups: 23x slower. A larger buffer helps a lot — with room for 100 items, most sends go through without waking anyone, and the gap narrows to about 2x.

The consequence is that the larger your tasks are, the better pthread-based concurrency works. If you use a channel for fine-grained, value-at-a-time streaming between threads, performance will suffer. But if you use a channel to pass whole work items to a pool, where each item takes tens of microseconds to process, the wakeup cost becomes negligible. The pool benchmarks on realistic workloads confirms this:

Each number is the wall-clock time for 8 workers to process the whole batch.

Here, So is within 1.1x of Go. The per-task dispatch cost is still present, but it's spread out over real work, and the performance penalty is pretty small.

-Ofast -march=native -flto -funroll-loops -DNDEBUG

Here's a summary of the strengths and weaknesses of the pthread-based approach:

• ➕ Coarse-grained pooled workloads are within about 10% of Go's performance.
• ➕ Uncontended locks and spin-friendly critical sections perform quite well.
• ➕ Atomic operations are as fast as in Go.
• ➕ The implementation is 100x simpler.
• ➖ Anything that needs to park and wake an OS thread is much slower than Go's userspace scheduler.
• ➖ The pool can't handle thousands of blocked waiters like goroutines can.

If you're looking for "thousands of cheap goroutines", the pthread-based approach will let you down. But if you're fine with "a few worker threads handling lots of tasks", it holds up well.

Design decisions

Three decisions influenced the way I implemented concurrency in Solod.

Pthreads, not fibers. I know there are coroutine/fiber libraries for C that avoid the kernel wakeup cost — single-threaded ones like neco, and multi-threaded ones like libfiber. A userspace scheduler is exactly what would help to match Go in the benchmarks above.

I decided not to use one. I wanted something dead simple — an approach I could explain in a paragraph, using tools every C programmer already knows. The trade-off is that you lose some performance with fine-grained blocking, but in many real-world situations, pthreads work fine if you use a worker pool. For me, keeping things simple is more important than saving a few microseconds during task hand-offs. For now, at least.

Standard library, not language. Go bakes goroutines, channels, and select right into the language. I decided to keep everything in the stdlib for two reasons.

➀ It follows So's "no hidden allocations" rule. In Go, go f() quietly allocates a goroutine stack, and make(chan T, n) allocates a buffer. In So, all allocations are explicit: you pass an allocator to NewChan and NewPool, and you always know exactly where the memory comes from — whether it's the system allocator, an arena, or something else.

➁ A library is more flexible. Since a pool is a regular value, you can have as many as you need, each sized for its specific purpose. In a multi-stage pipeline where each stage needs a different capacity, you can start one pool per stage, each with its own NumThreads and QueueSize, instead of being given a single global scheduler. The language stays simple, and the flexibility is in code you can easily read.

Timeouts, not select. Go's select waits on several channel operations at once and proceeds with whichever is ready first. Implementing it would require a lot of work — a thread has to register interest on multiple channels, block once, and then wake up when any of them is ready — so I left it out. Instead, Chan offers SendTimeout and RecvTimeout, which cover two common uses of select with a single channel:

• "Do this, but give up after a while" (Go's case <-time.After(...) idiom).
• "Do this only if it won't block" (Go's non-blocking default branch).

What's missing is the ability to block on multiple channels at once and continue with whichever one is ready first, as well as the option to mix sends and receives in the same selection.

Wrapping up

How close can you get to Go's concurrency using only pthreads? Close enough to be useful, but not enough to really match Go. You can wrap real OS threads with familiar APIs — mutexes, condition variables, pools, channels — and the code will look and act a lot like Go, at least until a thread needs to block. But there's no scheduler underneath, so when a thread blocks, it's an actual thread waiting in the kernel, not a goroutine that's paused for free. That's the main limitation of this approach.

What you get in return is brutal simplicity. Every primitive is a thin wrapper with no runtime hiding behind it, so the performance is exactly what the OS gives you: fast atomics, fast uncontended locks, and pooled throughput within ~10% of Go on coarse-grained work. But as soon as you switch to fine-grained, one-value-at-a-time hand-offs, the cost of kernel wakeups becomes the main factor, and you'll notice the slowdown.

If you think the pthread approach might work for you, I invite you to try Solod. It includes the sync and conc packages, along with many others ported from Go's standard library.

• Go developers who want low-level control without having to learn another language.
• C developers who like Go's style.

The previous version (v0.1) focused on porting core Go stdlib packages and providing convenient C interop. At the end of that post, I said the next release would focus on networking, concurrency, or both. Now, networking is here — the v0.2 release I'm sharing today includes support for TCP, UDP, and Unix domain sockets. Concurrency is still planned for the future, so for now, servers handle one connection at a time.

This release also lets you compile So to more targets, like 32-bit platforms, WebAssembly, and bare metal. And C interop even smoother!

Networking • TCP server • TCP client • Deadlines • IP addresses • Targets • Interop • Stdlib • Wrapping up

Networking

The main feature in v0.2 is the net package. It's a simplified version of Go's net package which supports the three most commonly used transports:

• TCP (networks tcp, tcp4, tcp6) via ResolveTCPAddr, DialTCP, and ListenTCP, with the TCPConn and TCPListener types.
• UDP (networks udp, udp4, udp6) via ResolveUDPAddr, DialUDP (a connected socket), and ListenUDP (an unconnected socket with ReadFrom/WriteTo).
• Unix domain sockets (unix for streams, unixgram for datagrams) via ResolveUnixAddr, DialUnix, ListenUnix, and ListenUnixgram.

The API mirrors Go closely, so most of it will feel familiar. The big difference is that So has no goroutines, so there's no concurrent server support — you accept and serve connections sequentially. More on that in a moment.

TCP server

Let's build a classic: an echo server that accepts a connection, reads a message, and sends it back.

package main

import "solod.dev/so/net"

func main() {
    // Resolve the local address to listen on.
    laddr, err := net.ResolveTCPAddr("tcp", "127.0.0.1:8080")
    if err != nil {
        panic(err)
    }

    // Start listening on the local address.
    ln, err := net.ListenTCP("tcp", &laddr)
    if err != nil {
        panic(err)
    }
    defer ln.Close()
    println("listening on", "127.0.0.1:8080")

    // Accept connections and serve them in a loop.
    for {
        conn, err := ln.Accept()
        if err != nil {
            panic(err)
        }
        serve(&conn)
    }
}

// serve reads one message from the connection, echoes it back,
// and closes the connection.
func serve(conn *net.TCPConn) {
    defer conn.Close()

    var buf [256]byte
    n, err := conn.Read(buf[:])
    if err != nil {
        return
    }
    conn.Write(buf[:n])
}

listening on 127.0.0.1:8080

If you've written a TCP server in Go, this should look familiar — ListenTCP, an Accept loop, and Read/Write on the connection. The only thing missing is a go serve(conn): without goroutines, each connection is handled to completion before moving on to the next Accept.

TCP client

The client starts the connection using DialTCP, then uses Write to send a request and Read to get the reply:

package main

import "solod.dev/so/net"

func main() {
    // Resolve the server address.
    raddr, err := net.ResolveTCPAddr("tcp", "127.0.0.1:8080")
    if err != nil {
        panic(err)
    }

    // A nil laddr lets the system choose the local address.
    conn, err := net.DialTCP("tcp", nil, &raddr)
    if err != nil {
        panic(err)
    }
    defer conn.Close()

    // Send a request and read the reply.
    conn.Write([]byte("hello"))

    var buf [256]byte
    n, err := conn.Read(buf[:])
    if err != nil {
        panic(err)
    }
    println(string(buf[:n]))
}

hello

UDP and Unix domain sockets work in a similar way. For UDP, an unconnected ListenUDP socket uses ReadFrom to get data and the sender's address, and WriteTo to send a reply. For Unix sockets, there are ListenUnix (stream) and ListenUnixgram (datagram).

Deadlines

By default, Accept, Read, and Write are blocking. In Go, you'd typically use goroutines and contexts to prevent getting stuck forever. Since that's not available in So (yet), every connection and listener supports deadlines instead:

// Give the client 5 seconds to send something.
conn.SetReadDeadline(time.Now().Add(5 * time.Second))

n, err := conn.Read(buf[:])
if err == net.ErrTimeout {
    // The client went quiet; drop the connection.
    return
}

SetDeadline, SetReadDeadline, and SetWriteDeadline are available on TCPConn, UDPConn, UnixConn, and listener types. When the deadline passes, any pending call fails with net.ErrTimeout. If you don't set a deadline, a blocked call will wait forever. This isn't concurrency, but it's enough to keep a single-threaded server responsive.

IP addresses

Along with net, v0.2 ports Go's net/netip package, which provides small, allocation-free value types for IP addresses. Addr represents an IP address, AddrPort combines an IP address with a port, and Prefix is an IP with a prefix length (a CIDR block):

addr, err := netip.ParseAddr("192.168.1.10")
if err != nil {
    panic(err)
}
println(addr.Is4())            // true

ap := netip.AddrPortFrom(addr, 8080)
println(ap.Port())             // 8080

prefix := netip.MustParsePrefix("192.168.1.0/24")
println(prefix.Contains(addr)) // true

These are simple value types that don't use any heap allocation, which fits well with So's explicit-memory approach. The net package also provides SplitHostPort and JoinHostPort functions to help you work with host:port strings.

New targets

Solod compiles to plain C, which (in theory) means it can target anything a C compiler can. Because of this, v0.2 adds new targets:

• 32-bit platforms. The compiler and stdlib now work correctly on 32-bit platforms, where int and pointers are narrower.
• WebAssembly (WASI). You can compile a So program to wasm32-wasi and run it under any WASI runtime.
• Freestanding mode. So programs can run on bare-metal systems without any C standard library. No libc means no malloc, but you can use mem.Arena instead.

Here's the complete toolchain you need to build a freestanding wasm32 binary using zig cc:

export CC="zig cc"
export CFLAGS="-Oz --target=wasm32-freestanding -nostdlib -Wl,--no-entry -Wl,--export=main"
so build -o main.wasm .

A large part of the standard library (bytes, strings, strconv, slices, maps, math, encoding/binary, and more) works just fine in freestanding mode. For more details, check out the freestanding guide.

Friendlier interop

A bunch of smaller changes make Solod nicer to write.

Three new directives for low-level work, all documented in the interop guide:

//so:volatile
var counter int       // emits a C volatile

//so:thread_local
var perThread int     // emits C11 _Thread_local

//so:attr packed
type header struct {  // emits __attribute__((packed))
    version byte
    length  int
}

so:attr works with variables, constants, types, and functions. You can use it on multiple lines, and the attributes will stack. For example, //so:attr aligned(16) will combine with //so:attr packed.

Type aliases. So now supports Go-style type aliases:

type Byte = uint8

Numeric C types. The so/c package now includes named types for C's numeric types — Int, UInt, Long, Short, UChar, LongLong, and others. When you declare an extern function, you can use the actual C types in its signature instead of trying to guess the correct fixed-width Go type for your platform.

Third-party packages. You can now add external So packages using go get or by vendoring, and you can organize your own code into multiple modules. So doesn't have a real package ecosystem yet, but it's a good start.

Better diagnostics. By default, panic messages report the C file and line. Pass --track-source to report the original So source location instead:

so run --track-source .

There's also an optional --check-nil flag that adds nil-pointer checks when accessing struct fields and calling interface methods. This way, if there's a bad dereference, the program will panic cleanly instead of causing a segmentation fault. Both options are off by default to keep the generated code more readable.

More stdlib

Beyond net and net/netip, v0.2 adds a few more packages:

• encoding/hex — hex encoding and decoding, including Dump for hexdump-style output.
• uuid — generating and parsing UUIDs (v4 and v7), with random components from a cryptographically secure source.

And a small but handy update to memory management: mem.Arena.Free now reclaims the last allocation if you give it the matching pointer. It's a minor optimization, but it means a quick alloc/free pair on an arena no longer wastes space.

Stdlib documentation

Wrapping up

With v0.2, Solod has evolved from just "command-line tools and C glue" into something you can actually use on a network — like a TCP or UDP server, a small protocol client, or a Unix-socket daemon. The new targets (32-bit, WASM, freestanding) mean the same code can now run in more places, even down to bare metal.

The big thing that's still missing is concurrency. A server that handles requests one at a time works for some tasks, but a real network service needs to manage many connections at once. That's the obvious goal for the next release — adding some kind of concurrency, along with the stdlib packages that support it.

If you're interested, take a look at So's readme — it has everything you need to get started. Or try So online without installing anything.

• Go developers who want low-level control without having to learn another language.
• C developers who like Go's style.

The initial version (let's call it v0) was focused on picking a subset of Go and translating it to C. The next logical step was to port Go's standard library and make it easier to interop with C. That's what the v0.1 release I'm presenting today is all about.

Standard library • SQLite bindings • Persistent map • Store and retrieve • Command-line interface • Performance • Wrapping up

Standard library

Solod 0.1 ships with the following stdlib packages ported from Go:

• io, bufio, and fmt — Abstractions and types for general-purpose I/O.
• bytes, strings, strconv, and unicode/utf8 — Common byte and text operations.
• slices and maps — Generic heap-allocated data structures.
• crypto/rand and math/rand — Generating random data.
• flag, os, and path — Working with the command line and files.
• log/slog — Structured logging.
• time — Measuring and displaying time.

And a couple of its own packages:

• mem — Memory allocation with a pluggable allocator interface.
• c — Low-level C interop helpers.

Stdlib documentation

In the following sections, I'll demonstrate some of the features using a simple example: a persistent key-value store backed by SQLite.

SQLite bindings

Since So doesn't provide database/sql yet, we'll call SQLite directly through its C API. To do this, let's import the necessary headers with the so:include directive and generate extern declarations using the sobind tool:

package main

import "solod.dev/so/c"

//so:include <sqlite3.h>

// SQLite constants.
//
//so:extern SQLITE_OK
const sqliteOK = 0
//so:extern SQLITE_ROW
const sqliteRow = 100
//so:extern SQLITE_DONE
const sqliteDone = 101

// SQLite types.
//
//so:extern
type sqlite3 struct{}
//so:extern
type sqlite3_stmt struct{}
//so:extern
type sqlite3_value struct{}
//so:extern
type sqlite3_callback func(any, int32, **c.Char, **c.Char) int32

// SQLite functions.
func sqlite3_open(filename string, ppDb **sqlite3) int32
func sqlite3_prepare_v2(db *sqlite3, zSql string, nByte int32, ppStmt **sqlite3_stmt, pzTail **c.ConstChar) int32
func sqlite3_step(arg0 *sqlite3_stmt) int32
func sqlite3_finalize(pStmt *sqlite3_stmt) int32
func sqlite3_close(arg0 *sqlite3) int32
func sqlite3_exec(arg0 *sqlite3, sql string, callback sqlite3_callback, arg3 any, errmsg **c.Char) int32

// more declarations...

The so:extern directive is required for constants (sqliteOK) and types (sqlite3_stmt). As for functions (sqlite3_prepare_v2), we can just declare them without a body — the transpiler will treat them as extern declarations even without so:extern.

Persistent map

With the SQLite API in place, let's implement a key-value type that wraps the database connection:

// SQLMap is a simple key-value store backed by an SQLite database.
type SQLMap struct {
    db *sqlite3
}

Add a constructor that connects to an SQLite database and creates a table to store the items:

var ErrCreate = errors.New("sqlmap: create schema failed")
const sqlCreate = "create table if not exists kv (key text primary key, val)"

// NewSQLMap creates a new SQLMap using the provided connection string.
// It opens a connection to the SQLite database and creates the underlying
// key-value table if it does not already exist.
//
// The caller is responsible for calling Close on the returned SQLMap
// when it is no longer needed.
func NewSQLMap(connStr string) (SQLMap, error) {
    var db *sqlite3
    rc := sqlite3_open(connStr, &db)
    if rc != sqliteOK {
        return SQLMap{}, ErrCreate
    }

    rc = sqlite3_exec(db, sqlCreate, nil, nil, nil)
    if rc != sqliteOK {
        sqlite3_close(db)
        return SQLMap{}, ErrCreate
    }
    return SQLMap{db}, nil
}

// Close releases resources associated with the SQLMap.
func (m *SQLMap) Close() {
    sqlite3_close(m.db)
}

As you can see, this So code looks a lot like regular Go code. However, there are some key differences:

• When compiled, the code is first translated to plain C, then compiled into a native binary using GCC or Clang.
• Unlike Go, there is no runtime (no automatic heap memory allocation, no garbage collection, no goroutine scheduler).
• There is no overhead when calling C functions, unlike Go's Cgo.
• The interop syntax is a bit cleaner. For example, Go's string (sqlCreate in the sqlite3_exec call) automatically decays to C's const char*.

Store and retrieve

First, let's implement the Set method:

var (
    ErrPrepare = errors.New("sqlmap: prepare failed")
    ErrExec    = errors.New("sqlmap: exec failed")
)

const sqlSet = "insert or replace into kv (key, val) values (?, ?)"

// Set stores a string value for the specified key.
func (m *SQLMap) Set(key string, val string) error {
    var stmt *sqlite3_stmt
    rc := sqlite3_prepare_v2(m.db, sqlSet, -1, &stmt, nil)
    if rc != sqliteOK {
        return ErrPrepare
    }
    defer sqlite3_finalize(stmt)

    sqlite3_bind_text(stmt, 1, key, int32(len(key)), nil)
    sqlite3_bind_text(stmt, 2, val, int32(len(val)), nil)

    rc = sqlite3_step(stmt)
    if rc != sqliteDone {
        return ErrExec
    }
    return nil
}

No surprises here, just a bunch of SQLite API calls.

The Get method is more interesting:

var ErrNotFound = errors.New("sqlmap: not found")
const sqlGet = "select val from kv where key = ?"

// Get returns the value associated with the specified key.
// The caller owns the returned string and must free it with mem.FreeString.
func (m *SQLMap) Get(a mem.Allocator, key string) (string, error) {
    var stmt *sqlite3_stmt
    rc := sqlite3_prepare_v2(m.db, sqlGet, -1, &stmt, nil)
    if rc != sqliteOK {
        return "", ErrPrepare
    }
    defer sqlite3_finalize(stmt)

    sqlite3_bind_text(stmt, 1, key, int32(len(key)), nil)
    rc = sqlite3_step(stmt)
    if rc == sqliteDone {
        return "", ErrNotFound
    }
    if rc != sqliteRow {
        return "", ErrExec
    }

    text := sqlite3_column_text(stmt, 0)
    tmp := c.String(text)
    result := strings.Clone(a, tmp)
    return result, nil
}

The pointer returned by sqlite3_column_text is managed by SQLite. It becomes invalid after calling sqlite3_finalize (which Get does before returning). Because of this, we need to allocate a copy of the returned value, using strings.Clone in this case.

So's approach to memory allocation is similar to Zig's — all heap allocations must be done explicitly by providing a specific instance of the mem.Allocator interface.

The caller, of course, must free the allocated string:

func main() {
    m, err := NewSQLMap(":memory:")
    if err != nil {
        panic(err)
    }
    defer m.Close()

    m.Set("name", "Alice")
    name, err := m.Get(mem.System, "name")
    if err != nil {
        panic(err)
    }
    println("name =", name)
    mem.FreeString(mem.System, name)
}

name = Alice

Here, mem.System is a specific allocator that uses libc's malloc and free. Alternatively, we could use mem.Arena or any other implementation of the mem.Allocator interface:

var buf [1024]byte // stack-allocated
arena := mem.NewArena(buf[:])

name, _ := m.Get(&arena, "name")
mem.FreeString(&arena, name) // no-op for arena; can be omitted

Command-line interface

With the SQLMap type in place, let's create a simple CLI using the flag package:

var (
    opFlag  string
    keyFlag string
    valFlag string
)

func parseFlags() {
    flag.StringVar(&opFlag, "op", "", "operation: get, set, or del")
    flag.StringVar(&keyFlag, "key", "", "key name")
    flag.StringVar(&valFlag, "val", "", "value (for set operation)")
    flag.Parse()
}

func main() {
    parseFlags()
    // ...
}

Then add command routing:

m, err := NewSQLMap("sqlmap.db")
check(err)
defer m.Close()

switch opFlag {
case "set":
    err = m.Set(keyFlag, valFlag)
    check(err)
case "get":
    val, err := m.Get(mem.System, keyFlag)
    check(err)
    println(val)
    mem.FreeString(mem.System, val)
case "del":
    err = m.Delete(keyFlag)
    check(err)
default:
    flag.Usage()
    os.Exit(1)
}

sqlmap -op=set -key=name -val=alice
sqlmap -op=get -key=name
alice

Again, no surprises here — the flag package works just as it does in Go.

Performance

Solod isn't trying to outperform hand-tuned C. Still, performance matters: the code is benchmarked and optimized to run reasonably fast. Since So compiles to plain C and then to native code with full optimizations, the results are sometimes better than Go's.

Here are some highlights from the benchmarks:

• Buffered I/O is 3x faster than Go.
• String and byte operations are up to 2.5x faster.
• Maps are 1.5x faster for modifications.
• Integer formatting is 2x faster.

There're no GC pauses and no Cgo bridge cost when calling C libraries. The tradeoff is that you have to handle memory yourself, but as the SQLite example above shows, So's allocator interface makes that pretty manageable.

Solod vs. Go benchmarks

Wrapping up

Solod is still in its early days, but with the v0.1 release, it's ready for hobby projects. The already-ported parts of the Go standard library make it easy to write command-line tools (check out the cat, head, sort, and wc examples). Plus, with native C interop, you can build just about anything else you need.

The next release will likely focus on networking, concurrency, or both — along with more stdlib packages.

If you're interested, take a look at So's readme — it has all the information you need to get started. Or try So online without installing anything.

Of course, this isn't something I could do all at once. I started with the io package, which provides core abstractions like Reader and Writer, as well as general-purpose functions like Copy. But io isn't very interesting on its own, since it doesn't include specific reader or writer implementations. So my next choices were naturally bytes and strings — the workhorses of almost every Go program. This post is about how the porting process went.

Bits and UTF-8 • Bytes • Allocators • Buffers and builders • Benchmarks • Optimizing search • Optimizing builder • Wrapping up

Bits and UTF-8

Before I could start porting bytes, I had to deal with its dependencies first:

• math/bits implements bit counting and manipulation functions.
• unicode/utf8 implements functions for UTF-8 encoded text.

Both of these packages are made up of pure functions, so they were pretty easy to port. The only minor challenge was the difference in operator precedence between Go and C — specifically, bit shifts (<<, >>). In Go, bit shifts have higher precedence than addition and subtraction. In C, they have lower precedence:

// Go: shift has HIGHER precedence than +
var x uint32 = 1<<2 + 3  // (1 << 2) + 3 == 7

// C: shift has LOWER precedence than +
uint32_t x = 1 << 2 + 3; // 1 << (2 + 3) == 32

The simplest solution was to just use parentheses everywhere shifts are involved:

// Go: Mul64 returns the 128-bit product of x and y: (hi, lo) = x * y
func Mul64(x, y uint64) (hi, lo uint64) {
    const mask32 = 1<<32 - 1
    x0 := x & mask32
    x1 := x >> 32
    y0 := y & mask32
    y1 := y >> 32
    w0 := x0 * y0
    t := x1*y0 + w0>>32
    // ...
}

// C: Mul64 returns the 128-bit product of x and y: (hi, lo) = x * y
so_Result bits_Mul64(uint64_t x, uint64_t y) {
    const so_int mask32 = ((so_int)1 << 32) - 1;
    uint64_t x0 = (x & mask32);
    uint64_t x1 = (x >> 32);
    uint64_t y0 = (y & mask32);
    uint64_t y1 = (y >> 32);
    uint64_t w0 = x0 * y0;
    uint64_t t = x1 * y0 + (w0 >> 32);
    // ...
}

With bits and utf8 done, I moved on to bytes.

Bytes

The bytes package provides functions for working with byte slices:

// Count counts the number of non-overlapping instances of sep in s.
func Count(s, sep []byte) int

// Equal reports whether a and b are the
// same length and contain the same bytes.
func Equal(a, b []byte) bool

// Index returns the index of the first instance
// of sep in s, or -1 if sep is not present in s.
func Index(s, sep []byte) int

// Repeat returns a new byte slice consisting of count copies of b.
func Repeat(b []byte, count int) []byte

// and others

Some of them were easy to port, like Equal. Here's how it looks in Go:

// Equal reports whether a and b are the
// same length and contain the same bytes.
func Equal(a, b []byte) bool {
    // Neither cmd/compile nor gccgo allocates for these string conversions.
    return string(a) == string(b)
}

And here's the C version:

// bytes_string reinterprets a byte slice as a string (zero-copy).
#define so_bytes_string(bs) ({                  \
    so_Slice _bs = (bs);                        \
    (so_String){(const char*)_bs.ptr, _bs.len}; \
})

// string_eq returns true if two strings are equal.
static inline bool so_string_eq(so_String s1, so_String s2) {
    return s1.len == s2.len &&
        (s1.len == 0 || memcmp(s1.ptr, s2.ptr, s1.len) == 0);
}

// Equal reports whether a and b are the
// same length and contain the same bytes.
bool bytes_Equal(so_Slice a, so_Slice b) {
    return so_string_eq(so_bytes_string(a), so_bytes_string(b));
}

Just like in Go, the so_bytes_string ([]byte → string) macro doesn't allocate memory; it just reinterprets the byte slice's underlying storage as a string. The so_string_eq function (which works like == in Go) is easy to implement using memcmp from the libc API.

Another example is the IndexByte function, which looks for a specific byte in a slice. Here's the pure-Go implementation:

// IndexByte returns the index of the first instance
// of c in b, or -1 if c is not present in b.
func IndexByte(b []byte, c byte) int {
    for i, x := range b {
        if x == c {
            return i
        }
    }
    return -1
}

And here's the C version:

// IndexByte returns the index of the first instance
// of c in b, or -1 if c is not present in b.
so_int bytes_IndexByte(so_Slice b, so_byte c) {
    for (so_int i = 0; i < so_len(b); i++) {
        so_byte x = so_at(so_byte, b, i);
        if (x == c) {
            return i;
        }
    }
    return -1;
}

I used a regular C for loop to mimic Go's for-range:

• Loop over the slice indexes with for (so_len is a macro that returns b.len, similar to Go's len built-in).
• Access the i-th byte with so_at (a bounds-checking macro that returns *((so_byte*)b.ptr + i)).

But Equal and IndexByte don't allocate memory. What should I do with Repeat, since it clearly does? I had a decision to make.

Allocators

The Go runtime handles memory allocation and deallocation automatically. In C, I had a few options:

• Use a reliable garbage collector like Boehm GC to closely match Go's behavior.
• Allocate memory with libc's malloc and have the caller free it later with free.
• Introduce allocators.

An allocator is a tool that reserves memory (typically on the heap) so a program can store its data structures there. See Allocators from C to Zig if you want to learn more about them.

For me, the winner was clear. Modern systems programming languages like Zig and Odin clearly showed the value of allocators:

• It's obvious whether a function allocates memory or not: if it has an allocator as a parameter, it allocates.
• It's easy to use different allocation methods: you can use malloc for one function, an arena for another, and a stack allocator for a third.
• It helps with testing and debugging: you can use a tracking allocator to find memory leaks, or a failing allocator to test error handling.

An Allocator is an interface with three methods: Alloc, Realloc, and Free. In C, it translates to a struct with function pointers:

// Allocator defines the interface for memory allocators.
typedef struct {
    void* self;
    so_Result (*Alloc)(void* self, so_int size, so_int align);
    so_Result (*Realloc)(void* self, void* ptr,
        so_int oldSize, so_int newSize, so_int align);
    void (*Free)(void* self, void* ptr, so_int size, so_int align);
} mem_Allocator;

As I mentioned in the post about porting the io package, this interface representation isn't as efficient as using a static method table, but it's simpler. If you're interested in other options, check out the post on interfaces.

By convention, if a function allocates memory, it takes an allocator as its first parameter. So Go's Repeat:

// Repeat returns a new byte slice consisting of count copies of b.
func Repeat(b []byte, count int) []byte

Translates to this C code:

// Repeat returns a new byte slice consisting of count copies of b.
//
// If the allocator is nil, uses the system allocator.
// The returned slice is allocated; the caller owns it.
so_Slice bytes_Repeat(mem_Allocator a, so_Slice b, so_int count)

If the caller doesn't care about using a specific allocator, they can just pass an empty allocator, and the implementation will use the system allocator — calloc, realloc, and free from libc.

Here's a simplified version of the system allocator (I removed safety checks to make it easier to read):

// SystemAllocator uses the system's malloc, realloc, and free functions.
// It zeros out new memory on allocation and reallocation.
typedef struct {} mem_SystemAllocator;

so_Result mem_SystemAllocator_Alloc(void* self, so_int size, so_int align) {
    void* ptr = calloc(1, (size_t)(size));
    if (ptr == NULL) {
        return (so_Result){.val.as_ptr = NULL, .err = mem_ErrOutOfMemory};
    }
    return (so_Result){ .val.as_ptr = ptr, .err = NULL};
}

so_Result mem_SystemAllocator_Realloc(void* self, void* ptr, so_int oldSize,
    so_int newSize, so_int align) {
    void* newPtr = realloc(ptr, (size_t)(newSize));
    if (newPtr == NULL) {
        return (so_Result){.val.as_ptr = NULL, .err = mem_ErrOutOfMemory};
    }
    if (newSize > oldSize) {
        // Zero new memory beyond the old size.
        memset((char*)newPtr + oldSize, 0, (size_t)(newSize - oldSize));
    }
    return (so_Result){.val.as_ptr = newPtr, .err = NULL};
}

void mem_SystemAllocator_Free(void* self, void* ptr, so_int size, so_int align) {
    free(ptr);
}