Solod v0.1: Go ergonomics, practical stdlib, native C interop

Solod (So) is a system-level language with Go syntax and zero runtime. It's designed for two main audiences:

Go developers who want low-level control and zero-cost C interop, without having to learn a new language or standard library.
C developers who like Go's style.

The initial version (let's call it v0) was focused on picking a subset of Go and translating it to C. The next logical step was to port Go's standard library and make it easier to interop with C. That's what the v0.1 release I'm presenting today is all about.

Standard library • SQLite bindings • Persistent map • Store and retrieve • Command-line interface • Performance • Wrapping up

Standard library

Solod v0.1 ships with the following stdlib packages ported from Go:

io, bufio, and fmt — Abstractions and types for general-purpose I/O.
bytes, strings, strconv, and unicode/utf8 — Common byte and text operations.
slices and maps — Generic heap-allocated data structures.
crypto/rand and math/rand — Generating random data.
flag, os, and path — Working with the command line and files.
log/slog — Structured logging.
time — Measuring and displaying time.

And a couple of its own packages:

mem — Memory allocation with a pluggable allocator interface.
c — Low-level C interop helpers.

Stdlib documentation

In the following sections, I'll demonstrate some of the v0.1 features using a simple example: a persistent key-value store backed by SQLite.

SQLite bindings

Since So doesn't provide database/sql yet, we'll call SQLite directly through its C API. To do this, let's import the necessary headers with the so:include directive and generate extern declarations using the sobind tool:

package main

import "solod.dev/so/c"

//so:include <sqlite3.h>

// SQLite constants.
//
//so:extern SQLITE_OK
const sqliteOK = 0
//so:extern SQLITE_ROW
const sqliteRow = 100
//so:extern SQLITE_DONE
const sqliteDone = 101

// SQLite types.
//
//so:extern
type sqlite3 struct{}
//so:extern
type sqlite3_stmt struct{}
//so:extern
type sqlite3_value struct{}
//so:extern
type sqlite3_callback func(any, int32, **c.Char, **c.Char) int32

// SQLite functions.
func sqlite3_open(filename string, ppDb **sqlite3) int32
func sqlite3_prepare_v2(db *sqlite3, zSql string, nByte int32, ppStmt **sqlite3_stmt, pzTail **c.ConstChar) int32
func sqlite3_step(arg0 *sqlite3_stmt) int32
func sqlite3_finalize(pStmt *sqlite3_stmt) int32
func sqlite3_close(arg0 *sqlite3) int32
func sqlite3_exec(arg0 *sqlite3, sql string, callback sqlite3_callback, arg3 any, errmsg **c.Char) int32

// more declarations...

The so:extern directive is required for constants (sqliteOK) and types (sqlite3_stmt). As for functions (sqlite3_prepare_v2), we can just declare them without a body — the transpiler will treat them as extern declarations even without so:extern.

Persistent map

With the SQLite API in place, let's implement a key-value type that wraps the database connection:

// SQLMap is a simple key-value store backed by an SQLite database.
type SQLMap struct {
    db *sqlite3
}

Add a constructor that connects to an SQLite database and creates a table to store the items:

var ErrCreate = errors.New("sqlmap: create schema failed")
const sqlCreate = "create table if not exists kv (key text primary key, val)"

// NewSQLMap creates a new SQLMap using the provided connection string.
// It opens a connection to the SQLite database and creates the underlying
// key-value table if it does not already exist.
//
// The caller is responsible for calling Close on the returned SQLMap
// when it is no longer needed.
func NewSQLMap(connStr string) (SQLMap, error) {
    var db *sqlite3
    rc := sqlite3_open(connStr, &db)
    if rc != sqliteOK {
        return SQLMap{}, ErrCreate
    }

    rc = sqlite3_exec(db, sqlCreate, nil, nil, nil)
    if rc != sqliteOK {
        sqlite3_close(db)
        return SQLMap{}, ErrCreate
    }
    return SQLMap{db}, nil
}

// Close releases resources associated with the SQLMap.
func (m *SQLMap) Close() {
    sqlite3_close(m.db)
}

As you can see, this So code looks a lot like regular Go code. However, there are some key differences:

When compiled, the code is first translated to plain C, then compiled into a native binary using GCC or Clang.
Unlike Go, there is no runtime (no automatic heap memory allocation, no garbage collection, no goroutine scheduler).
There is no overhead when calling C functions, unlike Go's Cgo.
The interop syntax is a bit cleaner. For example, Go's string (sqlCreate in the sqlite3_exec call) automatically decays to C's const char*.

Store and retrieve

First, let's implement the Set method:

var (
    ErrPrepare = errors.New("sqlmap: prepare failed")
    ErrExec    = errors.New("sqlmap: exec failed")
)

const sqlSet = "insert or replace into kv (key, val) values (?, ?)"

// Set stores a string value for the specified key.
func (m *SQLMap) Set(key string, val string) error {
    var stmt *sqlite3_stmt
    rc := sqlite3_prepare_v2(m.db, sqlSet, -1, &stmt, nil)
    if rc != sqliteOK {
        return ErrPrepare
    }
    defer sqlite3_finalize(stmt)

    sqlite3_bind_text(stmt, 1, key, int32(len(key)), nil)
    sqlite3_bind_text(stmt, 2, val, int32(len(val)), nil)

    rc = sqlite3_step(stmt)
    if rc != sqliteDone {
        return ErrExec
    }
    return nil
}

No surprises here, just a bunch of SQLite API calls.

The Get method is more interesting:

var ErrNotFound = errors.New("sqlmap: not found")
const sqlGet = "select val from kv where key = ?"

// Get returns the value associated with the specified key.
// The caller owns the returned string and must free it with mem.FreeString.
func (m *SQLMap) Get(a mem.Allocator, key string) (string, error) {
    var stmt *sqlite3_stmt
    rc := sqlite3_prepare_v2(m.db, sqlGet, -1, &stmt, nil)
    if rc != sqliteOK {
        return "", ErrPrepare
    }
    defer sqlite3_finalize(stmt)

    sqlite3_bind_text(stmt, 1, key, int32(len(key)), nil)
    rc = sqlite3_step(stmt)
    if rc == sqliteDone {
        return "", ErrNotFound
    }
    if rc != sqliteRow {
        return "", ErrExec
    }

    text := sqlite3_column_text(stmt, 0)
    tmp := c.String(text)
    result := strings.Clone(a, tmp)
    return result, nil
}

The pointer returned by sqlite3_column_text is managed by SQLite. It becomes invalid after calling sqlite3_finalize (which Get does before returning). Because of this, we need to allocate a copy of the returned value, using strings.Clone in this case.

So's approach to memory allocation is similar to Zig's — all heap allocations must be done explicitly by providing a specific instance of the mem.Allocator interface.

The caller, of course, must free the allocated string:

func main() {
    m, err := NewSQLMap(":memory:")
    if err != nil {
        panic(err)
    }
    defer m.Close()

    m.Set("name", "Alice")
    name, err := m.Get(mem.System, "name")
    if err != nil {
        panic(err)
    }
    println("name =", name)
    mem.FreeString(mem.System, name)
}

name = Alice

Here, mem.System is a specific allocator that uses libc's malloc and free. Alternatively, we could use mem.Arena or any other implementation of the mem.Allocator interface:

var buf [1024]byte // stack-allocated
arena := mem.NewArena(buf[:])

name, _ := m.Get(&arena, "name")
mem.FreeString(&arena, name) // no-op for arena; can be omitted

Command-line interface

With the SQLMap type in place, let's create a simple CLI using the flag package:

var (
    opFlag  string
    keyFlag string
    valFlag string
)

func parseFlags() {
    flag.StringVar(&opFlag, "op", "", "operation: get, set, or del")
    flag.StringVar(&keyFlag, "key", "", "key name")
    flag.StringVar(&valFlag, "val", "", "value (for set operation)")
    flag.Parse()
}

func main() {
    parseFlags()
    // ...
}

Then add command routing:

m, err := NewSQLMap("sqlmap.db")
check(err)
defer m.Close()

switch opFlag {
case "set":
    err = m.Set(keyFlag, valFlag)
    check(err)
case "get":
    val, err := m.Get(mem.System, keyFlag)
    check(err)
    println(val)
    mem.FreeString(mem.System, val)
case "del":
    err = m.Delete(keyFlag)
    check(err)
default:
    flag.Usage()
    os.Exit(1)
}

sqlmap -op=set -key=name -val=alice
sqlmap -op=get -key=name
alice

Again, no surprises here — the flag package works just as it does in Go.

Performance

Solod isn't trying to outperform hand-tuned C. Still, performance matters: the code is benchmarked and optimized to run reasonably fast. Since So compiles to plain C and then to native code with full optimizations, the results are sometimes better than Go's.

Here are some highlights from the benchmarks:

Buffered I/O is 3x faster than Go.
String and byte operations are up to 2.5x faster.
Maps are 1.5x faster for modifications.
Integer formatting is 2x faster.

There're no GC pauses and no Cgo bridge cost when calling C libraries. The tradeoff is that you have to handle memory yourself, but as the SQLite example above shows, So's allocator interface makes that pretty manageable.

Solod vs. Go benchmarks

Wrapping up

Solod is still in its early days, but with the v0.1 release, it's ready for hobby projects. The already-ported parts of the Go standard library make it easy to write command-line tools (check out the cat, head, sort, and wc examples). Plus, with native C interop, you can build just about anything else you need.

The next release (v0.2) will likely focus on networking, concurrency, or both — along with more stdlib packages.

If you're interested, take a look at So's readme — it has all the information you need to get started. Or try So online without installing anything.

★ Subscribe to keep up with new posts.