Go proposal: Goroutine metrics

Part of the Accepted! series, explaining the upcoming Go changes in simple terms.

Export goroutine-related metrics from the Go runtime.

Ver. 1.26 • Stdlib • Medium impact

Summary

New metrics in the runtime/metrics package give better insight into goroutine scheduling:

Total number of goroutines since the program started.
Number of goroutines in each state.
Number of active threads.

Motivation

Go's runtime/metrics package already provides a lot of runtime stats, but it doesn't include metrics for goroutine states or thread counts.

Per-state goroutine metrics can be linked to common production issues. An increasing waiting count can show a lock contention problem. A high not-in-go count means goroutines are stuck in syscalls or cgo. A growing runnable backlog suggests the CPUs can't keep up with demand.

Observability systems can track these counters to spot regressions, find scheduler bottlenecks, and send alerts when goroutine behavior changes from the usual patterns. Developers can use them to catch problems early without needing full traces.

Description

Add the following metrics to the runtime/metrics package:

/sched/goroutines-created:goroutines
	Count of goroutines created since program start.

/sched/goroutines/not-in-go:goroutines
	Approximate count of goroutines running
    or blocked in a system call or cgo call.

/sched/goroutines/runnable:goroutines
	Approximate count of goroutines ready to execute,
	but not executing.

/sched/goroutines/running:goroutines
	Approximate count of goroutines executing.
    Always less than or equal to /sched/gomaxprocs:threads.

/sched/goroutines/waiting:goroutines
	Approximate count of goroutines waiting
    on a resource (I/O or sync primitives).

/sched/threads/total:threads
	The current count of live threads
    that are owned by the Go runtime.

The per-state numbers are not guaranteed to add up to the live goroutine count (/sched/goroutines:goroutines, available since Go 1.16).

All metrics use uint64 counters.

Example

Start some goroutines and print the metrics after 100 ms of activity:

func main() {
	go work() // omitted for brevity
	time.Sleep(100 * time.Millisecond)

	fmt.Println("Goroutine metrics:")
	printMetric("/sched/goroutines-created:goroutines", "Created")
	printMetric("/sched/goroutines:goroutines", "Live")
	printMetric("/sched/goroutines/not-in-go:goroutines", "Syscall/CGO")
	printMetric("/sched/goroutines/runnable:goroutines", "Runnable")
	printMetric("/sched/goroutines/running:goroutines", "Running")
	printMetric("/sched/goroutines/waiting:goroutines", "Waiting")

	fmt.Println("Thread metrics:")
	printMetric("/sched/gomaxprocs:threads", "Max")
	printMetric("/sched/threads/total:threads", "Live")
}

func printMetric(name string, descr string) {
	sample := []metrics.Sample{{Name: name}}
	metrics.Read(sample)
	// Assuming a uint64 value; don't do this in production.
	// Instead, check sample[0].Value.Kind and handle accordingly.
	fmt.Printf("  %s: %v\n", descr, sample[0].Value.Uint64())
}

Goroutine metrics:
  Created: 52
  Live: 12
  Syscall/CGO: 0
  Runnable: 0
  Running: 4
  Waiting: 8
Thread metrics:
  Max: 8
  Live: 4

No surprises here: we read the new metric values the same way as before — using metrics.Read.

Summary

Motivation

Description

Example

Further reading