Simpler & Faster Concurrent Testing With testing/synctest

In Go 1.24 the Go team introduced the experimental testing/synctest package. With Go 1.25 we now see a version of that package being officially added to the standard library - testing/synctest. In this post we are going to explore a situation where the package is useful to help better understand its goals and purpose.

Warning: This may not be easy to digest if you are a beginner since it covers some pretty specific testing situations, so don’t be discouraged if you don’t understand everything. It will come with time.

A Flaky Interval Test

Imagine that we are building an simple application to monitor the health of a remote server and notify us if it goes down. To do this, we might start by creating a function similar to the one below.

// EveryInterval is a funciton that can be used to run a function every set
// interval. For instance, it might be used to check the health of a service
// every 1 minute.
func EveryInterval(ctx context.Context, interval time.Duration, work func()) {
  ticker := time.NewTicker(interval)
  defer ticker.Stop()
  go func() {
    for {
      select {
      case <-ctx.Done():
        // Context is done, so we exit
        return
      case <-ticker.C:
        work()
      }
    }
  }()
}

Unfortunately, this code has a bug. We are calling defer ticker.Stop() in the wrong place, so the ticker will stop pretty much immediately leaving us with a useless goroutine.

We decide to fix our code by relocating the defer statement to inside the goroutine, and add a test to ensure it never happens again. We start with a test like the one below.

func TestEveryInterval(t *testing.T) {
  ctx, cancel := context.WithCancel(context.Background())
  defer cancel()

  var counter atomic.Int32
  EveryInterval(ctx, 1*time.Millisecond, func() {
    counter.Add(1)
  })

  want := 100
  time.Sleep(time.Duration(want) * time.Millisecond)
  if got, want := counter.Load(), int32(want); got != want {
    t.Errorf("counter = %d; want %d", got, want)
  }
}

And a quick update to our EveryInterval function.

func EveryInterval(ctx context.Context, interval time.Duration, work func()) {
  ticker := time.NewTicker(interval)
  go func() {
    // Move this inside the goroutine
    defer ticker.Stop()
    // ... nothing else changes
  }()
}

We then proceed to run our tests and see that the test may pass form time to time, but it can also fail every once in a while.

go test --race
# --- FAIL: TestEveryInterval (0.10s)
#     main_test.go:23: counter = 99; want 100
# Context done
# FAIL
# exit status 1
# FAIL  github.com/joncalhoun/synctest125  0.296s

go test --race
# Context done
# PASS
# ok    github.com/joncalhoun/synctest125  0.510s

This is what is known as a flaky test. It is a test that will pass from time to time - possibly even most of the time - but it can fail in some circumstances. Flaky tests are quite common when testing concurrent code that is dependent on time.

In our particular case the test will fail fairly often due to the small time intervals being used. As a result, we might try to fix it by changing the interval to be every second, and then waiting something like 5.5 seconds to test the results. (Note: this is not a great solution, as it is still a flaky test. A better solution is shown below.)

func TestEveryInterval_flakyV2(t *testing.T) {
  ctx, cancel := context.WithCancel(context.Background())
  defer cancel()

  var counter atomic.Int32
  EveryInterval(ctx, 1*time.Second, func() {
    counter.Add(1)
  })

  want := 5
  time.Sleep(time.Duration(want)*time.Second + 500*time.Millisecond)
  if got, want := counter.Load(), int32(want); got != want {
    t.Errorf("counter = %d; want %d", got, want)
  }
}

Our tests might pass more often now, but they could still potentially fail. To make matters even worse, our test is now much slower, taking a full 5.5 seconds to run.

go test
# PASS
# ok    github.com/joncalhoun/synctest125  5.686s

Not So Great Solutions (pre-synctest)

Prior to the addition of synctest, most of the ways to address this problem were pretty bad. As we saw in the last code snippet, using larger time intervals would slow our tests down. A similar approach might be to test if the value we received is within a set range, rather than an exact value.

got := int(counter.Load())
if got < 98 || got > 102 {
  // Got is not in the expect range of 98 - 102
  t.Errorf(...)
}

While this might allow for faster tests, it introduces the possibility of a bug that our code doesn’t catch.

The most reliable solution pre-1.25 would likely be to add a new Ticker interface and use it for our EveryInterval function.

type Ticker interface {
  Stop()
  C() <-chan time.Time
}

func EveryIntervalWithTicker(ctx context.Context, ticker Ticker, work func()) {
  go func() {
    defer ticker.Stop()
    for {
      select {
      case <-ctx.Done():
        // Context is done, so we exit
        return
      case <-ticker.C():
        work()
      }
    }
  }()
}

This would give us the ability to implement both a real Ticker and a fake one to use for testing. We could then control exactly how many times the fake ticker ticks during a test, giving us a more reliable test.

While this approach may result in a faster and more reliable test, the cost is quite high. We end up adding a good bit of extra code that not only consumes developer time, but it also makes the EveryInterval function harder to both read and use.

synctest to the Rescue

The testing/synctest package simplifies this type of testing. To get started, we wrap our test in a synctest.Test function call.

func TestEveryInterval_synctest(t *testing.T) {
  synctest.Test(t, func(t *testing.T) {
    // test code goes here
  })
}

Code running inside of the synctest.Test function runs inside of a “bubble.” You can read more about this in the package docs, but the oversimplified version is that the synctest package uses a fake clock that only moves forward when all goroutines are blocked. When this occurs, time will move forward the bare minimum amount of time required to allow a goroutine to unblock.

Using a fake time allows our test code to run much faster, but it doesn’t quite solve our flaky test problem. We can see this in the example below.

// This will run much faster, but it will still be flaky
func TestEveryInterval_synctest(t *testing.T) {
  synctest.Test(t, func(t *testing.T) {
    ctx, cancel := context.WithCancel(context.Background())
    defer cancel()

    var counter atomic.Int32
    EveryInterval(ctx, 1*time.Second, func() {
      counter.Add(1)
    })

    want := 100
    time.Sleep(time.Duration(want) * time.Second)

    // This will often fail with counter being 99 instead of 100.
    if got, want := counter.Load(), int32(want); got != want {
      t.Errorf("counter = %d; want %d", got, want)
    }
  })
}

The test is flaky because on the 100th second both the ticker in EveryInterval, and the time.Sleep in our test will both be unblocked at the same time. When this happens, our test can proceed to check if the counter is 100, but the function we passed to EveryInterval that increments the counter may not have finished running yet.

There are two ways that we can fix this:

1. We can add a small time interval like 10ms to the time.Sleep call.
2. We can use synctests’ new Wait function to block our test until all goroutines are blocked, then the test can proceed.

The first approach will work because the 100th second will trigger the ticker in EveryInterval, but it will not trigger the time.Sleep in our test. Then once the work in our interval is complete all goroutines will block again, resulting in the time moving forward by 10ms to unblocking our time.Sleep.

The second approach is likely the better option, as it doesn’t require us to move the clock forward before our test proceeds. Instead, we allows every goroutine that unblocks due to the last time increase to finish running before moving on with the test.

func TestEveryInterval(t *testing.T) {
  synctest.Test(t, func(t *testing.T) {
    ctx, cancel := context.WithCancel(context.Background())
    defer cancel()

    var counter atomic.Int32
    EveryInterval(ctx, 1*time.Second, func() {
      counter.Add(1)
    })

    want := 100
    time.Sleep(time.Duration(want) * time.Second)
    // Wait for all goroutines to be blocked, then proceed without
    // moving the clock forward.
    synctest.Wait()

    if got, want := counter.Load(), int32(want); got != want {
      t.Errorf("counter = %d; want %d", got, want)
    }
  })
}

Another added benefit to using synctest is having a clock that starts at exactly midnight UTC 2000-01-01 and moves forward by the exact amount of time required to unblocked a goroutine. In our code we can see this if we print out time.Now() every time our interval runs, or after the synctest.Wait call. In each case we will see that the time moves forward by exactly 1 minute without any millisecond drift.