Build a Concurrent Link Checker in Go in Under 100 Lines

If you want to know why people like Go, a small project teaches more than a feature list. The tool below fits in an afternoon, is useful the day you finish it, and uses the parts of Go that matter most in real work. Those parts are goroutines, channels, contexts, the standard HTTP client, and one well-known helper package.

The project is linkcheck, a command-line tool. It reads URLs from standard input, checks them in parallel with a limit on how many run at once, prints the status code and response time for each one, and exits with a non-zero code if any link is broken. You can pipe a sitemap into it, run it in CI against your documentation, or point it at a list of webhook endpoints before a deploy.

Why this is a good Go project

Checking links is mostly I/O. Each request spends almost all of its time waiting on the network. Done one at a time, 500 URLs at 300 ms each takes two and a half minutes. Done 20 at a time, it takes under ten seconds. Go makes that change cheap: a goroutine starts with a stack of only a few kilobytes, so you can start one per URL without a second thought.

The hard part is not starting the work but controlling it. A good tool needs to:

• Cap concurrency so it doesn't flood a server or run out of file descriptors.
• Time out requests that hang.
• Stop cleanly when you press Ctrl-C.
• Report failures in a way scripts can act on.

Each of these maps onto a standard Go idiom, which is why the project works well as a learning exercise.

Setting up

Create a module and pull in the one dependency:

mkdir linkcheck && cd linkcheck
go mod init example.com/linkcheck
go get golang.org/x/sync/errgroup

errgroup lives in the golang.org/x repositories. These are maintained by the Go team but sit outside the standard library. It gives you a group of goroutines you can wait on, and since SetLimit was added it can also cap how many run at once. That one method replaces the worker-pool code people used to write by hand.

The code

Here is the whole program in main.go:

package main

import (
	"bufio"
	"context"
	"flag"
	"fmt"
	"net/http"
	"os"
	"os/signal"
	"strings"
	"time"

	"golang.org/x/sync/errgroup"
)

type result struct {
	URL     string
	Status  int
	Latency time.Duration
	Err     error
}

func check(ctx context.Context, client *http.Client, url string) result {
	start := time.Now()
	req, err := http.NewRequestWithContext(ctx, http.MethodHead, url, nil)
	if err != nil {
		return result{URL: url, Err: err}
	}
	resp, err := client.Do(req)
	if err != nil {
		return result{URL: url, Err: err, Latency: time.Since(start)}
	}
	resp.Body.Close()
	return result{URL: url, Status: resp.StatusCode, Latency: time.Since(start)}
}

func main() {
	limit := flag.Int("c", 16, "maximum concurrent requests")
	timeout := flag.Duration("t", 5*time.Second, "per-request timeout")
	flag.Parse()

	ctx, stop := signal.NotifyContext(context.Background(), os.Interrupt)
	defer stop()

	client := &http.Client{Timeout: *timeout}
	results := make(chan result)

	var g errgroup.Group
	g.SetLimit(*limit)

	go func() {
		sc := bufio.NewScanner(os.Stdin)
		for sc.Scan() {
			if ctx.Err() != nil {
				break
			}
			url := strings.TrimSpace(sc.Text())
			if url == "" || strings.HasPrefix(url, "#") {
				continue
			}
			g.Go(func() error {
				results <- check(ctx, client, url)
				return nil
			})
		}
		g.Wait()
		close(results)
	}()

	failed := 0
	for r := range results {
		switch {
		case r.Err != nil:
			failed++
			fmt.Printf("ERR  %-60s %v\n", r.URL, r.Err)
		case r.Status >= 400:
			failed++
			fmt.Printf("%d  %-60s %v\n", r.Status, r.URL, r.Latency.Round(time.Millisecond))
		default:
			fmt.Printf("%d  %-60s %v\n", r.Status, r.URL, r.Latency.Round(time.Millisecond))
		}
	}

	if failed > 0 {
		fmt.Fprintf(os.Stderr, "%d link(s) failed\n", failed)
		os.Exit(1)
	}
}

Build it and try it:

go build -o linkcheck .
printf "https://go.dev\nhttps://example.com\nhttps://example.com/nope\n" | ./linkcheck -c 4

How it works

One goroutine per URL, with a cap

g.SetLimit(*limit) is the key line. After it, g.Go blocks whenever the limit is already reached, and returns once a running goroutine finishes and frees a slot. So the loop that reads standard input slows itself down: it can't get more than -c requests ahead of the network. You get the effect of a worker pool without managing one. There's no job channel, no fixed set of workers, and no manual sync.WaitGroup bookkeeping.

The closure passed to g.Go uses url, so it's worth knowing that since Go 1.22, every iteration of a for loop gets its own copy of the loop variables. This applies to modules whose go.mod declares go 1.22 or later. In this program url is declared inside the loop body anyway, so it's safe on any version. Still, for modern modules the old "every goroutine sees the last URL" bug is gone from the language.

A producer, a consumer, and a channel between them

The reading-and-starting work runs in its own goroutine, and main reads from results. This split matters. If main both started the work and waited for it, nobody would be reading the unbuffered channel, and every worker would block forever on its send. With the split, the producer calls g.Wait(), which returns once all workers have finished, and then closes the channel. That ends the for r := range results loop in main. The rule to remember: the side that sends is the side that closes.

Printing from a single goroutine also means output lines never interleave. You get thread safety without writing a mutex.

Timeouts and cancellation

There are two separate controls:

1. http.Client{Timeout: ...} puts a hard upper bound on each request. It covers connecting, TLS, any redirects, headers, and reading the body.
2. signal.NotifyContext returns a context that is cancelled when you press Ctrl-C. Every request is built with http.NewRequestWithContext, so cancelling that context aborts all in-flight requests at once. They return straight away with a context canceled error. The producer stops queuing new URLs, the channel drains, and the program exits with status 1 instead of hanging.

Passing a context.Context as the first argument to anything that does I/O is one of the most important habits in Go, and this program shows why: one cancellation stops the whole tree of work.

Exit codes for automation

The program exits with status 1 if anything failed, so it fits straight into a shell pipeline or a CI step:

curl -s https://example.com/sitemap.xml \
  | grep -o '<loc>[^<]*' | sed 's/<loc>//' \
  | ./linkcheck -c 32 -t 10s

If a link breaks, the build fails.

Making it production-worthy

The core is finished, but here are improvements, roughly in order of how much they help:

• Fall back to GET. Some servers answer HEAD with 405 Method Not Allowed or handle it badly. If you get a 405, try again with GET and close the body without reading all of it.
• Retry with backoff. Network errors and 429/503 responses are often temporary. Two retries with jittered exponential backoff remove most false alarms in CI.
• Limit per host. A global limit of 32 can still send all 32 requests to one small server. Keep a map of per-host semaphores (a buffered chan struct{} works well) and be polite.
• Set a User-Agent. Some CDNs block Go's default User-Agent (Go-http-client/1.1, or Go-http-client/2.0 over HTTP/2). Send something honest like linkcheck/0.1 (+https://your.site).
• Structured output. Add a -json flag that writes one JSON object per line using encoding/json. Then the output can be piped into jq or a dashboard.
• Crawl mode. Parse returned HTML with golang.org/x/net/html, pull out href attributes on the same domain, and feed them back into the queue, keeping a map[string]bool of visited URLs. At that point it's a real site crawler. Watch for the extra coordination: the producer can no longer tell when it's done just by reaching the end of standard input.
• Tests. net/http/httptest.NewServer lets you start a local server that returns 200, 404, or hangs on purpose. Your tests can then check timeouts and exit codes without touching the internet.

Shipping it

One of Go's biggest practical advantages shows up at the end. go build produces a single self-contained binary with no runtime to install. (On Linux, a native build that uses net may link against the system C library. Set CGO_ENABLED=0 for a fully static binary.) Cross-compiling takes one command, and cgo is turned off by default when you cross-compile:

GOOS=darwin GOARCH=arm64 go build -o linkcheck-mac .
GOOS=windows GOARCH=amd64 go build -o linkcheck.exe .

Put those on a GitHub release, or publish the module under a real path (for example github.com/you/linkcheck, not the placeholder example.com/linkcheck). Then anyone with Go installed can run go install github.com/you/linkcheck@latest, and you've shipped a tool.

The takeaway

In under 100 lines, this program uses bounded concurrency, cancellation, timeouts, a producer/consumer channel, and Unix-style composability. These are the same patterns behind Go's larger infrastructure: proxies, crawlers, job runners, and API gateways all have a small loop like this one at their center. Build this one, then extend it. Each improvement on the list teaches another part of the language, and you'll have a tool you keep using.