Andrew Katsikas - Software Engineer

What I learned about HTTP caching the hard way

I run a self-hosted file-sharing service called slskd on a Raspberry Pi. After a routine update one day, a feature broke. The “Search for Additional Files in This Directory” function was returning errors. A hard refresh of the page fixed it immediately. I noted it was odd, moved on and forgot about it.

Then it happened again after using it on another device.

This time I dug in. The browser was running old JavaScript that referenced an API endpoint that had been removed in the new version. The HTML file the browser loaded was stale. It was pointing at assets from the previous build. A refresh forced the browser to fetch the current index.html, which referenced the new hashed JS filenames, and everything worked again.

The fix turned out to be a single line of code. Getting there taught me more about browser caching than I expected.

How modern frontend builds handle caching

slskd (and most Single Page Apps (SPAs) built with webpack or Vite) content-hash their JS and CSS filenames on every build. A file called app.js becomes app.d5d572a5.js and the hash changes whenever the content changes. This means every deployment produces new URLs for changed assets, which naturally busts any cached version in the browser. These assets are safe to cache aggressively:

Cache-Control: public, max-age=31536000, immutable

Setting it to “One year, immutable” means the browser will never revalidate. This is correct because if the content changes, the URL changes.

index.html is different. It’s the entry point that references all those hashed filenames. It can’t be hashed itself. The browser has to fetch it by a fixed URL to know what assets to load. And in a SPA, that same index.html is served for every route in the app, so a cached stale copy affects the whole application.

If index.html is cached and the hashes in it are from a previous build, the browser loads the old JS. If the old JS has been evicted from cache, it fetches it from the server, which may no longer have it. If it’s still cached, the browser runs old code against a new API. Either way, things break.

What was actually happening: heuristic caching

slskd didn’t set a Cache-Control header on index.html. Neither did the response include Expires. So why was the browser caching it?

The answer is heuristic caching. When a response has no explicit caching directives, browsers are permitted to cache it anyway using their own judgment. The spec describes this as “a workaround that came before Cache-Control support became widely adopted.” Heuristic caching can apply to any 200 GET response with no Cache-Control.

The MDN docs put it plainly: basically all responses should explicitly specify a Cache-Control header.

A detour through my reverse proxy

Before fixing this upstream, I experimented with enforcing it at my Go reverse proxy, which sits in front of several self-hosted services. The httputil.ReverseProxy type exposes a ModifyResponse hook that fires after the upstream responds:

proxy.ModifyResponse = func(resp *http.Response) error {
    if resp.Request.URL.Path == "/" {
        resp.Header.Set("Cache-Control", "no-store")
    }
    return nil
}

This works regardless of what the upstream sends. Set replaces any existing value. It’s a valid approach, and I also used it to think through the right directives for different asset types. Hashed JS and CSS files could be set to immutable at the proxy layer since they’re safe to cache forever.

But a proxy workaround isn’t the right fix. The behavior should be correct at the source.

The actual fixes

I opened issues on both slskd and Navidrome, a music server I run, and submitted PRs for both.

slskd uses ASP.NET’s static file middleware. The StaticFileOptions type exposes an OnPrepareResponse callback that fires before each file is served:

fileServerOptions = new FileServerOptions
{
    FileProvider = new PhysicalFileProvider(contentPath),
    RequestPath = string.Empty,
    EnableDirectoryBrowsing = false,
    EnableDefaultFiles = true,
    StaticFileOptions =
    {
        OnPrepareResponse = ctx =>
        {
            if (ctx.File.Name == "index.html" || ctx.File.Name == "default.html")
            {
                ctx.Context.Response.GetTypedHeaders().CacheControl =
                    new CacheControlHeaderValue { NoCache = true };
            }
        }
    }
};

no-cache is appropriate here. index.html is a static file with no sensitive content. The browser must revalidate before using it, but the server can respond with a 304 Not Modified if nothing has changed, saving bandwidth.

My PR for slskd is here.

Navidrome is a Go server, and its index.html is handled differently. It’s dynamically generated on each request, with server config and potentially an auth token injected into the template. The fix was a single line in serve_index.go:

w.Header().Set("Content-Type", "text/html")
w.Header().Set("Cache-Control", "no-store")

no-store is appropriate here because the response may contain sensitive data. Unlike no-cache, no-store tells the browser not to persist the response at all, neither to disk, nor to memory. The tradeoff is that every request results in a full download rather than a cheap 304, but for a dynamic response containing auth information that’s the correct behavior.

My PR for Navidrome is here.

Both PRs were merged.

Takeaways

The underlying issue is straightforward once you see it: index.html in a SPA is special. It’s the one file that can’t benefit from content-hash cache busting, but it’s also the file that everything else depends on. Serving it without an explicit Cache-Control leaves its caching behavior up to the browser, and the browser will often do something reasonable until it doesn’t.

A few things worth internalizing:

The MDN HTTP caching guide is worth reading end to end if this area is new to you.

#go #golang #http #caching #self-hosted #dotnet #asp-net