While working on site updates, I came across four bugs that had nothing to do with what I was actually trying to fix. Each one fell into the category of "I thought I fixed it, but it wasn't fixed" or "the tests pass, but something is broken." What they all had in common was that nothing went wrong at the moment the problem was introduced — the symptoms only showed up the next time someone triggered a rebuild or re-ran something. Here I want to document all four, along with what I took away from them.
The template was fixed, but production stayed the same
I found the same structural problem on three different pages. The site generates page output from shared templates, but somewhere along the way the template was updated without regenerating the actual HTML being served in production.
On the profile page, someone had patched the live file directly six days earlier, which meant that the next time the page was regenerated from the template, that patch would be completely wiped out. On the changelog page, the opposite had happened: four newly added entries were missing from the template, so regenerating all pages would have erased those entries from the live site.
The one that went unnoticed the longest was the footer on the English-language "打ち出の小槌" section. For five weeks, it had been served without links to the privacy policy or terms of use. When I was tracing the cause, I found a comment in the template saying something like "don't link to pages that don't exist in English" — and for a moment I nearly convinced myself this was intentional behavior. It turned out that policy had already been reversed; only the comment had been left behind. Checking the Git history to see when the comment appeared and when the policy actually changed helped me avoid misreading the situation.
What "all PASS" didn't actually mean
This is the one I think is most worth writing down.
Running the same test suite in two different ways produced different results. One method showed everything passing; the other showed failures. When I dug into it, I found that the code responsible for checking test results was recording problems in a list but had never been written to actually report a failure back to the test runner. So from one execution path's perspective, whatever was being checked, it always looked like a success.
The actual check that was silently broken was verifying the number of navigation items in the header against a hardcoded value. When I added the development journal page, the count went up by one. When I later restored another page, it went up by two. That mismatch had gone unnoticed for over five weeks.
And throughout the work I was doing this time around, I had been reporting "all tests passing, no unintended changes" — when in fact that "all passing" said nothing at all about the checks that were broken. The basic lesson here — that tests passing is not the same as tests working — is not a new idea, but this was a concrete reminder.
How summarizing a diff hid a change
When I confirmed the addition of links in the footer, I summarized the diff and concluded there were no unintended changes. In reality, an invisible control character at the top of 21 English-language pages had also been removed at the same time. Because I was summarizing the diff in a way that ignored whitespace changes, that single-character removal was lost among other incidental whitespace shifts.
In the end it caused no real harm. Every page already declared its character encoding through a separate mechanism, so the presence or absence of that character made no difference. The problem wasn't the outcome — it was that I stated with confidence that there were no unintended changes, when there were.
The pattern across all four
All four cases share the following:
- A script that reports success only guarantees that it ran, not that things turned out as intended
- A test suite that shows all passing only guarantees those specific checks ran, not that the checks themselves are working
- When summarizing a diff, anything the summary method ignores becomes invisible — unless you're actively aware of what's being left out
- In every case, the symptom appeared the next time someone ran something, not at the moment the problem was introduced
That's why I now think it's worth having a regular way to verify that templates and production files actually match, and that tests can genuinely fail. I built that this time around.
What I put in place
I added tests to detect mismatches between templates and generated HTML, split across three sections of the site: the overall site, 打ち出の小槌, and the development journal. I also fixed the test result propagation so failures are actually reported back to the runner, and replaced the hardcoded navigation item count with something that derives the count directly from the template.
That said, there are limits to this. The item-counting approach works by simply scanning the template string, so it can't correctly handle links that are conditionally shown or hidden. If a page is built with English-only links suppressed, this counting method doesn't apply as-is. I've noted that in the code.
Also, now that failures are properly reported, the first failure encountered in a check stops any subsequent checks in that same pass from running. It's not possible to surface all problems at once — fixing and re-running is now part of the process.
What's still open
All four of the issues I found this time have been fixed, but I can't rule out that other cases of the same kind — template-production mismatches, checks that don't actually work — exist outside the areas I looked at. The detection I've set up now is not comprehensive, so for a while I'll need to keep running it in practice and watch for anything I've missed.