One of these sounds is made at a specific place in your mouth. The other has no position at all. It is just breath, shaped by whatever vowel happens to follow.
Where Japanese H Drifts
Japanese h shifts to a breathy, almost f-like sound before certain vowels, which leaves many Japanese speakers carrying a single mental category over territory English splits firmly in two. That explains why this pair can feel like a distinction without a difference at first. Once you know that f has a fixed contact point and h has none, the two stop competing: you are no longer comparing two similar sounds, you are checking whether contact happened.
One Has a Location. The Other Has None.
This is the unusual thing about this pair, and the key to it.
f in fat
Made at a specific, fixed place: top teeth on bottom lip, air forced through a narrow gap. The position is the same no matter which vowel comes next.
h in hat
Has no mouth position at all. Nothing touches anything. It is just breath passing through a mouth already shaped for the vowel that follows.
Which Means You Can See It
Because f requires a specific contact and h requires none, this pair is one of the easiest to check visually. If the speaker's top teeth touched their bottom lip, it was f. If the mouth simply opened into the vowel with no contact anywhere, it was h. Nothing subtle is involved. This also explains why h is so easy to lose entirely — with no contact point holding it in place, it is barely more than breath, and it routinely disappears in fast speech and in several English accents. If you thought a word started with a vowel and it turned out to have an h, that was not a mistake on your part.