Okay this is wild: I just noticed that changing 'PasswordAuthentication' to 'no' in /etc/ssh/sshd_config is no longer enough to disable password authentication in #ubuntu. That's because Ubuntu Server now by default creates a sshd_config.d/50-cloud-init.conf file which contains 'PasswordAuthentication yes' which takes priority over sshd_config.
I would've unknowingly left password auth on if I hadn't double checked.
Why?
More: https://askubuntu.com/questions/1516262/why-is-50-cloud-init-conf-created