What if we could have first class SSL certs for IP addresses? You connect to the IP and verify the cert it presents you with your PKI, then switch to the desired host via SNI or some other mechanism after DHE is established. I suspect you could do this without any extra hops but I haven't really thought through how that would work.
What's the next hop for a cryptographic hash? With IP addresses, you have a heirarchy: You match on a prefix to find the router to handle the next path, and that one matches on a longer prefix to find the next hop, and so on.
That allows you to have routing tables that don't include every single host on the internet. This is what allows efficient routing to happen.