Imagine you need to pre-load caches on edge nodes, of which you have several in front of a back-end. Or you want to check if you're getting same content from all edge nodes.

You would need an HTTPS-enabled, SNI-capable web crawler which you can explicitly tell either a). which domain to use in the Host header and the TLS SNI ClientHello; or b). which IP to resolve a given domain as.

Turns out can do crawling, but can't do explicit SNI or domain resolution; the other way around. 🤦‍♀️


If one does the research around it, one finds amazing things. LD_PRELOAD tricks are cute, but my favorite has to be using perl to binary-patch to change the location it looks for the hosts file at:

My answer is, as always, way too much Bash:

It's terrible, it's buggy (redirects are not handled well yet), it's slow. But it gets the job done.

· · Web · 0 · 1 · 3
Sign in to participate in the conversation

Server run by the main developers of the project 🐘 It is not focused on any particular niche interest - everyone is welcome as long as you follow our code of conduct!