i was a very early employee at one of the first two commercial US ISPs, UUNET.
all of us were using cisco routers. there was a bug where at about 24-26 hours, the BGP session in them crashed, causing waves of BGP convergence issues, affecting all the tier 1 providers.
the solution was to reboot carefully at about 22 hours. we waited desperately for cisco to get us a patch.
we were at a usenix conference in the terminal room at about 11pm when we got an email that they'd isolated the bug and thought they had a fix. at this point, the senior router folks for UUNET, MCI, and Sprint sat in the terminal room, waiting.
about 1:30am we got email that tony thought he'd fix it. we asked how much testing he'd been able to do.
tony: "it booted in the lab"
us: "we'll take it!"
then, all 6 of us downloaded the code and proceeded to reboot about 70% of the internet backbone to start using the new code, all of UUNET, MCI, & Sprint.
fortunately, in those days, cisco let their senior engineers talk directly to customers and they got things fixed fast.
this never made the newspapers at the time. i can't even imagine any ISP these days allowing engineers to reboot their entire backbone from a terminal room with code that "booted in the lab".
kind of miss those old days.
@paul_ipv6 wow thats nuts! i bet you were happy it worked though. im sure the buttclench was real
after having to reboot every night for weeks and hoping nothing went wrong, it seemed worth the risk. :D
all sorts of fun back in those days.
less catastrophic story.
we had more UUCP than TCP customers early on. we had about 3000 UUCP customers. we had a billing script, written in BSD awk, that took UUCP log files, figured out connect time, created individual invoices per customer, then created this massive postscript job that printed out the 3000 invoices to be snail mailed out.
the awk script was so complex that the BSD developers used it as the final acid test after all beta testers had blessed new versions of awk. if it could survive doing a billing run for us, it shipped in the next release of BSD.
@paul_ipv6 whoa! does that make you bsd royalty?
or "you did *WHAT* with our software?!?!?!?!" :)
we used to call UUNET "where production code comes to die".
one thing i miss these days is being able to break things but being able to work directly with the folks writing the code and get fixes. BGP4, DNS, and other stuff all work better thanks to those days.
@paul_ipv6 @Viss It's nice to hear the lineage of that sacred role, thanks.
@paul_ipv6 @Viss BSD you mean BSDi or? ;) I ran a UUCP node at home years ago before us in the GWN had the Internet. ;)
actual BSD, before the folks went off to start BSDi. we did run/support BSDi too, with UUNET doing all the shipping of OS CDs, etc. i miss BSDi. it just ran and ran.
@paul_ipv6 @Viss Sorry for some reason I missed this reply. Ah yes. AT&T lawsuit days. I worked at BSDi towards the end. I did a lot of powerpc work.
yeah, proud moment for me that i got to wear one of the "i've been compromised" (or similar) since i'd seen at&t source code during all that foolishness.
BSDi was a great crew; loved working with them and supporting them.
UUCP reminds me of a minor story from my youth. Mid-nineties I got myself a colocated server so I could have a private always on computer with email and web, long before "ec2".
Since it was all mine, I went wild with RFC 822 edge cases and picked an email address of <#@qz.to>. Later I got reports of it breaking UUCP email because the # was being treated as the start of a comment.
So I changed it to * in the local part (which I still use but sadly do not have qz.to anymore).
email addrs were exciting. we had to eat the entire UUCP map project into pathalias, had hacks to have our sendmail grovel that directly. also had BITNET, X.400/MHS, "normal" SMTP/DNS. !, %, @ all had precedence depending. i still twitch if someone suggests i debug a ruleset 3 issue with sendmail. i tell them to just run postfix and be happy. ;)
@elithebearded Some people just like to watch the world burn. ;)
@paul_ipv6 @Viss i aspire to this level of integration test. legendary stuff!
@paul_ipv6 This is a good example of end-to-end testing which I'd love to see more widespread; testing new releases against real customer workloads before releasing them rather than just dropping them in the void and waiting to hear something back.
@paul_ipv6 @Viss That brings back memories of my first undergrad job outside uni: Although late 2000s it involved extending an XML parser written in awk because somehow the multinational telco equipment provider wouldn't make a scripting language with XML bindings available on the Solaris 10 servers. Seems telcos just love awk!