just for clarity's sake - i asked this same exact question on bluesky at the same time.
not one like, not one repost, not one reply. 100% completely ignored.
_CLEARLY_ the better converations are happening here.
@Viss Nah, I'm tired. Gonna go REST instead.
@vwbusguy the smart move
@Viss once I put a firmware tools DVD in the drive of a 1U server, kicked off a drive health scan, and left for the weekend because I knew it would take hours
@Viss what I did not know about that gen of hardware is that from the POV of the tool, the FC HBA was a “local” disk
@Viss the SAN on the other end of those fiber runs backed a lot of stuff, including the then on-prem Exchange
@Viss Sev1 tickets flew all weekend and nobody figured out it was me, including me. Next week after reading much of the carnage I just sat shaking my head and feeling lucky to not be a part of the org having to deal with that much ire from on high
@coxn whew!
@Viss at some point I read something in one of the ticket updates that makes the lightbulb go off, and I walk out to the data center floor where the old 1U box I had completely forgotten about was still happily scanning the SAN to death.
As Oisín lay dying, Saint Patrick came to him and urged him to accept Christ, that his soul may find Heaven.
"Are my friends in Heaven" asked Oiṡin.
"No," replied Patrick, "because they died as pagans."
"What about my dog?" asked Oisín.
"Dogs don't go to Heaven," answered Patrick.
"In that case," concluded Oisín. "I won't go to Heaven. I will join my friends in Hell, and together we will overthrow Satan."
@Viss *leans back in chair*
I once heard a story about a young cat that had full access to anywhere at her high school by looking innocent and busy (with a pocket full of hall pass notes, just in case), and had the ability to open any door at said high school in seconds.
Well, rumor has it she managed to find the dial-up number to the main administrative computer that housed the recorded grades for the entire district and a scribbled note with some changed passwords that just so happened to be close enough to the actual credentials.
Later that day, she supposedly managed to access that very system and rewarded her hard work with a slight grade change.
I wonder how that cat is doing these days..... >_>
@catsalad heh, excellent
@FritzAdalis @Viss @cR0w Ferris Bueller... allegedly
@Viss
it is from a time where I was young but I still remember it clear as day. I was tasked to migrate some stuff onto a new server and then I found...
*lowers voice*
the vb-script that rewrites itself every Tuesday
@xenontechs oh this seems fun
@Viss I'm pretty sure the person writing the script was either possessed or totally lost their mind during the creation. either option seems very reasonable.
maybe both, guessing from how fixing it has scarred me as well
*eye twitching*
@Viss There once was a time when the product I was responsible for was fairly new, and my wife and I went on vacation in Colorado. We went out in the backcountry in an ATV for hours. As we returned to town, my phone suddenly blew up with notifications.
On sorting through texts and voicemails, it appeared that my product was hanging on launch with the dreaded spinning beach ball, apparently for EVERY SINGLE CUSTOMER. Nobody knew what was going on.
Fortunately, I remembered an identical problem in testing when a particular change was made to the backend. With that bit of information, they were able to get the problem fixed on the backend. Unfortunately, for years afterward I was afraid of being unavailable.
@thomasareed oh i know these feels. what was the issue?
@Viss It had to do with database updates coupled with a bug in the client. The update system got into a particular state that the client was not prepared for, and the client was not resilient so it just got hung up. I don’t remember details about exactly what the problem was, though… corrupt database, bad database version info, etc.
@Viss Of course, there were other times when I was directly responsible for a problem, though not to that scale. Mostly times when I pushed a database update that started triggering a lot of false positives for something in common usage that I hadn’t tested with.
@thomasareed ive learned that if your api has an expired cert, if the client doesnt know how to raise an exception for that itll 'just fail' with no clearly discernable error. groan
@Viss Yeah, I’ve seen certificate pinning go bad because there wasn’t adequate communication between teams and a cert got changed…
@thomasareed ah, gotcha. yeah ive been bitten by that kinda plumbing before. orbital utilizes a queueing system and ive made booboos that caused it to loop forever or never finish a certain tier of scanning etc.
@Viss @catsalad
Worst I ever did was blow up a couple of workstations.
I had started a new job and we were setting up SMS (later SCCM/Config Manager) to do patching. I was having some trouble getting discovery working, the only thing showing up in the console was like two Windows 2000 SP3 machines. SP4 was current, so I figured I'd test deployment on those.
The next morning users in one remote office are complaining about performance. It turns out overnight SMS discovered all 500+ Win2k SP3 machines at that office, and they were all installing it over the wan at the same time from an HP Vectra P3 with a 5400 rpm IDE drive.
So, it probably would have eventually succeeded, except my coworker decided to yank the power. 500+ machines died in the middle of a service pack upgrade.
So what to do? An entire subdivision had bricks for PCs. So I created a Win2k ISO with SP4 already applied, sent it over to the on-site techs, and told them to boot from CD and do an in-place upgrade. It worked! And they all got upgraded to SP4.
And I learned that tasks in SMS were slow to pick up, but they eventually worked whether you wanted them to or not.
@FritzAdalis @catsalad whoooooof thats brutal
"And I learned that tasks in SMS were slow to pick up, but they eventually worked whether you wanted them to or not."
We always used to joke that SMS would wait until you opened the log file (to find out why it hadn't started) before actually starting the job.
Kinda "oh shit, they're checking up; everyone look busy".
@Viss I was part of the network operations team for a NASA supercomputer project back in the early 1990s. One of my jobs was handling DNS for our domains.
For historical reasons, our host information was actually maintained in /etc/hosts format, which was then massaged through a fairly ugly collection of scripts into DNS zone files. One of my tasks was to clean things up and consolidate the script pipeline into something more maintainable.
I slung some Perl code, produced some nice looking zone files, and out of sheer hubris/ignorance pushed them to our primary name server without doing much (if any) testing. And that was the day I learned an important lesson about the importance of dots at the end of FQDNs in your zone files, because the zone files I produced didn’t have any.
And because I’d updated our primary server, all of the secondary servers got the broken zone files as well. And then our whole network broke because name resolution was totally FUBAR.
We actually had to “sneaker net” corrected zone files to the primary. Then we had to visit the console of each secondary server, clear the broken zone files, and manually force a zone transfer of the fixed ones. Good times!
@hal_pomeranz fuuuuuuuuck. ive made that mistake before. first sysadmin job for a dinky dialup isp in san diego. same mistake. for periods, took down dns. i forfet what it was exactly i broke, but they got angry. i was ..20? 21?
@hal_pomeranz @Viss It's always DNS.
@Viss Yeah I mean I don't wanna tell you a story 300 characters at a time
@mttaggart apparently neither does ANYBODY on bluesky.
@Viss Jokes aside though, no you're totally right. I've had engagement over there but it's so low-value compared to what we get up to over here.
@mttaggart i get the imrpession that unless youre a celeb of some kind, brought tens of thousands of followers with you from another platform, or only talk about bluesky or politics, you're just going to get overlooked.
@Viss It's funny. I'm seeing a lot of the old "cool kids" show up but... absolutely zero substance.
@mttaggart the chief reason those 'cool kids' showed up in the first place was because bsky was invite only. you were only a cool kid if you ran the gauntlet to get an invite and were let in. the moment the invite only part fell off, that luster vanished overnight, and now since anybody could join, the people who worry their cool kid card was revoked are now desperately trying to keep it cool
but i dont think its working.
@Viss I actually mean newcomers whom I've not seen in years. But with few exceptions, I'm just not seeing relevant material. But hey is early days and I'm trying to keep an open mind
@Viss Also I have a story in mind for this thread once the kiddo is asleep and I have time to write it up.
@mttaggart @Viss there once was a man from...
@Viss @mttaggart I didn’t use it much for the last year because I wasn’t cool enough. My “on topic” infosec material want a fit for happy hour over there. I’m giving it another shot again and hoping that with the influx of people, the culture will change a bit.
@accidentalciso @mttaggart im not holding my breath.
@accidentalciso @mttaggart good example - just now i saw some reposts from lizzo saying something like how 'bsky is safe from propaganda and bots and racism'
she has no idea how wrong she is, and when that becomes clear its gonna be very jarring.
cuz they dont have any 'anti nazi / anti racist' policies over there, its all just 'labeling'. users are on their own in terms of moderation.
and the influx of shitheels from twitter looking for abuse victims has already begun.
@accidentalciso @Viss @mttaggart
I have a theory about this, but we shall see how it plays out
Twitter started out fun, but I don't think it's for the reasons everyone thinks it is. Nobody knew how to appease the outrage robot overlords, so everyone just did whatever, and that was fun
But some people figure it out, and that's what made it less fun
I have no doubt the robot overlords will be appeased much more quickly this time, maybe too quickly
@joshbressers @accidentalciso @Viss Very possible, but the quick-block culture is a positive complication for that dynamic. As ever, I'll remain optimistic and open to being wrong.
@mttaggart @Viss butts and dildos isn’t your thing, either, eh?
@accidentalciso @mttaggart oh theres definitely that, but theres a lot of ... posturing? folks are peacocking around going on and on about how great the place is, but if you want to have an actual discussion, you get ignored because they're too busy peacocking.
@winterknight1337 @accidentalciso @mttaggart oh you didnt go check out the feeds?
@Viss @accidentalciso @mttaggart I guess not those feeds
@mttaggart
> I'm seeing a lot of the old "cool kids" show up but... absolutely zero substance
I bet Blaze loves it ...
@strypey @mttaggart this thread is still getting bites too, the second day, and the one person who responded over on bsky came here to add a second story because they saw that nobody was interacting over on bsky. so like, thats another measure
@Viss I’m still trying to wrap my head around the differences between platforms.
@accidentalciso the 100,000 foot view is "bsky looks and feels like old twitter, and you dont have to make any 'decisions' while signing up like you do for mastodon'
and because it was invite only for a while, theres this sorta 'cool kids club' factor surrounding it.
i think it gets a lot more attention because they DIRECTLY AVOIDED the 'mastodon hoa' factor, and didnt chase new users away with screaming demands and condescension.
then they got a handful of celebs.
@Viss dude, my cat pictures barely get any traction there.
And, yeah, it’s because we have common hashtags here and… some sort of feed concept there? Which it’s not obvious how to get cat pictures to the cat picture enjoyers without reading documentation instead of using the same thing that every social network has done for the past decade+.
@Viss Yeah, there are better responses here tbh.
heck, I'll toss another here instead of bsky for the cool peeps. Malicious compliance type
/me throws backup tapes into fire to stoke it up a bit
So back when I was a lowly support tech at a backup company you've probably heard of, I got one hell of a call. It was another company who was a reseller of our product. we were tier 2 support for them and we were tier 1 normally for our own customers. They were walking one of their customer through a disaster recovery test, validating the backups like ALL OF YOU READING THIS SHOULD. On these calls, we're just supposed to be silent observers unless shit hits the fan then we jump in and save the exercise so they can still get a compliance cert or validate SLA's or whatever. the reseller just has us on the line as immediate tier 2 support waiting on the call to rescue them if crap happens.
When the call rolled in to me, i knew exactly who it was and knew that the rep was an arrogant prick from past experience. Great...
During the test, they needed to stand up a new vault server and point it at the right cloud server to replicate the data from cloud to local. I had a list of the correct vault domains for the reseller, and when they were going down their list of recovery info with the client I noticed something sounded off. I was basically a greybeard at this point within the support staff after being there a whole 2 years, and that domain sounded wrong. I looked in my system while they kept working, sure as heck, it was incorrect. about a month earlier they had migrated to a new domain, and while the new one worked for endpooint clients because DNS records had been updated for the endpoint backup clients to not need new configs, the vault servers used a different domain entirely from the clients and there were no DNS records anymore at all for the old domain. If the old config was used on any vault servers, they would fail.
I piped up and said something like I should. I straight up said "i believe you have the incorrect domain there." the reseller's tech being the arrogant prick he was cut me off and was adamant I was wrong. He told me in a quite rude tone "I am positive this is right, if I need your opinion then I'll ask for it! Until then let ME handle it!"
Okay boss, you've got it. Have fun!
So knowing the call is being recorded and captured that lovely little exchange, I let them continue. The customer bought into everything the dude said as well. He was a bit stressed and had no idea who I was or why I was even on the call. The reseller called us before getting the customer on the line as per standard procedure. so he understandably was skeptical of the info I had and trusted his rep. The poor soul... So I let them continue.
all went well until they got the new vault server stood up, the new hypervisor stood up for the VMs to be restored to, and were trying to replicate the data from the cloud to the on-site vault for restores. They kept troubleshooting, and testing, and checking logs. All sorts of errors, failures related to communication, and I dutifully performed my task as I should, simply answering questions, validating the info is correct when they ask me, validating this registry key is right, or what this log means. My shift was 10 hours long, this was my first call of the day, lets see how long they'll go.
After a few hours, they tore everything down, rebuilt it all and it still didn't work. they tried setting up another on site vault and validate they could connect, they asked tons of questions and I either corrected minor misconfigs or just confirmed it was correct. By hour 9 the test was completely failed. They were just trying to figure out why backups weren't able to be restored now. My shift was almost over, so I asked if I can take control and troubleshoot so it didn't have to be transferred to another tech on my team. Just to get the poor customer on my side so they'd listen to me (who was quite angry with his rep at this point) I introduced myself properly as tier 2 support for the cloud storage solution. It suddenly clicked in that guys head why I was on the call and what I had said earlier. he then asked:
"wait, clear back at the start of this call you mentioned something before $rep told you to shut up. What was it you mentioned?"
"Oh, I recognized the domain you're attempting to connect to appears to be the old one. I mentioned this however as $rep was adamant that the data was correct I allowed him to continue as per disaster recovery test procedure. We are tier 2 support and should only interject when there is a problem unresolvable by tier 1. Would you like me to validate that domain for you?"
"of course, please do"
So I did. It was wrong. as soon as I tried the new domain, it just worked. within 15 min of me providing the correct data that the other rep refused I had data flowing in saturating a 2 gig connection.
So right after data start flowing that other rep immediately tried throwing me under the bus.
"see this is why we do these tests. $k1n3ks_employer failed at providing us the correct info preventing us from.." and the customer cuts him off and just starts SCREAMING at him about if he would have just listened they would have been done, how this is the last straw and they're getting a lawyer as the SLA agreement was broken, yadda yadda. Because it ended up in a screaming match between the rep and the customer, I had to report it to my manager. He had to get on the call. He eventually had to listen to the entire call to see if I had done anything wrong as the reseller lost the client (to us ironically) and they were pissed off and out for blood.
Well, because of the reseller rep's statement telling me to shut up when I identified the problem in the first 10 minutes of the call, that dude got fired, I didn't get in any trouble and had a super easy day at work lol. I left the company shortly after but dayum was that a satisfying day after it was all over.
@k1n3k ive had similar exchanges in the past, when i worked support at websense. i was the only linux/unix guy out of maybe 35 staff. i routinely had a line outside my cube with folks asking for help