The environment is a government office. We had numerous documents with clear, numbered instructions for various things. Numpty had received one such form.

<Ring, ring> Hello, this is HA, how can I help you?

[Numpty]: WHAT'S THIS FORM YOU'VE SENT ME?

[HA]: Well, I'm not sure, what does it say at the top?

[Numpty]: It says "How to email a file".

[HA]: Excellent, and what is written below that title?

[Numpty]: Step 1.

[HA]: Ah, and what does it say next to Step 1?

[Numpty]: It says, "Open Microsoft Outlook from the Start Menu."

[HA]: Right, and have you tried that?

[Numpty]: Well no, of course not, I wanted you to tell me how to do this.

[HA]: Uh-HUH. You'd like me to talk you through it?

[Numpty]: Yes, I'd feel better with you talking me through it.

[HA]: Okay, so do you see the button at the lower left of your screen that says, "Start", with the little flying Window-y-looking logo next to it? Click on that.

[Numpty]: Left-click or Right-click?

[HA]: That would be LEFT-click ...<presses Mute button, takes a deep breath, "God help me", unmute>...

[Numpty]: Okay, I click-clicked it and something flashed up and went away.

[HA]: < ..... dear God ... > All right, I need you to just Left-click ONCE. If I need you to DOUBLE-click, I'll say "double-click", okay?

Dear reader, I'll let you use your imagination for what the rest of that call sounded like. The kicker here is that these people worked in an Education Department and were responsible for guiding the future leaders of our fine country. To get to work there, they had to have been in the system for years, using computers and writing curricula. These were not newbies.

This one popped up in my memories from about fifteen years ago and was so stupid I just had to share. I was subcontracted to a hospital, providing desktop support. A ticket came in one day for a PC that wouldn't boot.

No big deal. Check the machine out and it's just a dead CMOS battery. F1 to continue or swap the battery. I'd only been there about six weeks and didn't know where all the supplies were kept yet so I hauled the device back to my office.

While I am reaching into the case to pop the battery out, my boss walks in and starts flipping out.

"What are you doing?!"

"Um, just replacing the battery"

"You never do it like that! Who taught you?"

"Huh?"

"You never replace the battery with the machine powered off like that! You could corrupt the hard drive or worse, mess up the DNS settings!"

"Wait... what?"

"Did I stutter?"

"Nooo, you didn't stutter but what you said was so mind numbingly stupid I'm still trying to process it. You want me to replace the CMOS with... the machine... running?"

"Did you just call me stupid?"

"I'm sorry. I thought you already knew. No, I am not replacing the battery with the machine running."

Granted, this was the same mental giant that decided that an effective way to reduce service tickets was to give all users local admin access on all machine. In a production environment. Where the public could just walk up and start doing whatever they want because the systems weren't locked down.

And for those that want to question it, yes, I will talk to my management like that. Always have. Still do.

Call from an unknown number, but it's local, so I answer.

"Ol-gormsby computers"

Aged voice mumbles "This is ahhhhhh Tom. I've bought a computer and I need it put together and set up to work. How much do you charge?"

I tell him my hourly rate and ask "Where did you get this computer? Didn't they assemble it for you?"

"Ahhhhhhh I bought it off the internet"

"I see, did you buy an operating system, a copy of windows?"

Silence, then "No, I don't think so"

"Well, OK, I can take care of that. Do you have a keyboard, mouse, and monitor?"

"I've got a keyboard and a mouse"

"Do you have a monitor, a screen?"

"I thought we could use the one from the old computer"

"Well, possibly. Do you have all the cables?"

"Uhhhh, yes I've got cables."

"OK, just to make sure, you've bought a computer, all the components, but it's not put together, you need me to do that, and install an operating system, and copy your files from the old computer?"

"Yes, how long would that take?"

I'm not going to short myself, so I give him a long estimate. Better to do that in case we run into incompatible components, obscure unsigned drivers, etc. So he says OK, and we make an appointment for me to visit and make it all happen.

Not one hour later, he calls back and tells he won't need me to do it because his granddaughter can take care of it for him. My reply was a joyful "Good for you! Call me if there's anything else I can help with."

I sure hope the granddaughter can do it, because if he calls me to fix anything, there will be no pensioner discount this time.

Edit: Part 2 in comments below.

I worked in a state government body that was "attached" to the State education department, and within our small organization was a business unit responsible for the standardized testing of high school students. The test was a closely guarded secret, to the point where the business unit office was separated by a swipe-card access door. On each desk, they had two computers, without even a keyboard/monitor switch box. One computer was connected to the great unwashed (the regular network), and the other was on their own physically-separated air-gap network. No connection to the outside world, because, you know, security.

If these people wanted to get something off the internet onto their secret squirrel computer, they had to burn it to CD-ROM (yes, I'm that old) and then put the CD into the other computer. Before I left there, USB drives were just becoming useful, so they started using those.

Obviously, this doubled the cost of refreshing desktops, so a Study was commissioned to investigate a Truly Secure connection to the outside world. We settled on a system that we were told was the firewall of choice for the Department of Defense.

Armed with our Truly Secure solution, IT Manager approached the Director and presented the solution, which would save this many thousands over the next [n] years. The Director asked The Question: "So this is 100% guaranteed secure and un-hackable?" IT Manager's eyes glaze over as he ponders the many ways he could answer that question, and replies with "Well, I couldn't say that any system is guaranteed to be un-hackable, but this system is used by our armed forces to protect our national secrets, so I'm very confident in it."

Director: "So you're saying there's a risk that our standardized test could be hacked and we would lose thousands of hours of work and risk the integrity of the State's standardized testing for that year?"

IT Manager: "Well .... yes, there is a very minute chance that this system could be hacked."

Director: "Well, we can't take that risk. We'll keep going the way we've been doing it all along."

IT Manager: 😐

After we left that meeting, I asked the IT Manager, "Should we tell him about the multifunction printer that is connected to both networks and technically could be hacked via the dual NICs and is exponentially more unsecure than the Department of Defense solution?"

"No, PFY, we shall not tell him about that."

Simple story, but memories worth telling.

A long time ago, I was an assistant at an office, employed primary to change the printer paper and separate carbon copies. (Large print jobs there.) But being a computer nerd, I soon was helping with all kind of computer based tasks and problems. One day, a desktop computer didn't start Windows (then version 3.1 ~ oh the olde days...) - just a blank dark screen. As always, the user "didn't do or change anything". Other employees already tried this and that, but no error could be found. I investigated the usual stuff, the more unusual causes - hardware ok, all files ok, settings ok ~ so why? Then, during a test run, somebody interrupted me (delivered mail - paper type! or something like that). The computer was untouched for some minutes - and suddenly, Windows came up. ??? Did I change and/or repair the problem? After some more checking: The user had changed the previous background image to a really large true-color foto, and the computer had to calculate it down to the screen resolution and to 256 colors, which took several minutes - and nobody granted so much time to the poor machine. Changed background, problem fixed ;-)

Towards the end of my career, I worked for some managers who were control aficionados. We always had more stringent change windows than the rest of IT for even the most minor of changes, and there was always fear that touching anything would be a problem.

We generally supported a variety of vended software, plus design and coding around those packages. During rollout of one of these packages, we were a bit behind, so they suggested granting a whole bunch of cross-environment DB permissions that, once we went live, would be huge red flags to any audit. I was the person with the most DB experience on the team, and explained why we shouldn't take this angle, or at the very least, needed to clean them up before the go live date. I was overruled.

About a week before go live I went through a change to eliminate the ugly DB permissions to meet standards. If nothing else, doing so before go live would allow us to make the change at a normal time, instead of zero dark thirty on Sunday morning. Managers were nervous, because all changes are to be feared.

Eventually they secretly went to trusted employee (TE) next to me, whose work they respected more. TE was very sharp but had less database background. They asked him "are these changes that Dokter Z proposed safe?" He agreed to check on them.

The next time that all the managers were off in a meeting, he just stood up and asked me over the cubicle wall "dude, are these DB changes correct?" I said, "why yes, they are".

"Sounds good!" Later he went into their office and assured them that all would be well.

Far from the stupidest thing that occurred during my tenure in the area.

Production's ramping down for the year and the plant manager asked me to find a way to get music playing on the shop floor. I've not nothing better to do at the moment so I said I'd take a look.

It turns out, all I need is a component audio (RCA) cable that I can plug into the amp. The ONE cable I don't keep in my bag of tricks. After digging through an empty office, I found the cable. Unfortunately, it's got a 3.5mm audio jack on the end and none of our gadgets have those anymore. Dig through my bag of tricks again and find the adapter Apple included right after they ditched the audio jack years ago. That'll do the trick just fine.

Plug in my phone to the amp and hit play on one of my play lists. Adjust the audio so I can hear it and begin walking the production floor. IMMEDIATE complaints. Apparently, I'm the only one that wants to listen to Pantera while I count widgets.

Head back to the audio closet to change the tunes to something more depressing, like holiday shit, and the production manager stopped me. Music on the floor is no longer wanted. Oh well. I've got my headphones.

Oh... it's been forever since I've waded into these hallowed halls of coffee addicts trying to convince people that yes, a computer must be plugged in to work. Sorry it's been so long, but with my current job, I don't often have stories that would interest folks here.

I mean, sure I have stories. I still work in a form of technical support, but not like I did back in 2017 when I was one of many folks in a cube farm out in Salt Lake City. I know some folks there read this, so hi guys.

I currently work in game development, in a curious little tech support adjacent position that's generally called "bug tracking." It's not unlike your usual tech support situation, though there are times I get to be a bit more snarky with the explanations than a call center tech agent might otherwise be allowed to be. I mean, I can call someone out for complaining that their pirated version of the game isn't working, and not face any real consequences for pointing out their behavior.

This isn't that kind of story.

So, recently, I picked up a ticket for a rather curious bug, and set about doing the work tracking down just what on earth was causing the problem.

The bug report said, and I quote "Game crashes when I plug in my headphones".

Yeah, I read that, sat back and thought to myself "really?" I mean, how many times have you heard of a computer going "nope nope nope" and BSOD because someone plugged in some headphones? I sure as heck haven't.

So the following conversation began with the user:

Tech: So, if you plug in the headphones to play the game, the computer... crashes?

User: Your game sucks... (insert random typical rant here)... fix this now!

Tech: I can't fix it if you don't tell me what happens.

User: I told you. It blue screens.

At this point, I realize we're dealing with an entirely different beastie here. See, I was thinking that the user was getting one of those ever present unreal errors... but nope, he's getting the Blue Screen of not happy computer exprience.

Tech: That's different. That's more than just a crash. Does it tell you any kind of error message?

User: I dunno... what does that matter?

Tech: (with a bit of my natural snark) Oh, I dunno, maybe because it could, you know, help me figure out what's going on?

User: It's your game, I know it. (Followed by another long rant). Fix it... fix it now.

This kind of situation, me probing for answers, and getting complained at, went on for another twenty minutes, before i convinced the user that sure, I'd fix it. Though in the mean time, if they could send me the crash dump file for me to take a look at, it'd help. It took ages, but I finally received the file and started poking through it, trying to figure out what crashed.

IntcUSB.sys.

Huh... that's an audio driver file. Well, not specifically, but it typically pertains to audio files. Looking closer, I see that the user had two versions of it running at the same time, and the computer was effectively doing the same checks... twice... and confusing itself.

So, I head back to talk to the user.

Tech: Okay, so... it only crashes when you plug in your headphones?

User: I said that already!

Tech: I know... I know, but what kind of headphones... are they?

User: "Popular bluetooth headphone brand"

Tech: And when you plug them in to charge, it crash...

User: No, when I plug them in to use.

At this point I had to stop and re-read that a couple times. Plug them in to use? That brand of headphones, they're wireless. You only really need to plug them in to charge. I mean, when they're "dead" you can use them wired... but...

Wait a minute here.

Tech: Are they turned on when you're plugging them in?

User: Of course they're turned on... they won't work otherwise.

Face... meet desk. Now it makes sense. They're bluetooth, and the user likely has a dodgy cable to connect them. The cable they come with is maybe six inches long. I doubt the user has their head right against the PC... so it means they're getting a conflict. The connection was just enough that the bluetooth would kick in, connect, and start to play audio... only to have the wire make a momentary connection, and go "no... mine!" to the audio driver, putting the computer into a kind of schizophrenic loop arguing with itself about who had control of the driver. It was enough to cause the computer to throw up its hands and go "Screw it, I'm restarting, you figure it out!"

I actually started laughing at this one, as it was probably the dumbest error type I've ever seen, and yeah, I could see how it would be frustrating. Lord knows, if I started seeing random blue screens after plugging in some headphones, I'd wonder what was going on.

There is something of a happy ending to the story here. It took some time to walk the user through what was causing the issue, and explain to them that the headphones didn't need to be connected by wire to the computer to work. Once I'd gotten the user to understand that they work perfectly well without the wire, the "game" stopped crashing.

We have a running bet currently going about how long it'll take for the user to send in another bug tracker report, claiming that his game won't play audio through his headphones... after his battery finally dies on them.

When it rains it pours.

Now, excuse me while I wade back into the fray, trying to explain to people that yes, physics is a thing, we modeled it like that for a reason, and no, screaming "fix it now" isn't going to make the code work any better. (No matter what the code monkeys would like you to believe.)

Some time ago i worked in a government agency as an external employee, together with several other external sysadmins.

A lot of weird things happened during that time, but this one made me shake my head.

My colleague (I'll call him A) and I were in charge of equipment rental (notebooks / laptops, projectors, printers, etc.), so other employees could borrow this equipment for lecturing, workshops and other stuff.

To keep track of the equipment, we had a printed list and each borrower had to sign for every piece of equipment they took. Looks simple, doesn't it?

One day we did an inventory and realized that one laptop was missing. We went through the lists, asked the others colleagues, but the laptopwas nowhere to be found. And to make things worse the signature for the notebook was missing. Boss was understandably a little pissed.

However, the laptopwas actively logged into the network and we were able to ping it. Normally this is not a problem as you can use the IP address to find out in which building and room it is assigned.

The problem: the network segment was unfortunately too large, covered several buildings and it was impossible to find out exactly where the laptop was located.

A few months went by until another colleague called me and asked if we were missing a laptop. Hell yeah, we are missing one. Where is it?

Well, the solution was just stupid: there was a DNS outage in a certain building. Another colleague from the other sysadmin team took said laptop without signing - and used it as a replacement DNS server.

The laptop was in the server room in the building, sitting on a table and running happily.

There was also a post-it note saying “Attention, this is a DNS server, do not turn it off”

We also found out which colleague was to blame, but that sucker denied it and blamed it on other colleagues.

Outcome: no consequences for anybody, but after that we watched like a hawk over the rental equipment until our contracts expired.

Me: Hello, Mam How can I help?
Lady: My computer has turned evil, i need help!

Me: Wow, ok, what happened?
Lady: Whenever I try to open the app, it says "Demon failed to start". Why is the Demon trying to start in my computer?

Me: Oh no! Mam , is that spelled "Daemon" ?
Lady: let me take a look, yes!

Me: Oh mam, that's not a demon, it's a background process that runs in your computer. we commonly call it Daemon, think its short for Disk And Execution MONitoring.
Nothing to be worried of! Just needs a fresh installation and restart.

Lady: For holy sake, why they named it like that? Could't they do, DAEM or something, they had to pick the 16th century version of Demon.

Here we go again. Another last minute roll out of services to a client who has been promised the world in a gold-plated basket by my colleague known as GH.

I got the call, "the new client is a go, it's going to be a slog, 60 computers, we need to run round to each computer and get our software on them and rip out all the previous company's stuff"

Those words from GH were enough to give me a migraine the size of Neptune. It's like the scene from The Matrix when It's revealed the machines keep the humans in a simulation of 1999 at humanity's peak. His knowledge seemed to stop then too.

"we need to be there from 7:00am so we catch them as they come in, we need to get them onto the new wifi and get our software on there, get them configured for the new VPN, swap the antivirus client, patch them and deploy all our other software, if we grab two or three USB sticks we can work on multiple computers at the same time and be finished by 7pm as they are a nightmare to get off their machines..."

oh, dear god, why? I create all the automations for my company and its clients. I've been doing this for years, he knows i do this, and i can automate 99% of all the functions we require in minutes.

my brain went into defensive mode, it heard the words, but it bypassed them from actually being processed and dumped them in to the "bla bla bla waffle waffle" bin. To defend my sanity, i logged into their intune portal.

oh look, 60 computers checked in and all communicating without issue.

GH -"podgerama, are you listening?"

Me - "hold please caller"

GH - "what?!?"

Me - "give me a minute, i'm almost done"

GH - "done with what, you need to call *other client* and tell them you need to postpone your visit, this is urgent"

Me -"err no"

GH "did you not hear what we need to do, you are cancelling that visit and coming with me tomorrow!"

me - "how about no? I have just repackaged our agent installer as an intunewinapp and applied it to a new security group in entra"

GH - "but how are we going to get intune rolled out to them???" (he thinks its a piece of software thats needs installing)

It took 30 minutes to show him and break it down in a way he can understand

me "so, forget about your usb sticks and running round. i deployed the new wifi details in intune last week, At midnight when we are officially in contract, we make the computers members of this security group, they will get our RMM agent deployed, and the procedures i have created for the client will push the new printer queues, install our antivirus, deploy the new VPN client and its config. no runny roundy, no usb sticks, no local admin passwords, it's all automated now"

GH is on site now, he has his usb stick in hand. he's struggling to comprehend how they are joinign the wifi without his help (or thinking he has done a great job) and that the 7 machines he has visited already have everything we require done to them. I don't have the heart to tell him that the 50 machines that have come online so far have all succeeded.

This really happened, in 2018. I wrote it up as a Fb post, but figured you guys might get a kick out of it. Background: I was the sole sysadmin for a national non-profit from 1996 - 2010.

"Scott?"

Every husband knows the tone of that voice. A question couched in a demand.

"Can you come down here and help me?"

My wife has a silly little webcam that has captured foster cats being idiots more times than I can count. But it stopped working six months ago.

Her voice echoed up the stairwell: "You said the cam might work now."

While I was trying to figure out why our new printer wouldn't connect to our wireless network on Friday, I discovered that half of that network had crashed. We didn't notice because none of our other devices used it.

But the printer did, and so did the cam.

I closed my laptop and trudged downstairs with it. "What have you tried?"

"Everything!"

I raised an eyebrow.

"Stop being such a jerk. I really did!"

My wife Ellen has been married to a sysadmin for not quite twenty years, so I didn't doubt it. But I also knew how this all works.

Me: "So how do I connect it to my laptop?"

"What? You don't. It's not even connected to MY laptop. It's connected to my phone."

I looked upstairs and shouted to my (then 15 year-old) daughter. "Olivia!" ... "Olivia!"

"WHAT?!?"

"Bring me my phone."

Ellen started to tap on hers.

Me: "Stop."

She tapped more.

"Really, stop."

Olivia, after stomping downstairs: "Here!" She stomped back up, putting her headphones on so she couldn't hear us anymore.

Me: "Okay. So how do I connect my phone to the camera?"

Ellen: "I've been going through the list AGAIN. Here," a link shows up in my PM feed. "That's the... oh my God."

"What?"

Her voice dropped to a whisper. "It's working." She looked at me. "How did you do that?"

I told the truth. "I didn't do anything. I told you to go through the instruct--"

She waved me silent. "I have, several times. It never worked before. And now it does."

I shrugged. "Welcome to the world of a sysadmin."

"But... it just started... and all you did was walk downstairs..."

"No, it was you being methodical."

She stiffened. "I am ALWAYS methodical."

I knew better than to contradict someone who is always right. "Of course you are. That's why it worked."

After a moment, her eyes got wide. "Wait. You've talked about this. How it just starts to work."

All I could do was look at her.

"You don't know what fixed it?"

"Not at all."

She went pale. "You've built a career on this."

Now it was getting embarrassing. "And that's why I'm trying to become an author."

Olivia, from upstairs: "God, Dad. Does she need help with her iPad again?"

My company recently upgraded from Windows 10 to 11 and one of the biggest issues are some of the files on the network drive went missing. They are easy enough to restore, but they involve signing into the computer as an admin and disable offline files. I just had a call a early today that I wanted to share.

Me: Thank you for calling the IT help desk this is 'MY name', How may I assist you?
Customer: Yes, I recently upgraded to, You know what, it doesn't matter what happened. My files are missing, I need you to restore them.
Me: "Do you mean the windows update, If so this has been a problem with the upgrade itself. Do you mind If I sign into your computer, there is something I need to run first."
Customer: "What do you need to fix my computer. Are you saying I need to call IT every time I have this issue?"
Me: "Ma'am I will need to enter my admin password to fix this issue, If issue does occur afterwards then we can send this over to another department for a more permanent solution. "
Customer "So hat you're saying is that you're not going to be able to fix my issue"
Me: "No ma'am that's not what I am saying at all, yes you will need to call the IT help desk if this issue does occur, since only a system admin can fix. Now do you mind if I sign into your computer."
Customer "Fine, but I want a guarantee this issue will never occur, again."
Me "Ma'am I can't do that. There is never a guarantee that the issue won't reoccur"
Customer "Fine sign in, but I want it escalated regardless if you fix it or not. I'm a very busy woman, and I can't call the IT help desk for every issue. "
Me "OK I'll escalate, Now if you could give me the computer number and save and close any confidential documents that might be open, I should be able to assist you. "
Customer Shouting " What do you mean close my documents, you;re not goign to to delete anything are you?"
Me"No ma'am, I just need to run some processes on the computer and I don't want to sign in to a file that you don't want me to see."
Customer" I don't have any files open, and If I did I wouldn't want you to see them"
Me "OK that's what I asking for."
After that I sign into the computer, The customer is mostly silent, but under her breath I hear her muttering how useless IT is. I was able to fix part of her issue, but and sent it over.

Small office customer. Sold a new, bezel-less 24" LCD monitor to said customer. Customer takes it on-site and hooks it up, all works well. It sits on a counter by a large window.

Customer calls me a few days later. Says the new monitor is dead. The power light is on, but no signal. I have him try a different cable, different port on the PC, etc., but nothing. So we process a replacement, swap it out, and chalk it up to being DOA.

Customer calls me again a few days later still. Says you're not going to believe this, but this second new monitor is doing the same thing as the first. At this point I'm thinking their office must have a power problem or something that's killing monitors. But I decide I'll take a truck roll and see for myself.

I go on-site and bring a third new monitor with me just in case. I open the door and see a very pretty cat walking on the floor. I look at the old, original monitor which was replaced by the new 24". It's an old 17" LCD from a decade ago and had thick, beefy bezels as monitors from that era did. I see some bite marks at the top corners, but they're just on the bezel so no actual screen damage.

It's beginning to add up. Their office cat had been chewing the corners of the old thick-bezel'ed monitor. Which was fine, until they got a new monitor that had no bezels at all, and all it took was one bite from the cat to pierce the LCD itself. Twice. Once they were made aware, it was easy to see the teeth mark in the corner of the LCD of the new monitor.

Customer ended up getting a used 22" monitor with thick bezels. Cat still chews the corners.

EDIT: I found the pics!

https://ibb.co/b3s84DJ

https://ibb.co/Vx2H1sX

https://ibb.co/8rVKd56

This is from a few years ago. I was working at a medium sized company as an IT sys admin. The company had just recently moved to a new location that was able to more comfortable accommodate its operation. It had an on-site call center as well as a medium-scale manufacturing/repair center. Since we were new tenants and everyone was now under one roof, many things were still being figured out.

One day, we got notice of a gas leak in the manufacturing area. We didn't have an alarm system for a gas leak so people were running around telling everyone else that there was a mandatory evacuation of the building. The IT people all had laptops so we all grabbed them and made our way to our cars. By coincidence our director of IT and the head of IT support were on a business trip. As I'm walking out the door the Call Center Director (I'll call him Cal) start yelling at me and the other Sys Admin. "Hey, what are you guys going to do!?"

"Go to our cars."

"No, no you can't. We can't receive calls. You have to do something!"

I turned to my coworker and we both realized that the call center still used desktop computers and soft phones. They couldn't do their job. Cal was red in the face trying to slowly let people out the door to the outside. It was then that the fire department arrived probably to clear out the building officially. So I asked Cal, "What's your plan if there's a fire? Just do that."

"What? No, you need to do something."

I shrugged. "We can't do anything. The phone system probably doesn't work off of VPN." I was guessing at that. "Just follow your plan if there's a fire."

"You guys never gave us a plan for a fire." Cal responded.

Because of course it's IT's job to develop a business continuity plan for the entire company. More people were streaming out. It was then I decided to ignore him and go to my car. I tried to call the Director of IT in the slim chance the airplane diverted or was delayed. No answer. I looked up in the company SharePoint site for a business continuity plan or fire plan or something. But only found stuff for IT, including our offsite backup servers and how to run IT operations from VPN. There was nothing about moving our softphones to/through VPN.

Cal knocked on my car window after everyone was out of the building. "Well?!?"

I explained that there was no business continuity plan in the SharePoint site and IT didn't have anything in place to shift the softphones to VPN. Plus we didn't have enough laptops to support even half the call center. Cal didn't like my answer and walked over to the CEO who was the fire department. I could see Cal pointing at me and yelling. Clearly we were losing business. And clearly it wasn't just IT's fault, it was mine and mine alone.

The fire department cleared us to go back in after about 45 minutes. Later that day I had two meetings with Cal and the COO scheduled. Since IT was missing both leadership positions to travel I was the scapegoat. The first meeting was cancelled and the second the CEO stepped in and cancelled it since it was really the job of the Director of IT and a lowly sys admin shouldn't be in these meetings.

Nothing bad happened to me when the IT Director returned. And the company hired a consultant to develop an actual business continuity plan for fires, weather and other events. Turned out, IT shouldn't have a button they could press in the event of a gas leak. For several months Cal scowled at me after that every time we passed in the hall.

TL;DR Call Center Director assumes that because his department uses computers, any problem becomes an IT problem.

Hey! I've been lurking through this subreddit for more than a decade at this point; I have now become a telecom engineer, and I have some stories to give back to this wonderful place: this is the story of when our nameserver was just a dude.

I had just started working a volunteer position for a local NGO, I was already studying engineering and had been working with these guys for a while, and as the resident young guy that works with computers in a place filled with old people, I just slowly drifted into an IT of sorts; after getting Office running on a couple of laptops and fixing and documenting their heinous email situation, I got some one-on-one time with our librarian:

$librarian: Hey u/benjazio_xd, can you help me with something here?

There was a reference collection of books for internal use, around 30k books in total, managed by this one guy who also cared for the NGOs extensive paper archives, which were around a hundred years old. He's a cool guy who actually turned into a great sidekick for many projects I did while working there, and we remain friends even after I left.

$librarian: You see, we've had this ILS for a while, and I've been told it has an open access catalog so our guys can see what we have and come pick it up, but I've never gotten it to work right, could you take a look at it?

An Integrated Library System (ILS) is a piece of software that tracks pretty much everything inside a modern library: inventory, loans, labeling, shelving, late fees, you name it. They are very niche software but also extremely powerful: they are the beating heart of many libraries, big and small. This one was hosted on a local server in the office itself, and a quick browser check to the local IP address of the server revealed that it did, in fact, have an open access catalog.

$librarian: It's supposed to be on our webpage, but I've got no idea how it works and no one really explained it to me when I got here.

Sure enough, there was a link on our webpage that just returned a blank page every time, and not only that, it seemed to be an internal URL on our webserver, which didn't really make sense considering it was on a different machine halfway around the world.

Nothing in the world would have prepared me for what I saw when I clicked on "Inspect".

$me: So, um, has anyone ever told you anything about this before?

$librarian: the previous girl that had my job told me that the page had to be updated every couple of weeks, and left me a couple of links I had to follow, but she never taught me how to do it and that was like five years ago.

Jesus Christ, this hadn't been working in a long time.

In this blank page was actually an iframe, which pointed to the frontend on our public IP address. This was janky and unnecessary, but what turned it into depravity was one key little detail: we had a dynamic public IP address.

This meant that for years, someone had to connect via FTP to the site every couple of weeks, go to this page, and modify the iframe so the IP address matched to the current one we had. There were no notifications set up either, which meant someone had to notice and tell the librarian that this was going on in order for this to even work, and when they changed librarians no one bothered to write this down, and so that site was just permanently broken: Our dynamic DNS solution was just having a dude update a file on a remote server whenever they noticed the god damn page was down.

$me: This is extremely stupid, how did nobody notice this earlier?

$librarian: you're the first guy here who actually knows enough to care.

My heart sank a little. Apparently this guy had been complaining about this for years but because the dev team for that website was long gone no one had bothered to get someone to look at it. It was an unfortunately common scene in this place, and it was the reason that made me leave it some years later.

$me: Right, this is going to take me about an hour to get everything set up, but I'll get it fixed and running before the day is done.

My solution was just to get a DDNS provider and hook it up to a subdomain of our main site using a CNAME record and just changing the link to the page to the new address. This was fairly low traffic website and just have the server directly respond to requests was fine. I used a small script on the local server running every five minutes to update the IP address to the DDNS provider and that was it: it now just worked on its own.

$librarian: You have no idea how much rage you've removed from my system, let me buy you lunch tomorrow.

We got a static IP a few months later, and I made a friend in the process.

I work in tech support for a fairly large company, and I’ve had my fair share of bizarre calls. But this one really stuck with me.

A customer calls in, and the first thing I notice is that they’re clearly frustrated. I ask for details, and they explain that their computer is “just frozen” and nothing is working.

I tell them, as calmly as possible, “No worries, let’s start by rebooting the computer. Please hold the power button for 10 seconds to turn it off, and then turn it back on.”

There’s a pause on the line, then: Customer: “I don’t know how to do that.” Me: “You don’t know how to turn off your computer?” Customer: “No, I don’t know where the power button is.”

I’m trying to stay professional at this point, so I walk them through it. I even ask them if they can find the power button on the actual device. They respond that they don’t see one.

So, I ask, “Can you look on the side or the back of the computer for a button or a logo?” Customer: “It doesn’t have one.”

At this point, I’m a little confused, but I decide to walk them through the process anyway. I start asking if they see any lights on the device. They tell me no, nothing is lighting up.

Then it hits me. I ask, “Are you sure you're working with a computer?” Customer: “Well… no, I’m looking at my microwave.”

This person had been trying to reboot a microwave for 30 minutes, thinking it was their computer. After a long, awkward silence, I confirmed that microwaves don’t have the same functionality as computers, and recommended they try restarting their actual computer instead. They were extremely apologetic, and I just couldn’t stop laughing after I hung up.

Never a dull moment in tech support, folks. Stay strong out there!

**For context - this is a project I was called in to as an emergency resource. Basically, come fix our mistakes...anyways...

Oh yeah. Just found this one recently. One transaction runs 170 lines of code to verifying the creation of a single record across 3-4 classes.

The total line count of those classes and their corresponding test class is just over 8,300 lines of code. One of the classes on its own is over 2400. I saw a method that had over 300 lines just on its own. What a trooper.

The first kicker (maybe not, there's a lot of great stuff in here....8 layer if else statements....)

Okay, one of my favorites is that the big mama class is a global class and is integrated with an external system. This system, as evident by dozens of unnecessary permission checks, can have multiple users hitting this class at the same time.

That's a real problem because -every- -single- -method- is static and **THE SOURCE RECORDS ARE STORED AS GLOBAL STATIC VARIABLES**

For those who may be unaware or just unfamiliar, "static" means that there cannot be multiple instances of that variable even if multiple "things" are referencing that method/variable. In other words, if you have two requests come in at the same time - you could very easily end up swapping the source records mid process and corrupting the data on one or both sets of data. And you would never know because, while corrupted, all the data is likely still in a valid format and will pass through the system without a system error. Kind of a big deal when it's directly associated to every deal in your sales process.

But the thing that really prompted me to post this was such a level of "I don't care about the next guy" that I am actually stunned. I think I may be in denial still...

There is a class, and he's amazing. We know the rules - don't hard code values that are susceptible to change. If at all possible (and reasonable) don't hard code a constant that may change in the future.

This class, I shit you not, is just shy of 500 lines of grade-a, organic, non-GMO constants baby. They're global, they're hard coded, and they're susceptible to change.

But, that wasn't enough. It's the gift that keeps on giving. After spending the better part of 12 hrs wishing I was dragging my head across the pavement at high speeds, I noticed the comment at the top of the code (slightly paraphrased).

"Class to hold the static values...to avoid hard coding."

Sometimes, man, I really do wonder. I feel like I have done a pretty good job remaining positive about this absolute mess of a transaction - but why they gotta spit in my face like that? It's hard coded, for 500 consecutive lines, right below that message. That's EXCLUSIVELY what this class is.

C'mon man. Just....DAMN!

**Final bit of context, because idk what everyone's familiar with, this platform has built in functionality for getting the exact values at the time of running. There isn't a single thing in that constants class that needs to be declared as a hardset constant in the code. And there are no checks to verify that the transactionId remains consistent.

Years ago, I did programming and support for a system that had a lot of interconnected data. Users were constantly fat-fingering changes, so we put in auditing routines for key tables.

User: it (the software) changed this data from XXX to YYY…the reports are all wrong now! Me: (Looking at audit tables) actually, YOU changed that data from XXX to YYY, on THIS screen, on YOUR desktop PC, using YOUR userID, yesterday at 10:14am, then you ran the report yourself at 10:22am. See…here’s the audit trail…. And just so we’re clear, the software doesn’t change the data. YOU change the data, and MY software tracks your changes.

Those audit routines saved us a lot of grief, like the time a senior analyst in the user group deleted and updated thousands of rows of account data, at the same time his manager was telling everyone to run their monthly reports. We tracked back to prove our software did exactly what it was supposed to do, whether there was data there or not. And the reports the analysts were supposed to pull, to check their work? Not one of them ran the reports…oh, yeah, we tracked that, too!

So it's Halloween and as usual everyone is in the office (building B) for our Halloween party. On top of that it's time for the regular Tabletop/IRP review. So everyone in the department from desktop support to security, Property management, the CIO and CISO are all crammed into the hot conference room. Luckily I'm not as I asked to work from Building A that day

Anyway, we started going through our IRP scenarios. With some members strategically barred from answering. Our new security analyst pipes up and says we have spiders attacking our network. Confusion follows, is this a scenario? Then slight panic, this isn't a scenario. CISO ask where are you seeing this. Analyst says, we just got a critical ticket. Someone opens the ticket and reads web spider attacking printer in building A This makes no sense so network, security and webdev start checking their various metrics and logs. I'm in building A so I go to check the printers.I find spiders all right, tiny plastic spiders all over the flatbed in the exec suite.

TLDR: someone messed with the printers, a game of telephone leads to a VIP (automatically critical) ticket saying our printers are under attack. Turns out someone just covered the printers in plastic spiders.

A bonus story for today but going to be vague. won't get in trouble if im not but still probably a good thing if im vague.

So after some of the terrorist attacks in the early 2000, my company contracted a big aerospace company to build a system to automatically detect bio weapons. Said system is very expensive and requires a lot of maintenance and has multiple people monitoring it remotely.

One day my dispatcher received a call from a remote monitoring site saying that we need to check one of our machines because it's retesting a sample. My poor dispatcher interpreted this that it has detected something and being the only one on my shift trained for that system they called my cellphone directly. Dispatch doesn't know our cells because we have radios. So they got management to call me because no one wanted to talk about this over the radio. Was given direct orders to tell no one and go immediately to the machine. I arrive at the machine with everything running fine with no fault lights. So I logged in. The issue, a bad test tube. This machine has multiple so if it fails to get a "false" but also fails to get "positive" it will retest with a new tub. Nothing major, it just runs another test, the test do take awhile though so i got to sit and watch a screen. Did call dispatch back though and told them we are not under attack so they can calm down.was very over dramatic.

Time for a couple more badly written stories, words are hard and I never went to college. You get what you get.

Get some doordash, maybe some Adderall or whatever your vise is and enjoy

My job is tech support related but not directly. I work on anything from servers, networking, to automation(belts, motors, bearing) and PLCs. I'm a jack of all trades and definitely a master of none.

:The disappearing fault

So one day Operations calls us due to an output module fault. It looks like 7 modules lost communication. Well we first check the com cables, 2 40pin cables that create a loop for 4-19 modules. They seemed fine but Admittedly avoid these cables because I hate them, bulky, bad retention mechanism, and likely to have more problems just from touching them. All the cables for controlling coms, gates, actuators, and safety loop go to a backplane that slots into the main control PCB. So we replaced the main PCB, nothing happens except for even more faults. Then we got a second one, kinda worked just different faults this time. So we got a third one, most faults gone except one but communication is back for everything. At this point I called a remote SME, system matter expert. Who says to swap the board with another module to see if the fault follows the main board or stays with the module. One problem, it does nether, it just disappeared. Doesn't make sense to me but it's gone and the machine works.

Main lesson learned, just expect all your parts are bads.

:When the SME is wrong

So Operations calls about a machine intermittently stopping for a safety loop fault that never calls out where the fault is in the machine. The machine will act good when not processing but after 30s to 5m it will fault out. We arrive and start looking at interlocks but couldn't find anything. We keep pressing start till we get the fault to show up and not immediately disappear. We checked a 24v safety aux contact attached to a relay, even though it didn't test bad it's so common we replaced it. After checking every interlock we can't figure it out. So we call the remote SME. One piece of info I did have to diagnose the issue is the aux contact has no power. Letting the SME know this he first says to replace it, I told him I already did, also told him I'm not trained on the equipment though so I'm at a lost and need schmatics. He emailed me the schematics and also wanted me to follow the schematics after the aux contact which didn't make sense to me because it wasn't getting power. I felt I should trace where the power comes from and see where I get it back. So I lied to him that I would then with the self confidence of a stupid person, I did my own thing. Found power going into the safety loop at the breakers, the breakers have the ability to tell the computer if they're tripped, but not coming out. Started shaking the connectors on the back of each one until I found one that would make the machine go ready and not ready just from wiggling it, yes I wore gloves if anyone from OSHA is asking. After the machine was down for 5 hours, one aux contact and one breaker later it was fixed.

To explain what was happening, the machine vibrates when running due to motors, bearing and belts. This vibration would cause the tab inside the breaker to disconnect momentarily causing the machine to stop due to the safety loop opening.

:when the senior tech doesn't compute

So on day shift they had a machine go down due to the output modules not communicating. The senior tech(for day shift) found the module not communicating and replaced the board but still wasn't able to fix it. Even shift with the 2 most senior techs out of everyone refused to touch it. Finally when i came in on night shift and was questioning why we had a machine down i decided i would look at. Still not trained but from knowledege of last time I asked if anyone configured the board. All evidence pointed to no. So one call to the SME for the document on dip swich configuration and crawling inside the machine later the machine worked...

Unsure if anyone else shares my frustration but fixing stuff more experienced and trained people shouldv'e give mangement unreasonable expectations of your ablities. I love solving problem, i don't love being put on a pedistol.

Btw the down time of that machine probably cost $150k-300k

:how to solve a random persons problems from 500 miles away.

So the techs at my company have a facebook group for memes but also for help when SMEs are no help.

A person in another state posted they have had a machine down for over 7 days. The machine would only fault out if you tried to run it. With the fault being a communication fault from the operator PC to the on site OCR server, Optical Character Recognition. The issue was they could ping the server, and PC and server would show connected in their respective software. They even ran a new cable from the switch to no avail. I guess no one on site or the SME thought to actually see what the switch was reporting. I had access to see the monitoring of every facility just not make switch configurations. I was bored and looked them up and saw a ton of errors. The port was configured correctly, so most likely bad port.

So I messaged the guy. We got me =me, tech= guy from that facility, and supervisor = his boss

Me- hey, i saw your FB post i think it's the switch port

Tech- we are going to reboot again

Tech- I'm going to make a group chat with my supervisor

---new chat---

Me- hi, i think this is an issue with the IDF switch, do you guys have anyone with cisco CLI training and log in.

Supervisor- i think so but he hasn't logged in awhile

Supervisor- SME says to check switch at machine, we replaced it but that didn't work. SME now says to replace IDF switch

Me- before stopping all operations lets just try another port

Tech- we need a ladder

Me- i see the switch lost power recently, did you guys have a power outage.

Supervisor- actually yes, thats when the problems started

Me- please take the cable from port 6 and plug it into port 8

---Note, port 6 is for the machine having problems, port 8 is for a machine that is working

---23 messages and 4 hours later of being ignored

Me- please take the cable from port 6 and plug it into port 8

Supervisor- that worked we think port 6 is bad

Me- plug the cable ftom port 8 into port 6 and see if it faults out too

Supervisor- it does

Me- that comfirms 6 is bad, have your tech open and cofigure another port and label port 6 as bad.

Supervisor- thank you!

Moral of the story sometimes you need to repeat yourself i guess. Still working on being assertive.

On the plus side this interaction helped me pass the interview to become a SME, just waiting for an open postion.

:the normal tech support call

So us machine techs are only supposed to fix anything related to machinery and their functions "processing infrastructure side". We consider anything not related "Lan side"(printers and supervisor computers).

One problem, one onsite "Lan side" tech covered like 6 plants almost all 120 miles apart. They could drive 5 hours for one call and responce time is like 2 weeks.

Due to how over streched this guy was, even though he didn't want my help, and my interest in tech I would help when i can. It was against the union contract but keeping the bosses happy was in everyones interest. I mainly would just help with printer problems and was well known by management for solving printer problems. After the print server/directory failed i was the only one to get the printers working while we waited for it to get fixed. Anyways here's the story.

-Over the dispatch radio

Supervisor- hey OP can you help me with the printer by machine 9

Me- On my way

---i arrive stage left

Supervisor- I can't get the printer to print, i think it's broken

Me- please bring up what you're trying to print

Me- press print or ctrl p please

Me- can you select the printer labeled "printer by machine 9" please instead of "print to pdf"

--- exit stage right as it starts printing ---

When i was asked to work on their networking though i would say, "only if you can provide a network diagram/topology" . I perfectly well knew they couldn't because they never made one for their side of the network. Their network closet was an actual birds nest. Like you had to walk on the cables to get to the rack, like the rack looked like vines covering a tree and all the walls. There was more un used cables ran in there then used ones. Patch panels, what patch panels. Idk how it looked like that for only having lile 8 switches, 2 firewalls, and 2 routers.

Grammerly broke like half way through this so sorry not sorry.

In days gone by there was a project to refresh our site.

Pallets upon pallets of kit arrived. Desktops, monitors, as many as our storeroom could take. Desks were stripped and rebuilt. So shiny. So new. So uniform. So factory default. It had a certain unblemished beauty akin to thick, level, settled snow.

"Are we going to refresh the network cabling too?" I ask, thinking of the decades old CAT5 poking out through holes run directly from the server room, not a single RJ45 wall socket in sight. The response I receive only adds to the Christmas card like image in my head. The absolute crisp, delicate serenity of acoustically insulated silence.

Santa visits often, and is generous. Boxes arrive, boxes are opened, boxes are sorted for recycling.

Some of the boxes are different to the others. Some are flatter. Some are heavier. These boxes arrive, these boxes are not opened, these boxes are set aside in the corner.

Santa has been at it quite a while now. We must be the goodest and bestest boys and girls in the world. Boxes arrive, boxes are opened, boxes are sorted for recycling.

"Is Santa going to keep coming all year?" I ask, aware of the quantity we're accumulating. Such exquisite, pristine silence. Boxes arrive, boxes are no longer opened, boxes fill up our storeroom.

PCs are ready to be imaged. So exciting! All the new toys, so new and so perfect, so delightfully fresh and identical. So full of baseline, known condition, uncompromised promise.

Images deploy, oh no. Such strange and erratic behavour! Intermittent faults and discrepancies! If only we had gone for the network cabling refresh. What a sad little disappointment on the happiest day of the year.

I think all is not well with Santa either. Boxes arrive, boxes are shunted straight into stores, boxes pile high. We are given the schedule for several more weeks of deliveries.

"What are we going to do with all this equipment?" I ask, concerned. The snow is beginning to turn a little yellow, yet still there is silence.

Maybe something is wrong with Santa. I write Santa a short letter. Santa replies that this is none of my business and to mind my own.

I am sad that Santa would speak to me this way when up to now he has been nothing but generous. I write Santa a longer letter this time. One with a count of the total number of desks on site, the number already set up, and the number of kits in storage compared to the number of empty desks.

Santa calls me immediately. "Err.. are these numbers right?". Santa and I become close friends, he calls me every week now. He only wants to talk numbers and dates for deliveries though. For many many weeks we only talk about the same numbers again and again, and I tell him he has been far, far too kind. Surely there must be other very good children in the world as well? I begin to suspect Santa only wants to talk to me so he can tell all the other santas who to speak to if they find out their generosity was misguided.

One of the other santas writes a letter to me, Santa2 is very worried he has missed us off his list. Santa2 wants to know if we received an extra extra special box. I tell him we have that very special box yes.

"What about the other boxes?" I ask, wondering about the tower of switches in the very far corner of the storeroom. No one has said we can open those yet. Silent Night.

I count again the number of desks in the entire building and divide by 48. On no, Santa. We have been sent far too many of these boxes. There must be several good boys and girls with their own sites that didn't get their switches by mistake! Several! These can't possibly be all for us?

I'm worried about Santa2 now. I write a letter to Santa and Santa2 about the switches. "Santa was on a temporary contract and doesn't santa for us anymore". Oh. Santa did not even say goodbye.

Santa2 did not say anything about the switches. This time I write a letter to six or seven santas. "Why do we have X times 48 switch ports for a site with Y number of endpoints?"

Santa2 calls me straight away. "..ummm. Can you send me a photo of them?"

“And give me some disgustingly hot coffee. Black as midnight on a moonless night."

“Blurp?” the nameless thing behind the counter said.

"Oh, pretty damn black, I should think."

“Bloop.”

"What?"

"Whiiiir.”

"An overcast moonless night?"

“Phssssh?”

"I'm sorry? What did you say?"

“Beep-blip boop.”

"A moonless night, that is as black as coffee.”

"Glurgle drip-drip-drip."

"Thank you."

As I reach for my fresh cup of caffeinated jetfuel a colleague pokes his head in and calls for my attention. Something is amiss in the digital dreamlands and I am needed to risk my sanity diving the datastreams once again. Two hours later and the binary gremlins have been placated once more, vanquished but not dead, merely dreaming in their deep slumber until some other unfortunate soul trespasses on their domain once more.

I finally sip my coffee. 

It’s not hot anymore. 

Only disgusting.

TL;DR: Dropped into flow mindspace solving a bug ticket and even forgot about my newly-made coffee.

So, this isn't about a computer or an internet system, but I've posted "tech" tales beyond that realm that both remained and got more upvotes than down. That said, if you know a better place for this, drop me a line.

I worked for a communications company, that we’ll call Company A. We were on contract with Company B, to maintain their paging system.

Although pagers have been phased out in most places, some hospitals are an exception. This has one chief advantage that it can still work in parts of the building where you might not get a cell phone signal, provided the system is designed properly. Another thing is that it would be almost impossible for any type of spam or scam to get through, and very easy for a (company A) or (company B) technician to block if that ever did happen. In short, that means that it’s really only going to beep/vibrate for things that are work related.

Most of the transmitter racks were on the hospital sites themselves, connected to the backup power systems that held up the hospital’s critical stuff.

Back in the early days of cell phones, these paging systems once had the advantage of being more reliable as well. Unfortunately, (company B) never upgraded to a newer system, relying on 20+ year old technology to still be able to run 24/7. Mind you, the paging system was actively used multiple times a day for routine things, so this wasn’t something that just sits on standby until needed for an emergency.

The system was modular, somewhat like a desktop computer, and designed with field service in mind. Almost anything but the “backplane” board itself and the wiring could be serviced from the front of the rack. That, and the antenna would be on the roof of the building that the system was in. You could usually just unscrew things from the front, at most needing a Philips screwdriver if that (the original screws were thumb screws, but sometimes those got lost and you only had “regular” ones on hand at that particular moment)

The system was made up of 2 different power supplies, the motherboard, the keypad/display module, the amplifier module, and the “forget-me-not” module, whose purpose I forget, so we’ll just call it Module F if I refer to it again. Unlike most systems, the motherboard actually wasn’t the hardest thing to replace despite the necessity of everything connecting to it. The backplane board allowed you to swap out everything else, and it had nothing but connectors on it, specifically so it would be the least likely “module” to have anything go bad, BECAUSE the engineers realized it would be the hardest board to replace if it did go bad.

Both power supplies had the same voltages and current capacities, but one powered the amplifier, the other powered everything else. There were 2 indicator lights on the power supplies, one to indicate that it’s getting input power and it’s switched on, the other would indicate if it detected a fault with itself. Though the latter wasn’t foolproof (i.e. it could be “bad” without the fault light coming on) it DID help when it worked. Since they were the same PSU, you could swap them to see if the symptoms changed.

Most things that required more tech knowledge beyond basic computer servicing involved the antenna or configuring the software with a laptop. Even though it used a serial port, a proper USB adapter and their software design made it still usable with modern laptops, so at least we didn’t have to try and keep 20+ year old laptops working.

The most common things to go bad in the early days were the amplifier, and the power supply that ran it. That, and problems with the antenna wire connections, usually the ones on the roof that were exposed to the weather and such. Those usually resulted in missed pages only in the outskirts of the service area, assuming a preliminary “maintenance alert” didn’t show up first. Usually a total failure was either the motherboard or the “other” power supply that ran everything else. Module F rarely went bad until much later, and even the motherboard was pretty reliable until things started getting really old and rickety.

Unfortunately, that’s when I got hired on. By then, we were a frequent flier to nearly every location that still had these things. But none of the hospitals nor anyone else Company B supported had upgraded to a newer system. The only reason I can think of, was maybe that a new system wouldn’t be compatible with the old pagers and so they’d have to replace all of those?

Anyway, I’m dispatched for my first solo repair trip to Hospital 400. This site had totally gone down with no warning, which sounded like a motherboard or PSU issue. This hospital didn’t have an ER per say, so at least a missed page was less likely to spell disaster here. I just packed spares for everything we had, an antenna power meter, the laptop with the serial adapter and it’s charger, the whole 9 yards. We also had a little stack of paper explaining the command line interface and what you had to type to do certain things or reconfigure stuff.

The front desk staff gave me a temporary access card to get into the back room where the paging system (among other things) was. I scanned the card and, what do you know, it worked the first time they “programmed” it, which is more than can be said for other places I had been. Oh, and the doors swung open automatically, which I thought was cool, and pretty helpful if you had a rolling toolbox or a dolly full of stuff.

I went to the system, and sure enough, the little display is blank and didn’t respond to any key presses. There was a fan running, but it otherwise appeared “brain dead”. Well, replacing the motherboard would mean redoing all the configuration and such, so I turned everything off and started pulling out the power supply first. Ouch! I caught my thumb in the rack! That’s gonna leave a mark. Well, at least I’m at a hospital. So, after correctly placing my fingers this time, I removed the power supply properly without hurting myself any further. Putting the new one in, however, didn’t yield any progress. Okay, so the motherboard was the next natural suspect. I plugged it in and voila! No display, no sign of life other than the fans. How could this be? The keypad/display module had to be unplugged anyway to replace the motherboard, so if that had a loose connection, I would have already fixed it by unplugging it and plugging it back in. Well, I swapped the two power supplies between each other’s slots, in case the “new” one was also bad. No dice.

I’m out of ideas as to why the system wouldn’t boot at all, so I contacted Company B tech support. To their credit, I got a live person. Per their advice, I swapped the amplifier module: Apparently, there was digital 2-way communication between it and the motherboard, and if it were faulty in such a way as to put “garbage” on the communication line upon receiving power, it could prevent the motherboard from booting. That meant another trip out to the company car, seeing how I only had brought a power supply and motherboard.

I started toward the door I came in and was about to push the exit bar when I noticed the sign saying “DO NOT TOUCH DOOR: use sensor on wall”. I looked to the side of the door, and there was only what looked like a blank plate there where an exit button would have been. I waved my hand over it, and of course nothing happened. There was something above the door that looked like a sensor, but practically high-fiving it (without touching it) had no effect. There were also what looked like sensors on the door itself, but waving at them or doing jumping jacks did nothing.

Uh oh. I look around for other exits. There was one behind me, but it looks like I’d have to go clear around the building to get back to the car, and if the back lot was fenced in, I’d really be up a creek without a paddle. I did have my cell phone, but my eyes landed on the land-line phone in the room first. I figured it would be easier to call the front desk from that.

There was no written directory on or near the phone, and no directory on it’s little display. I tried the most obvious 0 for operator. I waited on hold for several minutes. As soon as I got someone, I asked “how do I get out of the back room without the alarm sounding?” and they were confused. I described the room I was in, and it dawned on both of us that I had reached someone in another building. I said “Can you put me through to the front desk at (hospital address) or security, maybe maintenance? I’m not trapped, but I just don’t want to trip the burglar alarm cause there’s signs telling me not to touch the door” “I’m sorry, I don’t know the right extension or who to contact for that” “Okay, never mind, and thanks for trying anyway. This isn’t your fault” and with that, I hung up.

At this point, I was just going to open the door manually and let the alarm sound. If anyone questioned me, I’d point out the process I went through to try and avoid the alarm. Just as I was about to open the door, I heard the card reader on the other side go “beep” and the door swung open. I explained my whole ordeal to the guy coming in, and he pointed out the little hand-wave sensor on the wall about 10 feet behind me, partially obscured by a shelf. Not only that, the sensor didn’t say “exit” or “do X to exit” nor was it the same color as the door or anything attached to it.

After the whole getting “trapped” in the room ordeal then being “rescued” I got back out to the car and got another amplifier module. And a Module F along with another power supply, for good measure. I planned on putting the original motherboard back in so I wouldn’t have to reconfigure everything if the amplifier turned out to be the problem. I put the original motherboard back in, then the “new” amplifier module. I left the first replacement power supply in place, just in case the original one had somehow fried other modules.

Before turning things back on, I put the antenna power meter in series with the transmitter line to the antenna. One might assume an output short circuit would be the only way to damage an amplifier, but for strong transmissions at high frequencies, an open circuit can result in “reflected” power back to the amplifier, which can also damage it as well. Therefore, it’s good practice to check for reflected power if the amplifier fails. (Better systems can detect this and shut themselves down. In fact, this system is supposed to be able to detect this, but just in case it doesn’t, it’s still good practice to check for it anyway)

I turned it on, and the original motherboard booted when I had the “new” amplifier in. So Company B tech support was right on the money. Score one for them. Good news: Reflected power wasn’t nearly high enough to cause any problems. (There is usually a small amount when the connectors are several years old, but it’s only a problem above a certain amount. Radio techs will know what I’m talking about)

So, I left the site for the day, declaring the ticket closed.

This one reminded me of our first experience at converting to VOIP.

When the day arrived to start replacing the 40 year old phone systems in our stores (due to replacements for broken phones costing $300 each, refurbished, at best, with a 50/50 chance they'd work), we talked to our phone (and internet) company about what was available. They were transitioning from providing analog service (over the internet, but converted on-site) to VOIP. OK, makes sense, it's the right choice.

But they were new to VOIP, and still getting a handle on how to make things work the way the customer wants, and most of their customers are offices, not brick & mortar retail. They don't know what we need, and we don't know what to ask for. OK, we'll work with them on that. They've always taken pretty good care of us. (Not the cheapest, by any means, but reliability is more important to us, because without the computers, we've got about 3 days offline before it all starts to fall apart.)

The problem was, the phones they sold us came out of the box configured for an office. The receptionist's phone rings, they answer, and transfer the call to the person it's for by extension number. Retail stores don't work that way. We need all phones to ring, be able to pick up the call anywhere, put it on hold, pick it up from another phone, lather, rinse, repeat as the associate runs around the store answering questions.

On the old phones, this is one button to put a call on hold, and one button to pick it up from any phone. The default way to do this on the new phones involves multiple buttons (including a four digit extension number) to park the call, and an equal number of buttons to pick it up. (And there's a button marked "Hold" that actually puts the call on hold - but it can only be picked up on the same phone.) Needless to say, this won't fly, even with a running start from a tall cliff. They knew this going in.

So they configured the phones to move the call to a park line with two buttons, and pick it up from any phone with two buttons. Worked . . . Okay, if not ideal. Except it relied on a feature that was deprecated, and when they did a software upgrade, it disappeared.

Then they found another way to do the same thing, but with three or four buttons. Not ideal at all, and it tended to result in dropped calls. And the particular store we converted first is in a beach city full of people rich enough to live at the beach, but not rich enough to live in the very rich beach city next door, which makes them . . . cranky. These people won't call back to finish their call, they will call back to scream obscenities at you. The only reliable way to put calls on hold such that they could be picked up on a different phone was the "official" way involving waaaay too many buttons. Plus, it also relied on a deprecated feature, and was never going to have the bugs addressed.

One of the cashiers - who has been with the company for a couple of decades and was the best they had for working with customers - simply refused to answer the phone. And her manager didn't blame her. The manager (also one of our best) was ready to quit over it. And we didn't blame him.

So a conversation was had, between me, who makes all the technical decisions, my boss, who has signature authority, our account rep (who was sympathetic, but had the technical savvy of a turnip), a senior VP from the phone company, and the lead technical guy for the phone company. It started off with my boss telling the VP "You have 30 days to make this work the way we want, or we're finding a new phone company." We were, at the time, about a quarter million dollar a year account (we're more now). Not their biggest, by any means, but big enough to get their attention. (Our account rep told me he had bigger accounts, but not many, and not by much. He was very concerned. I believe his bonuses were heavily based on total revenue generated.)

Took 'em two weeks, but they discovered some kind of macro feature ("key system emulation"?) that let them reprogram the buttons to do the complicated sequence to move the call around. One button to put a call on hold, one button to pick it up anywhere, and it worked (and we got CallerID to boot!) almost exactly like the old phones. Only difference was it was a different button to pick up the call than to put it on hold. I requested the tech guy write detailed instructions on how he did it and include it in the records for that store, so that I could refer future project engineers to that when we got to other stores. And he did, and I believe they still use those instructions to this day. And we're not their only brick & mortar customer these days, either.

Some years back, it was decided that our analogue phone system would be replaced.

Once this decision was made and everything signed, we in IT were notified of this change.

In that order. Yes.

My boss naturally let his many and well qualified thoughts be known, but as is common here these were dismissed. For those familiar with OFSTED, our overall rating was "Good", while their rating for Senior Management was "Needs Improvement". For those not familiar a government agency rated us as 3/4 stars overall and 2/4 stars for management (4/4 being Outstanding and 1/4 being Inadequate).

The person responsible for this was neither IT or senior management, I don't recall her role exactly now but she was the villain of many of my stories. How her proposal got accepted without our input or even knowledge would be mysterious and a cause for great concern anywhere else, but what can I say any more eloquently or succinctly that OFSTED have not?

So we meet with the supplier. Our questions are asked, and some are answered. One in particular was compatibility with ethernet daisy chaining computers with our existing setup - VLAN'd, solid and secure as it was. "Yes yes yes, all that will work". One of the techs in particular had an attitude that I could describe as "needs improvement" and customer service skills that were "inadequate". I had the strong feeling from him that he was in his very early 20s, possibly this was his first techy job, and was absolutely blindly loyal to the company having known little else in his career. His response to many of our concerns could essentially be translated to "No. Our product is good. Our product is beautiful. Our product is right, and you are wrong to question it".

I sat in on one training session. There was one member of staff in HR who I had a good relationship with and had been very kind and supportive to me over the years when I needed it, and she was always very appreciative when it was my turn to support her technical issues. We respected each other and were humble to each other's expertise, I had a soft spot for her and was always available to her - a few occasions in the fire together trying to get the monthly payroll processed with a third party on time will forge strong bonds. She was very excited and asked a very interesting, pertinent question about a certain feature. Mr Inadequate got Right. In. Her. Face. and hissed "NO! It doesn't do that!". She was absolutely crushed and I was incensed.

Do our desktops PXE boot through the phones? Do they balls. All staff are now without both their computer AND desk phone whenever we need to reimage. Mr Inadequate's response is of course to blame our network. I'm neither surprised or bothered by this, who amongst us, hey? Evasion and misdirection of blame between IT and a supplier? Bread and butter work, all the live long day. I'm not angry at Mr Inadequate for this, I'm deeply disturbed. He's not making excuses. He BELIEVES. He's of absolute faith in the infallibility of The Product. It's actually a little frightening to see the zealotry a young man can display for reselling a third rate IP telephony system.

My boss does all he can to mitigate the nightmares, there are delays and pushback from us and the general staff. Complaints roll in, we redirect everyone moaning to us in the Villain's direction and make it clear who is liaising (responsible) for all queries related to the new phone system. As we weren't consulted there is nothing we can do, there's no technical requirement to hold them to or UAT for them to complete. There's barely a week of snagging support, then we're shunted to their helpdesk for standard assistance.

The only happy ending to any of this was when the Villain who had unleashed all of this on us made a very genuine, very sincere, and very out of character apology to us.

A tale from before the pandemic when i was doing a lot more server maintenance.

Due to a botched update I got the fun(1) task to do midnight maintenance on one of the applications we are supporting.

Midnight hour strikes and I ssh into the machine, stop the application, untangle the bad update parts and do the needful. Pretty straight forward job, just remove a few folders and the application will recreate them correctly on startup. Time spent on this is maybe 5 minutes tops.

Now to restart the application and verify my work as good. I enter in the command to start up the application and switch to my browser to verify that it starts.

Nothing. Application URL only shows the hated nginx landing page.

Ok, something must be wrong. Let’s not panic, I’ll just restart the application. Maybe it couldn’t recreate the folders.

I restart the application again. No deal. Ok, time to do a deep dive in the logs.

I scour the logs, tailing in one tab and manually searching the same log in another tab. After scanning the log i can definitely guarantee that there is nothing out of the ordinary in them.

By this time my brain is thinking in terms most commonly transcribed in comics books by symbols most commonly found below the F-keys and the letters on a standard keyboard.

*ticktock ticktock time’s ticking*

I update the application browser page again in the vain hope it’ll work. No dice, still just the nginx landing page. After staring at the nginx page for almost 10 minutes I weigh my options:

a) Double down and bullhead my way through, hoping something i do will fix it

>bad idea, might frack it over worse

b) Leave it as is and grab SrTech first thing in the morning.

>worse idea, will generate a lot of incident reports about application being down before i get in to work tomorrow

c) Bite the bullet and call SrTech for help, waking him up in the middle of the night when i know he has a long day ahead of him tomorrow

>…oh blast. This is the best choice, isn’t it?

A few more minutes pass as i hope against hope i won’t have to call SrTech. No good, gotta do it.

I call up SrTech and the conversation goes something like this:

SrTech*: *mumbled sleepy greeting**

Me*: Hi SrTech, It’s me. Sorry for waking you up at this hour but i think i have messed up the application restart*

SrTech*: OK, walk me through what you have done*

Me*: I shut down the application, did the needful and then when i started up the application it won’t start. It just goes to the nginx landing page even if i update…*

Here i updated my browser to emphasise my words, even though i knew SrTech couldn’t see what i did.

It. Bloody. Worked.

Application is up and running as smoothly as if nothing ever happened

Me*: …um nevermind. It works now. Sorry for waking you. I’ll let you get back to sleep.*

SrTech: * mumbled sleepy goodbye *

(1) Fun in the Dwarf Fortress sense of the word. Remember: Losing is Fun.

(Reintro: Support engineer at a company based in Seattle who is known for a tornado)

A common wisdom is to never go into maintenance without "mount(ing) a scratch monkey". There's a story to why they call it a "scratch monkey" involving a swimming primate, but the point is this - if you're going into maintenance mode, make sure you've tagged in/tagged out, signed off, opened the maintenance window, inform your users that this is gonna be a little bumpy, and you do the thing within that temporary arrangement because if you don't, you're going to blow up the pager.

Here's one such story.

A call comes in, we say hi and all, and he needs a remote right away. The colleague o' mine who owns the case is out that day. Line's noisy, so I tell him we can't get that going without a diagnostic file.

...which he...can't...get.

At this point, I started asking for a read on the errors he's seeing. It took me four tries to get it in a way he could understand - though to be fair, English is a hell of a language. But he basically started reading a bunch of daemon restarts.

...ayup, we're going to Teams.

Issue at hand is simple: after upgrading the operating system from an RMA replacement, an attempt to load the configuration backup failed for reasons unknown to me. The result is multiple daemon restarts.

We go in. I can't take control, so I watch the daemon restarts. Can't run the diag dump on the CLI, it requires a daemon that's not starting to actually be able to run. Reboot...um, well, it did work fine for all of ten seconds and then they could not get a thing started. I think now's a good time to roll back.

Talking somebody through command line is sometimes painful.

We get the CLI going, I tell him to run the diagnostic once more...and it burps. OK, let's start from the top. Let's roll back to the previous version. Run the command to change volumes and...

...hey. Hey, wait a second. Where's the other volume?

Again, three times asked - you started on this earlier version, where'd it go? Same cagey answers. And then I ask the big one.

"Did you delete that volume?"

They hesitated, and responded. Yes. Yes, they did in fact delete that volume. Somebody grabbed onto that idiot ball hard and decided it was not needed. And this is where a snippet of "Poor, Unfortunate Souls" from Disney's Little Mermaid starts playing in my head. In a fit of ignorance, they manually dismounted their scratch monkey. They blocked their fire exit. There was only one way to respond, and it required the placement of my forehead into the palm of my hand.

"I really wish you hadn't done that."

See, there are two ways out of this jam. One is to go in, review logs, and see if you can spot the bogey. This can take some time. The other is to simply bust out some bootable media and reinstall. And with this level of palpable inexperience, the decision was simple: take off and nuke the site from orbit, as it's the only way to be sure.

And I suppose it was good news for them that they could arrange bootable media and a trip to a data center.

I heard they called back, but that was the end of it from my perspective. Even so, this appears, once again, to have been a combination of ingrained ignorance combined with some unfamiliarity of the English language that tends to come up with when English is your second language - and at least one of these guys could not communicate without simplification (thus the thrice-repeated parts above) - and given that they called apparently not knowing how to boot and install despite instructions being in front, I suspect their greatest weakness was reading my language - the sort of weakness that can have you thinking Bellyvoo^1 is wee ired^23. So in my frustration, these guys have some sympathy for me - because my two native languages^4 are insane.

^1 Bellevue

^2 Weird

^3 Phonics, man, phonics. Not 100% accurate beyond second grade reading.

^4 English and bad English

In my previous job, I worked as a field service engineer maintaining ophthalmic devices. In this role, I needed to be an IT specialist, mechanic, furniture mover, and truck driver. For some reason, they found it very hard to get anyone with all the requisite skills to apply! I quite liked the job at a base level, but they were screwing me over with the pay, so I left after the latest salary review wasn’t at all to my liking. But that’s another story.

One day, I was called out to repair an optical device at a hospital. The unit was a portable slit lamp—a handheld device used to examine the eye. Like most hospitals I visited, the biomedical department was tucked away in the bowels of the building, involving lots of long corridors and doorways to get through. This time, I was escorted in because I’d never have found the place otherwise.

They showed me to a bench with the offending device, and I got to work. It was a pretty simple fix—just a loose internal lens that needed to be glued in place. The issue was that the glue we used required UV light to cure, and I didn’t have a lamp with me. No problem; we usually just take the devices out into the sun for a minute, and the glue sets pretty quickly.

I looked around—no windows. I looked down the hall…no windows.

“Excuse me, I know this is going to sound silly, but where might I find the sun?”

I was directed to the loading bay, just down the hall and through a couple of doors. I gingerly carried my patient, being very careful not to bump the lens, which was positioned just right. I found the loading dock, but…no sun!

The hospital walls loomed upwards, giving me only the smallest sliver of sky. I could tell there was sunshine somewhere, but just not here. So I started walking, both hands keeping the device steady while also looking out for trucks and whatnot. Eventually, I found a welcoming beam of sunlight calling out to me. I walked into it and lifted the slit lamp into its rays like I was presenting a chalice to the gods of fire.

I stood there for a minute to ensure the job was done, trying to look casual and normal to the few people who passed me. But it’s not easy to look normal in that situation, so I just stood there like an idiot until the job was done. I found my way back, finished the repair, and tested the unit. Everything worked, and I packed up my stuff.

Later that day, I went online and found a nice, powerful UV torch that would handle the job without me roaming the halls looking for the sun like some reverse vampire.