A bonus story for today but going to be vague. won't get in trouble if im not but still probably a good thing if im vague.

So after some of the terrorist attacks in the early 2000, my company contracted a big aerospace company to build a system to automatically detect bio weapons. Said system is very expensive and requires a lot of maintenance and has multiple people monitoring it remotely.

One day my dispatcher received a call from a remote monitoring site saying that we need to check one of our machines because it's retesting a sample. My poor dispatcher interpreted this that it has detected something and being the only one on my shift trained for that system they called my cellphone directly. Dispatch doesn't know our cells because we have radios. So they got management to call me because no one wanted to talk about this over the radio. Was given direct orders to tell no one and go immediately to the machine. I arrive at the machine with everything running fine with no fault lights. So I logged in. The issue, a bad test tube. This machine has multiple so if it fails to get a "false" but also fails to get "positive" it will retest with a new tub. Nothing major, it just runs another test, the test do take awhile though so i got to sit and watch a screen. Did call dispatch back though and told them we are not under attack so they can calm down.was very over dramatic.

Time for a couple more badly written stories, words are hard and I never went to college. You get what you get.

Get some doordash, maybe some Adderall or whatever your vise is and enjoy

My job is tech support related but not directly. I work on anything from servers, networking, to automation(belts, motors, bearing) and PLCs. I'm a jack of all trades and definitely a master of none.

:The disappearing fault

So one day Operations calls us due to an output module fault. It looks like 7 modules lost communication. Well we first check the com cables, 2 40pin cables that create a loop for 4-19 modules. They seemed fine but Admittedly avoid these cables because I hate them, bulky, bad retention mechanism, and likely to have more problems just from touching them. All the cables for controlling coms, gates, actuators, and safety loop go to a backplane that slots into the main control PCB. So we replaced the main PCB, nothing happens except for even more faults. Then we got a second one, kinda worked just different faults this time. So we got a third one, most faults gone except one but communication is back for everything. At this point I called a remote SME, system matter expert. Who says to swap the board with another module to see if the fault follows the main board or stays with the module. One problem, it does nether, it just disappeared. Doesn't make sense to me but it's gone and the machine works.

Main lesson learned, just expect all your parts are bads.

:When the SME is wrong

So Operations calls about a machine intermittently stopping for a safety loop fault that never calls out where the fault is in the machine. The machine will act good when not processing but after 30s to 5m it will fault out. We arrive and start looking at interlocks but couldn't find anything. We keep pressing start till we get the fault to show up and not immediately disappear. We checked a 24v safety aux contact attached to a relay, even though it didn't test bad it's so common we replaced it. After checking every interlock we can't figure it out. So we call the remote SME. One piece of info I did have to diagnose the issue is the aux contact has no power. Letting the SME know this he first says to replace it, I told him I already did, also told him I'm not trained on the equipment though so I'm at a lost and need schmatics. He emailed me the schematics and also wanted me to follow the schematics after the aux contact which didn't make sense to me because it wasn't getting power. I felt I should trace where the power comes from and see where I get it back. So I lied to him that I would then with the self confidence of a stupid person, I did my own thing. Found power going into the safety loop at the breakers, the breakers have the ability to tell the computer if they're tripped, but not coming out. Started shaking the connectors on the back of each one until I found one that would make the machine go ready and not ready just from wiggling it, yes I wore gloves if anyone from OSHA is asking. After the machine was down for 5 hours, one aux contact and one breaker later it was fixed.

To explain what was happening, the machine vibrates when running due to motors, bearing and belts. This vibration would cause the tab inside the breaker to disconnect momentarily causing the machine to stop due to the safety loop opening.

:when the senior tech doesn't compute

So on day shift they had a machine go down due to the output modules not communicating. The senior tech(for day shift) found the module not communicating and replaced the board but still wasn't able to fix it. Even shift with the 2 most senior techs out of everyone refused to touch it. Finally when i came in on night shift and was questioning why we had a machine down i decided i would look at. Still not trained but from knowledege of last time I asked if anyone configured the board. All evidence pointed to no. So one call to the SME for the document on dip swich configuration and crawling inside the machine later the machine worked...

Unsure if anyone else shares my frustration but fixing stuff more experienced and trained people shouldv'e give mangement unreasonable expectations of your ablities. I love solving problem, i don't love being put on a pedistol.

Btw the down time of that machine probably cost $150k-300k

:how to solve a random persons problems from 500 miles away.

So the techs at my company have a facebook group for memes but also for help when SMEs are no help.

A person in another state posted they have had a machine down for over 7 days. The machine would only fault out if you tried to run it. With the fault being a communication fault from the operator PC to the on site OCR server, Optical Character Recognition. The issue was they could ping the server, and PC and server would show connected in their respective software. They even ran a new cable from the switch to no avail. I guess no one on site or the SME thought to actually see what the switch was reporting. I had access to see the monitoring of every facility just not make switch configurations. I was bored and looked them up and saw a ton of errors. The port was configured correctly, so most likely bad port.

So I messaged the guy. We got me =me, tech= guy from that facility, and supervisor = his boss

Me- hey, i saw your FB post i think it's the switch port

Tech- we are going to reboot again

Tech- I'm going to make a group chat with my supervisor

---new chat---

Me- hi, i think this is an issue with the IDF switch, do you guys have anyone with cisco CLI training and log in.

Supervisor- i think so but he hasn't logged in awhile

Supervisor- SME says to check switch at machine, we replaced it but that didn't work. SME now says to replace IDF switch

Me- before stopping all operations lets just try another port

Tech- we need a ladder

Me- i see the switch lost power recently, did you guys have a power outage.

Supervisor- actually yes, thats when the problems started

Me- please take the cable from port 6 and plug it into port 8

---Note, port 6 is for the machine having problems, port 8 is for a machine that is working

---23 messages and 4 hours later of being ignored

Me- please take the cable from port 6 and plug it into port 8

Supervisor- that worked we think port 6 is bad

Me- plug the cable ftom port 8 into port 6 and see if it faults out too

Supervisor- it does

Me- that comfirms 6 is bad, have your tech open and cofigure another port and label port 6 as bad.

Supervisor- thank you!

Moral of the story sometimes you need to repeat yourself i guess. Still working on being assertive.

On the plus side this interaction helped me pass the interview to become a SME, just waiting for an open postion.

:the normal tech support call

So us machine techs are only supposed to fix anything related to machinery and their functions "processing infrastructure side". We consider anything not related "Lan side"(printers and supervisor computers).

One problem, one onsite "Lan side" tech covered like 6 plants almost all 120 miles apart. They could drive 5 hours for one call and responce time is like 2 weeks.

Due to how over streched this guy was, even though he didn't want my help, and my interest in tech I would help when i can. It was against the union contract but keeping the bosses happy was in everyones interest. I mainly would just help with printer problems and was well known by management for solving printer problems. After the print server/directory failed i was the only one to get the printers working while we waited for it to get fixed. Anyways here's the story.

-Over the dispatch radio

Supervisor- hey OP can you help me with the printer by machine 9

Me- On my way

---i arrive stage left

Supervisor- I can't get the printer to print, i think it's broken

Me- please bring up what you're trying to print

Me- press print or ctrl p please

Me- can you select the printer labeled "printer by machine 9" please instead of "print to pdf"

--- exit stage right as it starts printing ---

When i was asked to work on their networking though i would say, "only if you can provide a network diagram/topology" . I perfectly well knew they couldn't because they never made one for their side of the network. Their network closet was an actual birds nest. Like you had to walk on the cables to get to the rack, like the rack looked like vines covering a tree and all the walls. There was more un used cables ran in there then used ones. Patch panels, what patch panels. Idk how it looked like that for only having lile 8 switches, 2 firewalls, and 2 routers.

Grammerly broke like half way through this so sorry not sorry.

In days gone by there was a project to refresh our site.

Pallets upon pallets of kit arrived. Desktops, monitors, as many as our storeroom could take. Desks were stripped and rebuilt. So shiny. So new. So uniform. So factory default. It had a certain unblemished beauty akin to thick, level, settled snow.

"Are we going to refresh the network cabling too?" I ask, thinking of the decades old CAT5 poking out through holes run directly from the server room, not a single RJ45 wall socket in sight. The response I receive only adds to the Christmas card like image in my head. The absolute crisp, delicate serenity of acoustically insulated silence.

Santa visits often, and is generous. Boxes arrive, boxes are opened, boxes are sorted for recycling.

Some of the boxes are different to the others. Some are flatter. Some are heavier. These boxes arrive, these boxes are not opened, these boxes are set aside in the corner.

Santa has been at it quite a while now. We must be the goodest and bestest boys and girls in the world. Boxes arrive, boxes are opened, boxes are sorted for recycling.

"Is Santa going to keep coming all year?" I ask, aware of the quantity we're accumulating. Such exquisite, pristine silence. Boxes arrive, boxes are no longer opened, boxes fill up our storeroom.

PCs are ready to be imaged. So exciting! All the new toys, so new and so perfect, so delightfully fresh and identical. So full of baseline, known condition, uncompromised promise.

Images deploy, oh no. Such strange and erratic behavour! Intermittent faults and discrepancies! If only we had gone for the network cabling refresh. What a sad little disappointment on the happiest day of the year.

I think all is not well with Santa either. Boxes arrive, boxes are shunted straight into stores, boxes pile high. We are given the schedule for several more weeks of deliveries.

"What are we going to do with all this equipment?" I ask, concerned. The snow is beginning to turn a little yellow, yet still there is silence.

Maybe something is wrong with Santa. I write Santa a short letter. Santa replies that this is none of my business and to mind my own.

I am sad that Santa would speak to me this way when up to now he has been nothing but generous. I write Santa a longer letter this time. One with a count of the total number of desks on site, the number already set up, and the number of kits in storage compared to the number of empty desks.

Santa calls me immediately. "Err.. are these numbers right?". Santa and I become close friends, he calls me every week now. He only wants to talk numbers and dates for deliveries though. For many many weeks we only talk about the same numbers again and again, and I tell him he has been far, far too kind. Surely there must be other very good children in the world as well? I begin to suspect Santa only wants to talk to me so he can tell all the other santas who to speak to if they find out their generosity was misguided.

One of the other santas writes a letter to me, Santa2 is very worried he has missed us off his list. Santa2 wants to know if we received an extra extra special box. I tell him we have that very special box yes.

"What about the other boxes?" I ask, wondering about the tower of switches in the very far corner of the storeroom. No one has said we can open those yet. Silent Night.

I count again the number of desks in the entire building and divide by 48. On no, Santa. We have been sent far too many of these boxes. There must be several good boys and girls with their own sites that didn't get their switches by mistake! Several! These can't possibly be all for us?

I'm worried about Santa2 now. I write a letter to Santa and Santa2 about the switches. "Santa was on a temporary contract and doesn't santa for us anymore". Oh. Santa did not even say goodbye.

Santa2 did not say anything about the switches. This time I write a letter to six or seven santas. "Why do we have X times 48 switch ports for a site with Y number of endpoints?"

Santa2 calls me straight away. "..ummm. Can you send me a photo of them?"

“And give me some disgustingly hot coffee. Black as midnight on a moonless night."

“Blurp?” the nameless thing behind the counter said.

"Oh, pretty damn black, I should think."

“Bloop.”

"What?"

"Whiiiir.”

"An overcast moonless night?"

“Phssssh?”

"I'm sorry? What did you say?"

“Beep-blip boop.”

"A moonless night, that is as black as coffee.”

"Glurgle drip-drip-drip."

"Thank you."

As I reach for my fresh cup of caffeinated jetfuel a colleague pokes his head in and calls for my attention. Something is amiss in the digital dreamlands and I am needed to risk my sanity diving the datastreams once again. Two hours later and the binary gremlins have been placated once more, vanquished but not dead, merely dreaming in their deep slumber until some other unfortunate soul trespasses on their domain once more.

I finally sip my coffee. 

It’s not hot anymore. 

Only disgusting.

TL;DR: Dropped into flow mindspace solving a bug ticket and even forgot about my newly-made coffee.

So, this isn't about a computer or an internet system, but I've posted "tech" tales beyond that realm that both remained and got more upvotes than down. That said, if you know a better place for this, drop me a line.

I worked for a communications company, that we’ll call Company A. We were on contract with Company B, to maintain their paging system.

Although pagers have been phased out in most places, some hospitals are an exception. This has one chief advantage that it can still work in parts of the building where you might not get a cell phone signal, provided the system is designed properly. Another thing is that it would be almost impossible for any type of spam or scam to get through, and very easy for a (company A) or (company B) technician to block if that ever did happen. In short, that means that it’s really only going to beep/vibrate for things that are work related.

Most of the transmitter racks were on the hospital sites themselves, connected to the backup power systems that held up the hospital’s critical stuff.

Back in the early days of cell phones, these paging systems once had the advantage of being more reliable as well. Unfortunately, (company B) never upgraded to a newer system, relying on 20+ year old technology to still be able to run 24/7. Mind you, the paging system was actively used multiple times a day for routine things, so this wasn’t something that just sits on standby until needed for an emergency.

The system was modular, somewhat like a desktop computer, and designed with field service in mind. Almost anything but the “backplane” board itself and the wiring could be serviced from the front of the rack. That, and the antenna would be on the roof of the building that the system was in. You could usually just unscrew things from the front, at most needing a Philips screwdriver if that (the original screws were thumb screws, but sometimes those got lost and you only had “regular” ones on hand at that particular moment)

The system was made up of 2 different power supplies, the motherboard, the keypad/display module, the amplifier module, and the “forget-me-not” module, whose purpose I forget, so we’ll just call it Module F if I refer to it again. Unlike most systems, the motherboard actually wasn’t the hardest thing to replace despite the necessity of everything connecting to it. The backplane board allowed you to swap out everything else, and it had nothing but connectors on it, specifically so it would be the least likely “module” to have anything go bad, BECAUSE the engineers realized it would be the hardest board to replace if it did go bad.

Both power supplies had the same voltages and current capacities, but one powered the amplifier, the other powered everything else. There were 2 indicator lights on the power supplies, one to indicate that it’s getting input power and it’s switched on, the other would indicate if it detected a fault with itself. Though the latter wasn’t foolproof (i.e. it could be “bad” without the fault light coming on) it DID help when it worked. Since they were the same PSU, you could swap them to see if the symptoms changed.

Most things that required more tech knowledge beyond basic computer servicing involved the antenna or configuring the software with a laptop. Even though it used a serial port, a proper USB adapter and their software design made it still usable with modern laptops, so at least we didn’t have to try and keep 20+ year old laptops working.

The most common things to go bad in the early days were the amplifier, and the power supply that ran it. That, and problems with the antenna wire connections, usually the ones on the roof that were exposed to the weather and such. Those usually resulted in missed pages only in the outskirts of the service area, assuming a preliminary “maintenance alert” didn’t show up first. Usually a total failure was either the motherboard or the “other” power supply that ran everything else. Module F rarely went bad until much later, and even the motherboard was pretty reliable until things started getting really old and rickety.

Unfortunately, that’s when I got hired on. By then, we were a frequent flier to nearly every location that still had these things. But none of the hospitals nor anyone else Company B supported had upgraded to a newer system. The only reason I can think of, was maybe that a new system wouldn’t be compatible with the old pagers and so they’d have to replace all of those?

Anyway, I’m dispatched for my first solo repair trip to Hospital 400. This site had totally gone down with no warning, which sounded like a motherboard or PSU issue. This hospital didn’t have an ER per say, so at least a missed page was less likely to spell disaster here. I just packed spares for everything we had, an antenna power meter, the laptop with the serial adapter and it’s charger, the whole 9 yards. We also had a little stack of paper explaining the command line interface and what you had to type to do certain things or reconfigure stuff.

The front desk staff gave me a temporary access card to get into the back room where the paging system (among other things) was. I scanned the card and, what do you know, it worked the first time they “programmed” it, which is more than can be said for other places I had been. Oh, and the doors swung open automatically, which I thought was cool, and pretty helpful if you had a rolling toolbox or a dolly full of stuff.

I went to the system, and sure enough, the little display is blank and didn’t respond to any key presses. There was a fan running, but it otherwise appeared “brain dead”. Well, replacing the motherboard would mean redoing all the configuration and such, so I turned everything off and started pulling out the power supply first. Ouch! I caught my thumb in the rack! That’s gonna leave a mark. Well, at least I’m at a hospital. So, after correctly placing my fingers this time, I removed the power supply properly without hurting myself any further. Putting the new one in, however, didn’t yield any progress. Okay, so the motherboard was the next natural suspect. I plugged it in and voila! No display, no sign of life other than the fans. How could this be? The keypad/display module had to be unplugged anyway to replace the motherboard, so if that had a loose connection, I would have already fixed it by unplugging it and plugging it back in. Well, I swapped the two power supplies between each other’s slots, in case the “new” one was also bad. No dice.

I’m out of ideas as to why the system wouldn’t boot at all, so I contacted Company B tech support. To their credit, I got a live person. Per their advice, I swapped the amplifier module: Apparently, there was digital 2-way communication between it and the motherboard, and if it were faulty in such a way as to put “garbage” on the communication line upon receiving power, it could prevent the motherboard from booting. That meant another trip out to the company car, seeing how I only had brought a power supply and motherboard.

I started toward the door I came in and was about to push the exit bar when I noticed the sign saying “DO NOT TOUCH DOOR: use sensor on wall”. I looked to the side of the door, and there was only what looked like a blank plate there where an exit button would have been. I waved my hand over it, and of course nothing happened. There was something above the door that looked like a sensor, but practically high-fiving it (without touching it) had no effect. There were also what looked like sensors on the door itself, but waving at them or doing jumping jacks did nothing.

Uh oh. I look around for other exits. There was one behind me, but it looks like I’d have to go clear around the building to get back to the car, and if the back lot was fenced in, I’d really be up a creek without a paddle. I did have my cell phone, but my eyes landed on the land-line phone in the room first. I figured it would be easier to call the front desk from that.

There was no written directory on or near the phone, and no directory on it’s little display. I tried the most obvious 0 for operator. I waited on hold for several minutes. As soon as I got someone, I asked “how do I get out of the back room without the alarm sounding?” and they were confused. I described the room I was in, and it dawned on both of us that I had reached someone in another building. I said “Can you put me through to the front desk at (hospital address) or security, maybe maintenance? I’m not trapped, but I just don’t want to trip the burglar alarm cause there’s signs telling me not to touch the door” “I’m sorry, I don’t know the right extension or who to contact for that” “Okay, never mind, and thanks for trying anyway. This isn’t your fault” and with that, I hung up.

At this point, I was just going to open the door manually and let the alarm sound. If anyone questioned me, I’d point out the process I went through to try and avoid the alarm. Just as I was about to open the door, I heard the card reader on the other side go “beep” and the door swung open. I explained my whole ordeal to the guy coming in, and he pointed out the little hand-wave sensor on the wall about 10 feet behind me, partially obscured by a shelf. Not only that, the sensor didn’t say “exit” or “do X to exit” nor was it the same color as the door or anything attached to it.

After the whole getting “trapped” in the room ordeal then being “rescued” I got back out to the car and got another amplifier module. And a Module F along with another power supply, for good measure. I planned on putting the original motherboard back in so I wouldn’t have to reconfigure everything if the amplifier turned out to be the problem. I put the original motherboard back in, then the “new” amplifier module. I left the first replacement power supply in place, just in case the original one had somehow fried other modules.

Before turning things back on, I put the antenna power meter in series with the transmitter line to the antenna. One might assume an output short circuit would be the only way to damage an amplifier, but for strong transmissions at high frequencies, an open circuit can result in “reflected” power back to the amplifier, which can also damage it as well. Therefore, it’s good practice to check for reflected power if the amplifier fails. (Better systems can detect this and shut themselves down. In fact, this system is supposed to be able to detect this, but just in case it doesn’t, it’s still good practice to check for it anyway)

I turned it on, and the original motherboard booted when I had the “new” amplifier in. So Company B tech support was right on the money. Score one for them. Good news: Reflected power wasn’t nearly high enough to cause any problems. (There is usually a small amount when the connectors are several years old, but it’s only a problem above a certain amount. Radio techs will know what I’m talking about)

So, I left the site for the day, declaring the ticket closed.

This one reminded me of our first experience at converting to VOIP.

When the day arrived to start replacing the 40 year old phone systems in our stores (due to replacements for broken phones costing $300 each, refurbished, at best, with a 50/50 chance they'd work), we talked to our phone (and internet) company about what was available. They were transitioning from providing analog service (over the internet, but converted on-site) to VOIP. OK, makes sense, it's the right choice.

But they were new to VOIP, and still getting a handle on how to make things work the way the customer wants, and most of their customers are offices, not brick & mortar retail. They don't know what we need, and we don't know what to ask for. OK, we'll work with them on that. They've always taken pretty good care of us. (Not the cheapest, by any means, but reliability is more important to us, because without the computers, we've got about 3 days offline before it all starts to fall apart.)

The problem was, the phones they sold us came out of the box configured for an office. The receptionist's phone rings, they answer, and transfer the call to the person it's for by extension number. Retail stores don't work that way. We need all phones to ring, be able to pick up the call anywhere, put it on hold, pick it up from another phone, lather, rinse, repeat as the associate runs around the store answering questions.

On the old phones, this is one button to put a call on hold, and one button to pick it up from any phone. The default way to do this on the new phones involves multiple buttons (including a four digit extension number) to park the call, and an equal number of buttons to pick it up. (And there's a button marked "Hold" that actually puts the call on hold - but it can only be picked up on the same phone.) Needless to say, this won't fly, even with a running start from a tall cliff. They knew this going in.

So they configured the phones to move the call to a park line with two buttons, and pick it up from any phone with two buttons. Worked . . . Okay, if not ideal. Except it relied on a feature that was deprecated, and when they did a software upgrade, it disappeared.

Then they found another way to do the same thing, but with three or four buttons. Not ideal at all, and it tended to result in dropped calls. And the particular store we converted first is in a beach city full of people rich enough to live at the beach, but not rich enough to live in the very rich beach city next door, which makes them . . . cranky. These people won't call back to finish their call, they will call back to scream obscenities at you. The only reliable way to put calls on hold such that they could be picked up on a different phone was the "official" way involving waaaay too many buttons. Plus, it also relied on a deprecated feature, and was never going to have the bugs addressed.

One of the cashiers - who has been with the company for a couple of decades and was the best they had for working with customers - simply refused to answer the phone. And her manager didn't blame her. The manager (also one of our best) was ready to quit over it. And we didn't blame him.

So a conversation was had, between me, who makes all the technical decisions, my boss, who has signature authority, our account rep (who was sympathetic, but had the technical savvy of a turnip), a senior VP from the phone company, and the lead technical guy for the phone company. It started off with my boss telling the VP "You have 30 days to make this work the way we want, or we're finding a new phone company." We were, at the time, about a quarter million dollar a year account (we're more now). Not their biggest, by any means, but big enough to get their attention. (Our account rep told me he had bigger accounts, but not many, and not by much. He was very concerned. I believe his bonuses were heavily based on total revenue generated.)

Took 'em two weeks, but they discovered some kind of macro feature ("key system emulation"?) that let them reprogram the buttons to do the complicated sequence to move the call around. One button to put a call on hold, one button to pick it up anywhere, and it worked (and we got CallerID to boot!) almost exactly like the old phones. Only difference was it was a different button to pick up the call than to put it on hold. I requested the tech guy write detailed instructions on how he did it and include it in the records for that store, so that I could refer future project engineers to that when we got to other stores. And he did, and I believe they still use those instructions to this day. And we're not their only brick & mortar customer these days, either.

Some years back, it was decided that our analogue phone system would be replaced.

Once this decision was made and everything signed, we in IT were notified of this change.

In that order. Yes.

My boss naturally let his many and well qualified thoughts be known, but as is common here these were dismissed. For those familiar with OFSTED, our overall rating was "Good", while their rating for Senior Management was "Needs Improvement". For those not familiar a government agency rated us as 3/4 stars overall and 2/4 stars for management (4/4 being Outstanding and 1/4 being Inadequate).

The person responsible for this was neither IT or senior management, I don't recall her role exactly now but she was the villain of many of my stories. How her proposal got accepted without our input or even knowledge would be mysterious and a cause for great concern anywhere else, but what can I say any more eloquently or succinctly that OFSTED have not?

So we meet with the supplier. Our questions are asked, and some are answered. One in particular was compatibility with ethernet daisy chaining computers with our existing setup - VLAN'd, solid and secure as it was. "Yes yes yes, all that will work". One of the techs in particular had an attitude that I could describe as "needs improvement" and customer service skills that were "inadequate". I had the strong feeling from him that he was in his very early 20s, possibly this was his first techy job, and was absolutely blindly loyal to the company having known little else in his career. His response to many of our concerns could essentially be translated to "No. Our product is good. Our product is beautiful. Our product is right, and you are wrong to question it".

I sat in on one training session. There was one member of staff in HR who I had a good relationship with and had been very kind and supportive to me over the years when I needed it, and she was always very appreciative when it was my turn to support her technical issues. We respected each other and were humble to each other's expertise, I had a soft spot for her and was always available to her - a few occasions in the fire together trying to get the monthly payroll processed with a third party on time will forge strong bonds. She was very excited and asked a very interesting, pertinent question about a certain feature. Mr Inadequate got Right. In. Her. Face. and hissed "NO! It doesn't do that!". She was absolutely crushed and I was incensed.

Do our desktops PXE boot through the phones? Do they balls. All staff are now without both their computer AND desk phone whenever we need to reimage. Mr Inadequate's response is of course to blame our network. I'm neither surprised or bothered by this, who amongst us, hey? Evasion and misdirection of blame between IT and a supplier? Bread and butter work, all the live long day. I'm not angry at Mr Inadequate for this, I'm deeply disturbed. He's not making excuses. He BELIEVES. He's of absolute faith in the infallibility of The Product. It's actually a little frightening to see the zealotry a young man can display for reselling a third rate IP telephony system.

My boss does all he can to mitigate the nightmares, there are delays and pushback from us and the general staff. Complaints roll in, we redirect everyone moaning to us in the Villain's direction and make it clear who is liaising (responsible) for all queries related to the new phone system. As we weren't consulted there is nothing we can do, there's no technical requirement to hold them to or UAT for them to complete. There's barely a week of snagging support, then we're shunted to their helpdesk for standard assistance.

The only happy ending to any of this was when the Villain who had unleashed all of this on us made a very genuine, very sincere, and very out of character apology to us.

A tale from before the pandemic when i was doing a lot more server maintenance.

Due to a botched update I got the fun(1) task to do midnight maintenance on one of the applications we are supporting.

Midnight hour strikes and I ssh into the machine, stop the application, untangle the bad update parts and do the needful. Pretty straight forward job, just remove a few folders and the application will recreate them correctly on startup. Time spent on this is maybe 5 minutes tops.

Now to restart the application and verify my work as good. I enter in the command to start up the application and switch to my browser to verify that it starts.

Nothing. Application URL only shows the hated nginx landing page.

Ok, something must be wrong. Let’s not panic, I’ll just restart the application. Maybe it couldn’t recreate the folders.

I restart the application again. No deal. Ok, time to do a deep dive in the logs.

I scour the logs, tailing in one tab and manually searching the same log in another tab. After scanning the log i can definitely guarantee that there is nothing out of the ordinary in them.

By this time my brain is thinking in terms most commonly transcribed in comics books by symbols most commonly found below the F-keys and the letters on a standard keyboard.

*ticktock ticktock time’s ticking*

I update the application browser page again in the vain hope it’ll work. No dice, still just the nginx landing page. After staring at the nginx page for almost 10 minutes I weigh my options:

a) Double down and bullhead my way through, hoping something i do will fix it

>bad idea, might frack it over worse

b) Leave it as is and grab SrTech first thing in the morning.

>worse idea, will generate a lot of incident reports about application being down before i get in to work tomorrow

c) Bite the bullet and call SrTech for help, waking him up in the middle of the night when i know he has a long day ahead of him tomorrow

>…oh blast. This is the best choice, isn’t it?

A few more minutes pass as i hope against hope i won’t have to call SrTech. No good, gotta do it.

I call up SrTech and the conversation goes something like this:

SrTech*: *mumbled sleepy greeting**

Me*: Hi SrTech, It’s me. Sorry for waking you up at this hour but i think i have messed up the application restart*

SrTech*: OK, walk me through what you have done*

Me*: I shut down the application, did the needful and then when i started up the application it won’t start. It just goes to the nginx landing page even if i update…*

Here i updated my browser to emphasise my words, even though i knew SrTech couldn’t see what i did.

It. Bloody. Worked.

Application is up and running as smoothly as if nothing ever happened

Me*: …um nevermind. It works now. Sorry for waking you. I’ll let you get back to sleep.*

SrTech: * mumbled sleepy goodbye *

(1) Fun in the Dwarf Fortress sense of the word. Remember: Losing is Fun.

(Reintro: Support engineer at a company based in Seattle who is known for a tornado)

A common wisdom is to never go into maintenance without "mount(ing) a scratch monkey". There's a story to why they call it a "scratch monkey" involving a swimming primate, but the point is this - if you're going into maintenance mode, make sure you've tagged in/tagged out, signed off, opened the maintenance window, inform your users that this is gonna be a little bumpy, and you do the thing within that temporary arrangement because if you don't, you're going to blow up the pager.

Here's one such story.

A call comes in, we say hi and all, and he needs a remote right away. The colleague o' mine who owns the case is out that day. Line's noisy, so I tell him we can't get that going without a diagnostic file.

...which he...can't...get.

At this point, I started asking for a read on the errors he's seeing. It took me four tries to get it in a way he could understand - though to be fair, English is a hell of a language. But he basically started reading a bunch of daemon restarts.

...ayup, we're going to Teams.

Issue at hand is simple: after upgrading the operating system from an RMA replacement, an attempt to load the configuration backup failed for reasons unknown to me. The result is multiple daemon restarts.

We go in. I can't take control, so I watch the daemon restarts. Can't run the diag dump on the CLI, it requires a daemon that's not starting to actually be able to run. Reboot...um, well, it did work fine for all of ten seconds and then they could not get a thing started. I think now's a good time to roll back.

Talking somebody through command line is sometimes painful.

We get the CLI going, I tell him to run the diagnostic once more...and it burps. OK, let's start from the top. Let's roll back to the previous version. Run the command to change volumes and...

...hey. Hey, wait a second. Where's the other volume?

Again, three times asked - you started on this earlier version, where'd it go? Same cagey answers. And then I ask the big one.

"Did you delete that volume?"

They hesitated, and responded. Yes. Yes, they did in fact delete that volume. Somebody grabbed onto that idiot ball hard and decided it was not needed. And this is where a snippet of "Poor, Unfortunate Souls" from Disney's Little Mermaid starts playing in my head. In a fit of ignorance, they manually dismounted their scratch monkey. They blocked their fire exit. There was only one way to respond, and it required the placement of my forehead into the palm of my hand.

"I really wish you hadn't done that."

See, there are two ways out of this jam. One is to go in, review logs, and see if you can spot the bogey. This can take some time. The other is to simply bust out some bootable media and reinstall. And with this level of palpable inexperience, the decision was simple: take off and nuke the site from orbit, as it's the only way to be sure.

And I suppose it was good news for them that they could arrange bootable media and a trip to a data center.

I heard they called back, but that was the end of it from my perspective. Even so, this appears, once again, to have been a combination of ingrained ignorance combined with some unfamiliarity of the English language that tends to come up with when English is your second language - and at least one of these guys could not communicate without simplification (thus the thrice-repeated parts above) - and given that they called apparently not knowing how to boot and install despite instructions being in front, I suspect their greatest weakness was reading my language - the sort of weakness that can have you thinking Bellyvoo^1 is wee ired^23. So in my frustration, these guys have some sympathy for me - because my two native languages^4 are insane.

^1 Bellevue

^2 Weird

^3 Phonics, man, phonics. Not 100% accurate beyond second grade reading.

^4 English and bad English

In my previous job, I worked as a field service engineer maintaining ophthalmic devices. In this role, I needed to be an IT specialist, mechanic, furniture mover, and truck driver. For some reason, they found it very hard to get anyone with all the requisite skills to apply! I quite liked the job at a base level, but they were screwing me over with the pay, so I left after the latest salary review wasn’t at all to my liking. But that’s another story.

One day, I was called out to repair an optical device at a hospital. The unit was a portable slit lamp—a handheld device used to examine the eye. Like most hospitals I visited, the biomedical department was tucked away in the bowels of the building, involving lots of long corridors and doorways to get through. This time, I was escorted in because I’d never have found the place otherwise.

They showed me to a bench with the offending device, and I got to work. It was a pretty simple fix—just a loose internal lens that needed to be glued in place. The issue was that the glue we used required UV light to cure, and I didn’t have a lamp with me. No problem; we usually just take the devices out into the sun for a minute, and the glue sets pretty quickly.

I looked around—no windows. I looked down the hall…no windows.

“Excuse me, I know this is going to sound silly, but where might I find the sun?”

I was directed to the loading bay, just down the hall and through a couple of doors. I gingerly carried my patient, being very careful not to bump the lens, which was positioned just right. I found the loading dock, but…no sun!

The hospital walls loomed upwards, giving me only the smallest sliver of sky. I could tell there was sunshine somewhere, but just not here. So I started walking, both hands keeping the device steady while also looking out for trucks and whatnot. Eventually, I found a welcoming beam of sunlight calling out to me. I walked into it and lifted the slit lamp into its rays like I was presenting a chalice to the gods of fire.

I stood there for a minute to ensure the job was done, trying to look casual and normal to the few people who passed me. But it’s not easy to look normal in that situation, so I just stood there like an idiot until the job was done. I found my way back, finished the repair, and tested the unit. Everything worked, and I packed up my stuff.

Later that day, I went online and found a nice, powerful UV torch that would handle the job without me roaming the halls looking for the sun like some reverse vampire.

I had a situation today which caused me a little panic, until I was able to think about it clearly.

On one of our website servers, there is a fairly strong and sometimes persnickety caching mechanism. It is so persnickety, that when we have to make an edit to a page -- such as a blog -- we have to be sure to make sure we check the page in incognito mode. Otherwise, if we are logged into the CMS and visit the page in regular mode, the update will appear, but it won't appear for others until the cache is cleared. However, I don't know what the cache retention policy is, so usually, we just clear the cache after an update and move on.

Today, a change was made to a page and it was passed over to me for my QA review, so I checked it in a new incognito session. The update had been made and everything was happy, so I reported up the chain that the update had been verified.

About 15 minutes later, the account person responsible for that website chatted me and said that she was not seeing the update. She has been bitten with cache issues before, so when she chatted me, she said that she had tried Chrome in both regular and incognito mode, and had also tried Safari. The update was not showing up on any of her browser instances.

I had someone else double-check for me, and that person was able to see the updates.

It was somewhat reminiscent of a problem I had encountered several years ago when I was at another company. In that instance, we had a weird load balancer situation, and a person would get assigned to one of the two load balancer URLs. So, instead of randomly getting Server1 or Server2, if you were assigned to Server1, it took a random, cosmic event of the universe to get you switched over to Server2. (Yeah, I know, that's not how load balancers are supposed to work. Don't care, that was about 8-10 years ago.)

Anyway, I knew that was not the issue in this case, because we don't have a load balancer, but something was preventing the user from seeing the updates, even though others could see it.

We got on a conference call and she even showed me that she was starting with a new incognito session. I even had her send me the URL she was using, thinking that maybe there were two instances of this page, but with different URLs.

Nope. Same URL, new incognito session, hard refreshing two or three times ... update still visible.

Then, she happened to mention, "I even tried it on my phone, and I'm still not seeing the update."

Everything is pointing to a stubborn cache somewhere between her and the website. She is about 175 miles away from me under a different ISP, so we definitely are not going through the same intermediate hops.

Then I asked her, "Is your phone going through your home's wifi?"

Turns out, it was, so she turned off that setting on her phone and hit the page using her phone's data connection. Hmmm ... the updates are appearing ... how nice!

From what I can tell, either her #WifiRouterModemThingie has some sort of stubborn cache mechanism, or, one of the hops she is going through has the stubborn cache.

So ... lucky guess or experience? You be the judge.

(Also, does anyone else have any suggestions on how I can check where the cache mechanism could be located? The user on the other end is not technical, so doing a tracert is not really an option.)

Got an alert about half an hour ago about a server on one of our remote sites being low on storage. Pulled up WinDirStat to poke around and see if there was anything I knew I could purge, click on a large folder to see the files inside, and my remote access locks up.

"Must be a lot of shit in that folder.." I'm thinking to myself, before I get completely disconnected. Minor warning bells start going off, but Connectwise Control isn't perfect software and I've seen this before, so I wait a second, and then..

Blip, blip, blip, other endpoints start going offline. "What the hell did I even do??" I start thinking to myself. Almost panicking at this point, I swap over to our network controllers access page and pop in on the site. They're completely offline. This actually calmed me down a bit as there was no way in hell WinDirStat was taking the entire network down, and checking in on another site in the same building confirmed it was a brief outage. By the time I was done in there all the endpoints were back up already..

Too early for that sort of panic lol. How's everyone else's day been?

One from my education tech support days.

Two students walk up to the helpdesk, and I walk out to greet them and ask them what's going on. They told me they were having troubles doing a maths test online, so I get them to open the laptop, log in and show me what's going on.

The website they use to do the tests will grey out the boxes or display an error on screen if the internet drops out or something fails to load. It happens once in a while, so I figured that was the issue. I pull the laptop towards me and type some numbers into the two boxes. It works, and they're connected to the internet, so I ask them what the issue is because as far as I can see, everything is working fine.

They proceed to tell me that they didn't know the answer to the question, and neither did their (substitute) teacher, so they sent the students over to IT for help. They said their normal teacher didn't know the answer either when they were in class the day before, so they've come to us for the answer.

I told the kids "this isn't IT related, so I can't help you". I asked who the teacher was (they didn't know, substitute, but I worked it out later on), and send them back.

So I guess the school wanted me to add "maths wizard" to my long list of jobs that aren't my job, like "coffee machine repairman", "lockpicker", "window repairman" and "delivery boy"

This one is from a few years ago but I was reminded of it today so figured I'd share.

My desk used to be near our help desk, which was handy because they could easily come around the corner and ask me questions as needed. It was also a great spot for listening for drama going on. One day I'm working and hear one of our guys talking with a client, everything was going fine until I hear him ask "Wait, aren't we in the same building? Uh, call me back later if you're still having issues."

He hung up and let us know that his caller had said her building was being evacuated because of a bomb threat, then he realized that we're all in the same building. No one had alerted us yet. We were standing there trying to figure out if we should evacuate too when I look over and see people streaming out of the fire exit just outside our office and suggested we do the same. Everything ended up fine, it was a false alarm, but one of our next projects was setting up an alert software that would notify people on their desk phones if an issue like that came up again.

This may not exactly qualify as tech support, but I'll give it a shot. I'll state I've been working in t/s since the mid-Eighties, so I used some valuable skills--mainly bullshitting* a client--in this situation.

The scene: A crowded bar--a good friend was dating the owner so I went with her to his bar to watch the Super Bowl. At the time this was one of the few bars with a big-screen TV, so the bar, while smaller and a little out-of-the-way, was a bit more crowded than usual, but not mind-numbingly overfilled like damn near any bar is on Super Bowl Sunday.

As it was, some idiot got hold of the only remote, and kept f'ing with the volume. Up, down, too loud, too soft....he just couldn't control it. The beer was flowing, and everyone was getting angrier and angrier at this guy, to the point I think someone (not me) was going to throw him out the window.

Fortunately, the time came when the beers took one of its effects and the guy put the remote down to head to the bathroom. The bar breathed a sigh of relief, when a thought came to me.

I picked up the remote, adjusted the volume to something reasonable, took out the batteries, put them in my pocket, and put the remote back down. I think the bar realized what was happening, because the chatter seemed to pick up a bit as he returned from the head.

After a few moments, he picked up the remote, pointed it at the TV, and was clearly pushing a button. It came time to bullshit him.

"PERFECT!" I yelled out after a moment or two, and he put the remote down.

Five minutes or so later, he picked it up, started pushing a button, and someone else had caught on: "PERFECT!" someone else yelled.

This went on for the rest of the game, with eventually damn near the entire bar yelling, "PERFECT!" after this guy picked up the remote. I returned the batteries to the bar owner sometime in the fourth quarter.

*By "bullshit a client" I don't mean lie to them. I mean play a little fast and loose with the truth to tell them what they want to hear, while doing what absolutely needs to be done to get the issue fixed. You know: The client says the problem is "X" and doesn't want to hear anything else, but you know it's "Y" and when they see you did something that wouldn't fix "X" you tell them it was actually "Y" but an adjusted setting at "X" will help prevent a recurrence. You know the drill: Ya bullshit 'em.

I work as an IT Tech for a local council who provides us to support the schools in the area. Been working for them a few years now and while I was based at one site, I moved to current site this year. I have been to current site before to cover a coworkers holiday and worked on an ongoing issue here.

The issue was with the Drama rooms Interactive Screen. It would only ay sound out of 1 side. As stat3d, I've been here before so I have already had a go at fixing it, as well as the 2 usual techs. At one point the boards techs were called in to look and stated they would need to take the board off the wall to troubleshoot.

Yesterday I decided to have a fresh go at it. Started going through standard troubleshooting. Snugging and hugging in cables, verifying that sound output from the board co.es out one channel. Next I started delving into the board settings, only to find Audio options were almost non-existent. Until I noticed 3 dots on the channel changer. Clicking it bought up the audio options and what do I see? Balance set to Max Right.

Really?

Yes really. I changed the balqnce setting back to middle and there we go. My weird music tastes blasting out of both speakers. All the hours wasted troubleshooting this board, Internet searches, discussion s on Teams and outside techs. In the end, all it took was a fresh set of eyes and 1 Setting change.

I am both proud of myself for fixing it and deeply ashamed for not spotting it before.

Tl;Dr I can hear, I can hear, I'm going deaf.

I work in Networking. A ticket I had earlier was involving a Network Printer. Teacher mentioned that 3 other tech people were here prior to me and they couldn't get the Printer to print wireless instead of her using the USB.

For the School District I work for. We always tried to have our Printers on the LAN Switch instead of Wireless Network just due to traffic increase across the wireless would be insane.

I get to the school. Start doing my thing. Drop was activated already. Wiring was good. I was able to get out to the internet. So I knew instantly The wireless was enabled on the Printer. Easy Peasy right.

The Teacher was looking at me while I'm configuring the Printer. Like Wow. You seem so calm while you work. Haha. I'm like well. I been doing this awhile but Got the printer switched to DHCP and all is good. After a Reboot and the right IP pulled. She was able to print wireless.

After she goes. Omg!!. You did what 3 other techs couldn't do. Thank you so much. It's the simple things you know

tl;dr: Wanna hoist yourself with your own petard? Trychmod -R 777 /var on for size!

So, it's been a while. About a decade ago, I was the technical "triage nurse" at $UberNetworks. Well...I'm still there. And I've been promoted - more or less "senior attending physician. And after nearly eight years in this role here, I'm quite astounded by the number of people who make me wonder how they even got into our field....

This is one such story.

We're setting the Wayback Machine to sometime earlier this year. Don't remember when. Not important. But it was an afternoon ticket, lands in my desk, and I take a look: customer having a peculiar error when he tries to ssh into the box, and he wants to know why.

My usual technique for support is to stare at a ticket, see if we have any diagnostics, get them as needed, and we did all this and then I went in.

First check: a particular file somewhere in /var/*/ssh/* was apparently set world everything. This is momentarily confusing to me, and being I'm a Linux nerd, I already know that permissions do not just change on their own, somebody has to do it...

...in the immortal words of the sage Lister, "Aw, smeg!"

OK, clearly somebody did this, this sounds like an indicator of compromise. /var/log...oh this needs words stronger than smeg. Like shit. Shit's a good word for this. Look at all those permissions errors. Audit logs...nothing here, OK, it's probably CLI and don't tell me he just....

Check diagnostics...hmm, ntp is broke, nobody's answering hails, better tell 'im.

Look at commands, and being that our product is built on a Linux box, we have a full Linux install on there. I wish we had emacs^(1), but that's another story and we have nano, I'll live. But this means we also have 'history'.

I look at the output of history. Lots there, let's do a simple text search for chmod...ohhhhhh, shit, no he didn't. Oh, my gods, he did....

There it was, like platform double suede - exactly what I was hoping he did not do, and my hopes dashed, 'cause there it was, like disco lemonade.

In the history, with a username that I could only identify as being the customer contact's username just by the spelling of it, I see what I was afraid of. chmod -R 777 /var.

I stared at my screen in disbelief for five minutes, so we're going to pause the tape here and fast forward.

See, I've been dealing with computers since I was a child whose dad bought the TI 99/4A as the family home computer. I've been working in this field since 2006 in some way or another, with the exception of two years of college. I've seen people who I can't help but wonder if they got their A+ as the secret toy surprise in a pack of Cracker Jack. And in all that time, I had never seen somebody make a mistake that is the same grade of mistake as some wannabe skr1pt k1ddi3 who was trying to impress other nerds with l33t sk1llz. Until that day. When this guy, for whatever reason, altered the file permissions for - quite literally - everything in a Linux install that could be found in /var.

The reason the file permissions were changed were because this guy did exactly that.

My response and conclusion was thusly passed via email. Not five minutes later, I get a response - a request to close, sent as I was informing his sales team.

And then I check his ticket history.

Come to find out, he opened another case for the exact same problem right after he requested closure of mine.

Double you. Tee. Eff. Is this guy even thinking? No, really, is this guy even thinking?

Oh, it's on like Donkey Kong, motherfucker, you do not get away with pulling this kinda mommy daddy game^(2) horseshit on my watch.

Ticket intercepted. Pulled in, advised closing as duplicate, do just that. At this point, the sales team has been contacted. Oh hey, they're still here. Teams time! Passed word as to the update since this point, he nods, and he's gonna call the guy after he and I talk on the phone a minute. At this point, I'm wondering to my sales guy as to what exactly would even possess somebody to do just that, like what makes someone think this is a good idea?

A couple days later, I checked back in with the salesperson. He still had his job at that time, but it took a lot of convincing ot get him to admit it and stop denying that we were on to him. As best as we can tell, he was apparently doing it to prove some kind of point about the security of the VM installation - by doing the exact things you do not do. But after the Crowdstrike incident and my hearing that nobody actually got canned from that debacle, I guess I'm not surprised that this guy still had a job at that point. But at this point, I can't help but wonder if he is considering prospects in the wonderful world of convenience stores, because that - in my book - is a potential career-limiting move.

^(1) Yes, I know, ed is the standard editor...

^(2) What's the mommy daddy game? Well...if you have kids, you've probably played this game with them, and perhaps to some level of amusement. If you don't, it's the game where a kid asks mom for something, and on refusal tries dad.

Hello again! Or maybe for the first time!

Some of you might remember me from such stories as Peanut Butter Jelly Drive, Interrupting Cow, and the Wireless Printer Before Time.

I've been quiet for a while, and I have a very good reason for that! I'm lazy!

Err, that's not a good reason... What was I supposed to say?

Oh! I've changed jobs and am now actually in Tech Support! For reals!

That's right: I'm now half of the one man, one woman crew that supports four US based plants, plus a Canadian warehouse, a Mexican warehouse, and a whole gaggle of remote salesmen, customer support, and a bunch of other characters from a few different countries! And I've been compiling stories... Most may only be posted after a good amount of time has passed to better abstract people and places to keep nosey nellies from making incriminating connections.

But that's ok, because I've been there for three years now and I have a few stories to tell.

So, with that introduction/reintroduction out of the way, have a recent story involving Big Blue, Old Data, and Money!

Time: August 2024

I've been with the company for nearly three and a half years at this point. It's been great: the people are generally great, the work is sometimes tough but rewarding, and I've really helped my boss out by taking over projects and most daily tasks, freeing her up to actually be able to take vacation and enjoy herself from time to time.

This was not one of those times.

Background: several years before my employment, the company I now work for was bought out by a multinational company because we were the leading producer of a very specific product catalogue that meshed well with what they produced.

The US operations kept the old name, and the tagline, "A xxxx Company" was added to show who our overlords were. In addition, our operations were forcibly catapulted into the modern era. New Cisco blades, switches, industrial switches, wifi APs, etc... were bought, installed, and configured. Old systems were removed or integrated into the new systems, including a huge migration of data into SAP.

Except for some custom software that did one thing: software that integrates the weighing scales into SAP. That software was written by a person who is No Longer Employed. And for technical reasons, that custom software was never adapted/rewritten into SAP, and calls for funding for a newer hardware system that could be brought into SAP had, heretofore, fallen on deaf ears.

You can probably guess where this is going...

One Thursday we get panicked messages that the scales aren't working. Weighing the raw product is the first step in production, so being able to correctly weigh the right product in the right mixture is very, very important.

I'm usually at the frontline of this, so I did the usual: since all the scales were affected I check the wireless connection between the scales and the system. MODAS clients are all up and working, but the scales can't get any data from SAP.

I quickly realize that this is going to be something I haven't handled yet, so I contact my boss. She's several hours away, camping, trying to enjoy some time off. Ha. Ha. Ha.

I tell her what is going on and what I've done. She checks a few things... The IBM server that hosts the scales program, as well as the old Power8, and some old data that hasn't been migrated, isn't responding. So she makes her way back.

Again, and I cannot put too find a point on this: not being able to weigh the raw product means that production can't happen. Once production runs out of already mixed raw product, no more product can be made. There's other work: finishing, shipping, etc..., but moving all of press to finishing will be tough.

To make the story progress faster, here are some highlights from This Comedy of Errors:

• My boss does a 4 hour call with our contracted third party support for the IBM server. Together, they decide that the disk backplane has gone out.
• By contract, the third party support had to have a tech dispatched with the replacement part within 4 hours of the ticket being created.
• We work on a yearly contract that is paid once a year.
• My boss rec'ed and submitted the bill 30 days before the bill was due.
• Someone in billing changed the terms to 90 days.
• The IBM server backplane went out FIVE DAYS after the contract ended.
• By this time, IBM support was closed so we had to wait until Friday.
• Even if we could get the third party paid, it would have been late into the following week before they would work with us.
• We had to buy an out-of-warranty support contract with IBM, but they wouldn't be able to send a tech until the following Monday.
• They ship the replacement part and it is rec'ed Monday morning. I note that it isn't a backplane, but the entire system board sans backplane.
• IBM tech comes out Monday late morning. He pokes around for a while and makes a long call to the IBM Support Line and comes to the conclusion that the backplane went out (which is the conclusion we came to earlier with IBM AND the third party support), but because they sent out the system board and not the backplane he couldn't help.
• IBM Tech found the right part and we had it overnighted to be delivered first thing Tuesday morning.
• After talking with the IBM tech, I discovered he had to drive several hours to our location. I quickly ask if we can pay for a hotel room so he didn't have to drive home and back the next morning. "Oh, I can't come back Tuesday as my daughter is having her tonsils removed. I should be back Wednesday or Thursday."
• My boss' heart nearly explodes. This means we would have been almost a week with no mix being made. Very Important People with Very Interesting Accents are asking questions and not liking the answers.
• Fortunately IBM says they can send someone else on Tuesday.
• Tuesday comes and the backplane is replaced, but the server isn't responding.
• I tell the new IBM Tech that I'm pretty sure the previous one put the server in Manual Mode. "Huh, look at that, he did put it in Manual Mode." My boss' heart rate slightly drops.
• It is finally up and running, but degraded because... a Hard Drive Has Died!
• Another call to IBM is made, and they send out a replacement drive, and a Tech should be on site by Friday to replace the drive.
• Drive arrives. Tech arrives. Tech drops replacement Hard Drive onto floor. Installs it anyway.
• System is back to purring like a very loud kitten. For now.
• And then most of the old scales died anyway...

Oh, at least now the funding for a new scales system has been approved!

Have a great day!

I feel like I'm losing my mind.

A client of the MSP I work at recently contracted an external marketing AI Driven personalized email sales generation firm. They send bulk template emails to a list of potential customers and try to convince them to buy something. But they're not marketing, and will correct you every time you so much as insinuate they are.

Whatever. Not the issue I have with them. Because rather than send mail from their own infrastructure or a dedicated bulk sending service, they apparently require a standard licensed user mailbox to send spam generate personalized sales leads.

We warn them that this won't fly, that account is going to get blocked within 24 hours, and that the client runs the risk of having their entire domain blacklisted. Marketing company says it's fine, they've done this with hundreds of clients, including on the Fortune 500. Client says do it, boss says the inevitable stupid tax will be a good source of revenue, us techs are just paid to push buttons so we create them their account.

Twenty four hours pass. Security alert hits the queue, marketing.bozos@clientdomain.com has been restricted from sending out of 365 due to suspect outbound messages. Checking into it...the account was sending out standard boilerplate spam. We have a moment of 'I told you so,' get affected parties together, reiterate that this won't fly and recommend that they do what we told them to in the first place.

No, says the marketing company. This happens all the time. 365 just needs some time to adjust to their sending patterns. They "mimic human behavior" after all. But, we should create them a second marketing account so they can split their sends between them. This will totally fix it, promise. Argument ensues, but at the end of it the second account is created.

Twenty four hours pass. Two security alerts hit the queue. marketing.bozos@clientdomain.com and marketing.bozos2@clientdomain.com have been restricted from sending out of 365 due to suspect outbound messages. Both accounts were sending out standard spam. The 'I told you so' is said with a sigh today. We again recommend they do what they're supposed to.

No, says the marketing company. This has been happing increasingly often. What we really need is a third marketing account so they can be super absolutely sure this doesn't happen again, super duper pinkie promise. The ensuing argument has more tension this time around. A third account is created at the client's insistence.

Twenty four hours pass. Three security alerts hit the queue. marketing.bozos@clientdomain.com, marketing.bozos2@clientdomain.com, and marketing.bozos3@clientdomain.com have been restricted from sending out of 365 due to suspect outbound messages. All three accounts are sending out standard spam. The 'I told you so' is said through gritted teeth. Boss finally puts his foot down, says that we are not going to be creating an infinite series of licensed marketing user accounts. You are going to need to find both a new IT provider and a new domain at the current rate. Argument ensues, further spam sales generation sends are paused until a resolution can be reached. A meeting is scheduled.

The meeting happens, between myself, one of our senior techs/technical executive, stakeholders at the client, and the non-technical account manager from the marketing company. Account manager insists on giving us the sales pitch for their company. "We send bulk template emails to a list of potential customers and try to convince them to buy something" says the account manager in her native tongue of corporate buzzword slop. Great. Amazing. Tell us what shitty bulk sending platform you use and the spf record you want to us add and we can be done with this.

No no no, says the account manager. It's not our business process to use those. We prefer a personalized approach. You see, we mimic realistic human behavior. Our weird proprietary tool that we've grafted to this poor mailbox sends a message once exactly every 120 seconds - just like a human! We personalize our messages by using the same subject line every single time! These are not standard marketing messages, they're an AI driven, personalized sales generation platform. Transcendent. Enrapturing. You're sending spam. You're going to get the client blacklisted. I refuse to believe that we are the first people to have pointed this out to you.

Well, the account manager admits, we have been noticing these issues recently. Since last month, apparently. But we're totally 100% certain that if we just keep at it, 365 will give up eventually! We tell the client this is untenable, unsupportable, and poses a serious risk to their business operations. Marketing company refuses to budge. It is eventually 'agreed' to buy a clientdomainmarketing.com, use it to create a seperate 365 environment, and let marketing company go wild without risk of contaminating the primary domain's reputation.

Am I crazy? Does this sound like anything remotely reasonable? I feel like I'm going insane.

Years ago, had a college student bring by his laptop for repairs. Keyboard stopped working, according to him, and he had no idea what the cause could be.

After he left, I quickly surmised that someone spilled a sugary beverage on it, so I contacted tech support for the model (let’s say it was HP) and they quickly place an order for a replacement part. During the call, support also mentioned that a previous support call was made on this laptop for, you guessed it, a spilled soft drink. Noting that information, I proceeded with the order and, when the part arrived, swapped out the keyboard.

After verifying that the laptop was functioning properly, contacted customer to pick it up. I left it running on the repair table and moved on to other tickets. The following morning, I noticed that the screen was blank and decided to tap the keys to awaken it. Nothing happened. Listened and could hear cooling fan running, so I cycled the power. Powers back on, except the screen is still blank.

Reached out to the customer to tell them the situation and see how they wished to proceed. Here is when dad, a local attorney and expert radio/TV commentator, gets involved. He starts cussing at me and threatening me with a lawsuit if I don’t replace/repair his son’s computer. I calmly inform him that, no, I will do no such thing for a previously damaged computer. Incredulous, he accuses me of lying about previous damage to cover my ass for negligence. That’s when I inform him of the conversation I had with HP.

Now, I had him dead to rights, but this is where I was surprised. After his brain audibly glitches, he says, “wtf are you talking about HP? My son owns a Dell.” My response was that clearly there is some misunderstanding here on your part because I’m looking at an HP, not a Dell.

No apologies, nothing comes from Mr. Attorney. Instead he sends the kid to come get the laptop and pay the bill. I had to know what the hell just happened, so, when the kid shows up, I ask. He sheepishly admits that he had his frat bro’s laptop, because frat bro had broken the kid’s Dell laptop and given the kid his HP laptop. Guess frat bro never mentioned spilling a coke on the HP and this kid figured his parents would be none the wiser.

To this day, Mr. Attorney is on TV/radio to offer his opinions on whatever legal case is in the news, and I chuckle every time I see or hear him.

Cast your mind back to 1995 when Windows 95 became a thing. This is set in late 95 or early 96 when new computers came exclusively with Win 95.

I had a customer who had an existing system for inventory, POS and accounting. It was a DOS based system and was written in Visual Basic. Not Studio or .net. Just visual Basic.

So customer wanted to upgrade computer and existing system to latest version. I was not a reseller for the system, but they agreed to sell a new version to me. I asked them if it worked under Win 95. They assured me 100% it worked.

I assumed they had tested it under Win 95 after such a definitive statement. I was wrong they lied.

First problem was they wanted customers details before they sold it to me. I was suspicious, so i gave them his name and my fax and phone number.

So then first problem was all the data didn't transfer correctly. I rang them and they asked for a copy of the system. They stated that Windows backup onto floppy discs would be sufficient. So backup was done and airbag to remote city.

Clue 1:

A day after they received it I received a fax addressed to customer with page after page of error messages and a suggestion that customer contact their Authorised Dealer locally. I rang them and asked politely what restore version they used. They said DOS 6.2.

We all know why they got errors don't we? Versions of backup and restore were not compatible between versions of DOS. I asked less politely why they were attempting to throw me under the bus when the mistake was theirs alone. The answer was not really satisfying. We resolved the restore problem and then they sent a new version of the transfer program which did transfer correctly.

Then the customer attempted to do the end of month on the new system. He sold things and at the end of every month he ran a summary report of everything he had ever stocked with on hand and sales per month columns. He then looked at it on the screen. On his old system it worked fine. On his new system it gave "Out of memory" errors before it finished the report. It did this even when I quit Win 95 and the underlying DOS system showed 640K free.

Clue 2:

I contacted the supplier of this program and the help person told me I had to run memmaker on the system to allow enough RAM for the report to run. Evidently every report ran in memory and had no spooling capability. I advised them that this was a brand new Win 95 system and as such did not have an autoexec.bat or config.sys. It also did not use (or need) memmaker.

The help person told me "Trust me, you just need to run memmaker" I asked them if they had run this program on Win 95 and it turned out that they had not. In fact they had a copy of Win 95, but had not installed it. They had plans to install it on one of their own computers at home. As well they had no access to the developers who existed in another country. They had no idea how to fix it or even to go about finding anything to do.

I realised I had been lied to by lying liars who lie. (pants on fire). I had anger issues in those days, especially when people lied to me. I gave them a roasting for being idiots. Unfortunately I did this in the customer's shop and I'm sure he heard me call them liars (and worse).

I told him that there was no way for me to make his system work for his report without some limit being placed on the number of items selected. In perfect hindsight perhaps some limit on vcache in system.ini may have helped, but I had no knowledge of this. In those days there was not the plethora of websites available with all the knowledge anyone could want.

I was never invited back to his shop, and I found out later that the local Authorised Dealer for that program took over that customer. I have no knowledge if they ever fixed that (unpolished) piece of .... I decided I wanted nothing more to do with them.

In hindsight perhaps I could have questioned them more about their blanket statement that it 100% ran on Win 95. Perhaps I could have been more tactful when talking to them on the phone. Part of the problem was that the customer was an hours drive from me (and an hour back) and that the main supplier of this program was 2000 Kms away from both of us. I was calling them on my mobile phone which cost me lots of money in those days and I had other customers who needed me more.

Story 1: My job is very broad, working on industrial processing equipment, servers, and networking. A lot of this equipment runs on windows xp and NT. The operator software is locked so they can't minimize or close it but every time a warning pops up they try to click outside of it instead just pressing enter to close it. So then we get a call saying the keyboard and mouse doesn't work when all they needed to do was use the mouse to click back into the pop up to close it.

Story 2: The bad batch- So we replaced like 80% of our computers that run the Operator software a couple years ago. A part of the upgrade was to replace the old ps2 mechanical keyboards with new usb membrane ones. Recently one started going bad. Only the control key which we use to switch computers with the kvm. So a tech went and got a new one and the keyboard wouldn't work with the machine but would with a normal computer. So they got another and same thing. So I got called to look into the problem. The new keyboards were the same model as the old one so they should just work but no dice. After testing around 20 keyboards on multiple machines I found out all the ones not working had sequential serial numbers. I had only one that was not made in that batch and it worked. All of them worked with a normal pc but not the machinery. I couldn't find anything from the manufacturer saying this but there's different keyboard communication standards not all kvms support multiple standards so they couldn't communicate through the kvm.

Story 3: How did you do that? So i got a call because an operator somehow managed to press all the right keys to switch computers with the kvm but they didn't know how. Easy fix to get back to theirs but they really wanted to know how to get back. Basic IT is to not tell them how to access something they shouldn't touch so I didn't tell them.

Story 4: The lazy tier 3 support: So some of you might be familiar with a network monitoring tool called Nagios. It tells you the status of your network equipment and being a company with over 200 sites it's one of the few reasonable ways to monitor that much networking hardware. So the intranet website for my site stopped working and I didn't know where the hardware at my site hosting it was, but even if I did, I didn't think I should touch it. So I opened a support ticket. Then they said they didn't know how to fix it and closed it. Same result with 2 more tickets. I had to go to our training facility in another state, and while I was there, I asked my previous networking teacher what the problem was, and they didn't know. So, while in a stupid comptia class that i was taking for the heck of it, i started researching the server that hosts the software. Most security conscious companies, including mine, have a rule that shared logins are stored in a password vault that only shows you the logins required for your job and, if possible, have personal logins/SSO. For the heck of it, while in another state, i pinged the server and got a reply. Then i tried ssh and got a login prompt. So i tried my SSO so that works on the other servers and cisco Switch, no dice. So i checked the vault, and there was no login. Well, the login for this server hosting the software for some reason was just listed in our KB that anyone can access, and I was able to get it. Saw the uptime was a couple of months, and no one bothered to reboot when I opened a ticket. One reboot later, and now I can access the website. Probably shouldn't have the login and ssh open, though, to all 600k employees.

Sorry, these aren't action-packed, but hopefully, they were interesting.

As the only person in my family who works in IT, I'm permanently on call as my parents' tech support despite living halfway across the country. My dad usually calls about a server he wants to set up, and my mum usually calls about basic "how do I do this on Excel" questions.

The other day, dad called me saying he's thinking of upgrading the graphics card in his main PC. This machine is about 13 years old at least and has integrated graphics only, and about 4GB of DDR3. he says it struggles to run Photoshop which....yeah. I bet it does.

After a bit of a chat he says he wants to upgrade the CPU as well. He says he has an early generation i5, and wants to upgrade to a late gen i9.

Without upgrading the motherboard.

At this point I'm thinking, just get parts for a whole new PC at this point, but he's adamant that it would be cheaper to just swap the GPU and CPU. Oh dad, I wish it worked like that!

I went through with him all the reasons he can't just pop a new i9 in there and be done, in particular the fact that it literally won't fit in the socket. Eventually, he begrudgingly accepted the PC was no longer fit for purpose, and I offered to find the parts for him and give an idea of the price.

It struck me later that night that my dad taught me how to build computers. He and I built my XP machine which I had all the way up until 2013! He built every computer in his office and at our house. But technology has moved on, and left him behind. So now I get to be the teacher :)

Characters:-

Customer: Owner of business, payer of bills

Me: OP

Doctor FW: A specialist of some fame in a small town. Travelled to foreign countries fixing people.

Back in the late 1980s I had my first customer. He was using a program to run Autocad to do drawings for clients. He would put in the dimensions and the program would print a list of components and then do a drawing automatically and print it. To make it work you needed to create an autoexec.bat and config.sys that not only had lots of buffers= and files=, but also loaded device drivers in high memory.

Being Dos 3.3 it needed to be QEMM and after using OPTIMIZE (supplied with QEMM) you then needed to adjust each memory segment manually to get the best results. Of course being somewhat paranoid I created a "Backconf" directory and two batch files saveconf.bat and rest.bat. One I used as I altered config,sys and autoexec.bat files, and the other I had as insurance.

Then came the phone call.

Customer: "Doctor FW was visiting and while he was looking at my computer he told me I didn't need files=50 and buffers=45. He reset them to 20 each and now Autocad doesn't work. I need it to work first thing tomorrow. Is there any way you can come fix it for me?"

Me: "Tell Doctor FW not to muck with my setups and type this "Rest"

Customer: "It says '2 files copied'"

Me: "Now turn it off and turn it on again."

Customer: "Oh I hope this works."

Me: "Now it's fixed."

Customer: "Let me try it" (Printer sounds in the background) "It's working! You're a genius! What do I owe you?"

Me: "Today it's free, Next time it will be lots."

Doctor FW never touched that computer again.

Way back I was working on the service desk for a large organization who almost exclusively used Dell for their end user hardware.

On a fairly quiet day I get a call.

Dell: "Hi, this is [name] from Dell, have I come through to [company name]'s IT department?"

Me: "Yes, this is the service desk at [company name]"

Dell: "I was just calling to see, how often do you refresh your hardware, specifically monitors?"

At this point, I'm pretty sure it's a sales call, but it's fairly quiet, and if I am on the phone, I can't get another call, so I play along

Me: "Usually 4 years, but can be more or less than that"

Dell: "Ahh ok. So you wouldn't dispose of one after say 2 or 3 months?"

Me: "Very unlikely"

Dell: "And what do you do with your disposed IT equipment?"

Me: "We use a computer recycler who collects it. I don't know what they do with it after that"

Dell: "Hmm. So the reason I'm asking is someone has made a warranty claim for a faulty Dell screen.

When I ran the service tag (serial number) through our system, I can see we sold it in a bulk order to [company name] about 3 months ago. Looks like you also purchased a few extra years warranty on it too.

The person who put through the claim mentions they purchased it new on eBay about a month ago"

Me: "huh. Yeh that is a bit strange."

Dell: "Yeh, just wanted to see what is happening as it does seem a little out of the ordinary you would dispose of a screen so soon, especially with the extra warranty"

Me: "If you can give me the service tag, I'll check our CMDB. That should confirm if it has been retired or not"

(I get and run the service tag in our CMDB)

Me: "Yep it's showing as it's an active asset (i.e. not disposed of), we got it about 3 months ago and it should be in our IT store room ready for deployment right now.

What I'll do is log a ticket, noting the service tag with the team that handles purchases and find out what happened"

I log the ticket, we exchange references numbers, and end the call. Then I basically just forget about it.

A week or so later, an immediate termination request is put through for one of the other IT guys. We were told he no longer works for our company and he left very suddenly without explanation.

Later on, I find out through the grape vine he was fired for theft of company property.

Basically, he stole a new Dell monitor from the IT storeroom that was intended for stock on hand, and sold it as new on eBay.

The monitor had a fault, and the purchaser on eBay logged a warranty claim directly with Dell, using the eBay purchase record as her proof of purchase.

The seller's eBay account belonged to person who stole the monitor, and that's how he got caught.

Back in the 90s, during the Dialup Era when dinosaurs (486s) still roamed the earth and line noise ate your downloads for dinner, I was working for a local ISP. I was recently promoted out of support into jr. sysadmin, but I was still the person they went to for "problem calls." And I actually enjoyed that. Some guy with a Commodore that was having trouble dialing in? Sure, I'll help. That OS/2 user? I used to use OS/2, I can help. Linux? I use that at home, I'll help. It was fun. [1]

But not this call.

One fine morning the sunlight was streaming in the window, I was sitting in my office[2], and a support person (SP) walked in my door, saying "I've got a problem call on hold. Can you help?"

That was typical. But what was odd was SP's demeanor: his tone of voice was pleading, he looked actually afraid that I might say no.

I asked, "What's going on?"

"She says all her pictures have been turned into horses."

Pause. My brain was having trouble with that sentence.

"Uhhh, what?"

"Yeah. She says all her pictures are now horses."

"What pictures?"

"I don't know. She's frantic, mad, and clueless. She can't even explain. Please help?"

"OK, sure."

SP departed at a much higher velocity than usual for a person that was about to return to his office and take more support calls.

I picked up the call. The customer, who I'll call HL for reasons that will become clear, was indeed frantic, mad, and not particularly computer-literate.

"Hi, this is Universal_Binary, how can I help?"

"I've been hacked! Your system is terrible! How could you let someone turn all my pictures into horses?"

After much discussion, I determined that the photos were on her website. Like most ISPs at the time, ours offered each customer a few MBs of disk space (which was plenty to host a website at the time). HL had somehow managed to figure out how to put up a website, and I pulled it up.

It looked like a run-of-the-mill amateur website at the time, and indeed all photos on the site were of horses. Incongrous horses. Instead of whatever was supposed to be there -- navigation icons, a map, etc -- EVERYTHING was now a horse (or more). I had to mute myself when I saw it come up on screen or the customer would have heard my laughter. Nothing on the site had anything to do with horses, and yet there it was -- full of horses.

I looked into it more. Nothing had been recently modified. It turned out that she didn't have any pictures in her public_html directory at all. Every image was coming from a differnt server by using its URL in her IMG SRC= tags. In other words, she was basically stealing photos & bandwidth from someone else.

I suspected that person found out and replaced all their images with horses[3], but maybe they just took a random turn for the equine.

In any case, despite my attempts, it was impossible to get Horse Lady to understand that she had not been hacked. Or how IMG tags work. Or even that she was mooching off someone else, and that what is behind a given URL that she doesn't control might change at any time.

Finally I said, "OK, let me ask the company owner to look into it and make sure you weren't hacked. OK?"

She sounded relieved. "Finally!"

Now it was my turn to go to an office. I went to my boss's office (who happened to be one of the owners of the company), stood in his doorway with that same pleading tone of voice, and:

"I have a mad customer on the line, and she is sure she has been hacked. I don't think she has, but the only thing that will make her happy is knowing you've double-checked." I explained the saga, watching him try -- and fail -- to contain the smile that grew into a chuckle.

"Who is this customer?"

"HL."

Now it wasn't a chuckle; it was outright laughter.

Without turning to look at his screen or touch his keyboard, he said, "Tell her I've checked and her account is secure."

"OK, thanks."

I backed out, told this to HL, and it somehow pacified her a bit and we ended the call.

Boss's office was right next to mine, so occasionally we could hear each other's conversations. I heard several conversations from his office that day that went like this:

"Universal_Binary came to me today to ask of a customer account had been hacked. Apparently all her photos changed to horses."

"What? Horses? Had it been hacked?"

"Of course not."

"Then what happened?"

"The customer was HL."

"Ahh, Hahahahahaha!"

Apparently I was one of the few that had never had a run-in with HL before. But I still remember it, nearly 3 decades later.

[1] Clearly I hadn't been doing support long enough then yet. This call was one that helped cure me of that.

[2] This was the 90s; the pay was bad, but even though I was a part-time jr. sysadmin, I had an office with a window, desk, a couple of visitor chairs, and a door that could close.

[3] Yeah, the 90s was a different era. I'm sure it would have been a lot worse than horses if someone had tried that today.

If my main responsibilities run dry (running the IT side of in-house events) I'm expected to assist with our IT Hotline. We have two infrastructures: one for internal (desktops) and one external (laptops) use, where external video conferences are allowed. To connect to the external environment, the user needs to start a remote tool and enter a code I provide.

Today I had following fun call:

User: I have an urgent problem. I can't hear anything in a external video conference. I had the same issue yesterday.
Me: Oh, you should’ve called us earlier—usually the sooner, the better. (some conversation while I start my external remote tool and login there)
User: Well, I was busy. And just joined from a coworker yesterday. Now I urgently need to hear in this call. Can you come over?
Me: Not really, I'm in Home Office today. But no worries, I will remotely connect. Please start Program X.
User: I can’t find Program X.
Me: But you’re outside of Citrix, right?
User: Yes, I am. On my desktop.
Me: Hmm... Wait a second. Desktop? Not on your laptop?
User: Yes, I can’t find the program on the desktop.
Me: But the video conference with the problem is running on your laptop, right?
User: Yes.
Me: ...then please try searching for Program X on the laptop.
User: I found it!
Me: *remoting in* Ah! You seem to have an headset connected and the sound is routed there.
User: Oh yeah. I didn't want to use them.
Me: *switches settings, sound starts in the background* There you go, I set everything to your laptop audio devices.

Ah, the good ol' helpdesk days... relaxing calls, full of small riddles, and always good for a laugh. What more could you want? *sips coffee* xD

Years ago while doing tech support at a school, I helped a teenager with an issue on his laptop. His assignment was due that day, but the file was corrupted, so his teacher sent him over to the helpdesk to get it sorted out.

I tried to open the file in Word, no dice. I renamed the file to .zip (because .docx files are just zip files with the contents inside), still no dice. I opened the file in Notepad to view the raw contents, and in the header, I saw the letters "PNG", so I renamed assignment.docx to assignment.png.

Staring back at me, was the kid's name, scrawled in his own handwriting using the tiny netbook touchpad, in orange. I turned the laptop around and said "your document was actually a picture with your name written on it. You'll need to actually do the assignment instead of lying to your teacher".

The kid then said to me "I didn't do that. The computer must have done that because I didn't. I just did my assignment and next time I opened the document, it wouldn't open!"

I said "so the computer wrote your name, in your handwriting, in this particular shade of orange, and renamed it to a Word document, overwriting your already completed assignment?". They shrugged and said "yeah", so I said "here's your laptop, head back to class and start working on your assignment, I'll let your teacher know"

Many many moons ago I worked at a call center providing email support for a popular VR company. We'd just released the first version of a standalone headset that didn't need to be connected to a PC.

Our site was email and chat support only. Phone support cost a metric buttload of extra money the client wasn't willing to shell out, so we were very locked into our role. Phone calls are a no-no.

We had one very determined, difficult, and technically dis-inclined user just could not figure out our email and chat system.

She would chat in, disconnect herself, and then start a new chat immediately. She sent in email after email, but she could never seem to figure out which of our emails to respond to, or how to keep a chat session open.

Over the span of three days, we received 115 tickets from her (one ticket per chat, or email) and she only managed to respond to one email:

"Please call me this is for my son 555-123-4567"

As much as I hate talking to people, I hate closing 115 tickets by hand (we weren't allowed to use the bulk operations in ZenDesk...), so I work my way up the chain asking my boss, operations manager, and site director if we can just call this lady.

Boss: No. Client doesn't pay us for phone calls, and your utilization is only at 79% get back to work fuckface.

Operations Manager: No. Client doesn't pay us for phone calls and we don't want to devalue our labour by providing a paid service for free.

Site Director: Love the attitude! Synergistic thinking! Really outside the box! No.

Me: Pretty please?

Site Director: (big sigh) Okay, let me make some calls.

So they call the client, who LOVES the idea and approves it as a one off, and we borrow a phone from another contract so I can make the call.

So I call this lady with the phone number she gave us, and she was the sweetest grandmotherly type you'll ever meet.

It was an awkward call (email support means a quiet floor - my coworkers could hear every dumb thing I said) but it was worth it!

She told me that her son is heavily autistic and he's almost entirely non-verbal. But she told me that he thrived in VR - he could actually look people in the "eye". She bought a headset because she wanted to spend time with her son in an environment where he felt comfortable and she was DETERMINED to get her headset working. She hates technology, but she loves her son more. How do you say no to that?

We spent a full two hours on the phone just explaining the basics - how to turn on the headset, how to put it on comfortably without slipping, how to buy an app, how to connect to the internet, how to add a friend, how to invite each other to a game, how to tell which games support multiplayer, how to reset your password when you forget it for the fourth time on our phone call... and while we were at it we went over how to reply to an email and keep a frickin' chat session open

She asked me to pause many times so she could write out notes. She got up to six pages of notes by the time we were done. We ended our troubleshooting with her sending her son a friend request.

She asked me my name, and I gave her my support alias: "Bartholomew" (from the Bandy Papers by Donald Jack - good book series!)

She says to me "You're my guardian angel, Bartholomew. When you're in Montana you look me up, okay? Save my phone number, I mean it. You've always got a place here."

I didn't save her phone number, and she never created another ticket. It's been five years and I still think about her often. We'll never meet again and I'll never know how it all played out, but I hope with all my heart that her and her son are still hanging out in VR.
.

.

.

.

.

.

.

.

.

(Three months later the client requested we start offering phone support too, which my coworkers absolutely loved and didn't blame me for at all)