This is going to be a rather short-ish one.

A few weeks ago I had someone come in and complain that they couldn't log in to a web-based system that we use. It's a niche but still off the shelf product, and let's call it SimpleSoft. Each instance of SimpleSoft is branded to whatever organisation is using it, and a fair number of organisations use it.

I grab their laptop and take a look, and they're trying to log into another organisation's instance of SimpleSoft. It's quite obvious, as not only is the logo on the page not our logo, but it's also not our organisation's colour scheme, and the banner along the top doesn't say our organisation name.

I tell them "that's not the login page for our SimpleSoft" and they said "but that's the first page that came up when I searched for it". I point out the wrong logo, colour scheme and name, then proceed to show them how to get to the login screen for ours. I give them two ways: first is just to go to simplesoft.example.com, second is to go to our website, example.com, and click on "Log in to SimpleSoft", and I got to our website by googling our organisation's name as a kind of "if you can't remember our address, just google it". That way, if they can't remember the URL, they can remember "ah, I can find it on the website"

Anyway, we get them logged in and they walk off, happy.

About a week later, they show up to the service desk again. "I still can't log in to SimpleSoft". I spin their laptop around and.. it's the same incorrect login page. Wrong logo, colour scheme and organisation name. I point this out again and they say "well that's the only one that shows up when I search for it".

I remind them that they need to access SimpleSoft either via the direct URL of simplesoft.example.com, or by searching for our organisation and clicking on the "Login to SimpleSoft" link. They walked off happy again, and I think this time it stuck, as I've seen them at the service desk, but I haven't seen them for that specific issue.

I don't know what it takes for someone to reach a page, enter in their credentials, and not notice that the page says "Other Company Inc." instead of "Example Inc.", that the logo doesn't match the one on their uniform, and that the red doesn't match the blue we use in our logo, but it sure was a facepalm moment.

A somewhat unpleasant user, whom I shall call Richard, wanted two ergonomic mice. He created two tickets, one for work and one for home.

In both tickets, Richard wrote the item number for the mice. As in, the ID number assigned to the product by the manufacturer. As in, if you search for this number on any search engine or on any resellers website, this specific product comes up every single time.

So, Richard specifically asked for these specific mice, twice, in two separate tickets. So, I ordered them. He was away when they arrived, so I put them on his desk.

Half an hour later, he emailed me.

Richard: These are not the mice I ordered! Order the correct mice, like I wrote in the tickets!

Me: Hi Richard. These mice match the item number you entered in both your tickets. Did you want a different model? *Attached: Screenshot of Richard's tickets and suppliers website, showing the item numbers next to each other*

Richard: *No response*

The next day.

Me: Hello Richard. I can order some different mice and we can swap. How about that?

Richard: *No response*

The day after that.

Me: Hello again, Richard. Are you fine with using the mice I gave you? We can still swap, if you'd like.

Richard: *No response*

And the day after that.

Me: I'm closing these tickets for inactivity. Create a new ticket if you would like a different model.

And he never mentioned those mice again. I later had some other task close to Richard's office. As I walked past it, I noticed that he had unpacked one of the mice and plugged it into his computer.

The mice have a soft armrest palmrest, covered in a thin layer of imitation leather, for comfort. He had been using this mouse for less than two weeks and had already managed to tear a fingernail sized patch out of the leather, exposing the foam underneath.

help urgent someone took over my pc

i know the title is clickbaity but here is the story. i am not a sysadmin. we are a small company in france and i do most of the usual IT stuff. we are only two developers, well we recently hired a third intern. so this girl has very little background in software development

. she did some bootcamps in programming but shes a quick learner. anyways i am working on my desk and she came running to me that she has some emergency. she does some javascript programming to give you some context. i go there and she says someone has taken over her pc. i am like ok....... then she shows me her cursor moving automatically but not in a random way like how someone would actually use a mouse. i am like what, thats not normal

. i immediately disable network interfaces wifi and eth. she tells me she was running some js code and it started happening. my mind goes ok....m maybe she downloaded some compromised library. well i scanned with windows defender nothing comes up. hmmmm. lets try malware bytes. i download malware bytes on my pc. copy it to her pc, install, scan. nothing comes up. suddenly keyboard keys start getting pressed, windows search bar popping up randomly. i am like this pc is definitely infected and i will need to reinstall everything. not something i really want to do. ..

. i am like lets try disabling hardware i disconnect keyboard and mouse connected via bluetooth. still no change. i notice a usb dongle plugged in a usb port and remove it, voila it stopps happening. meanwhile her colleague on the side desk is going crazy why his mouse and keyboard are not working and he is fanaticalky pressing buttons on the keyboard and cliking mouse. they switched usb dongles for their mouse and keyboard.. 🤣

I work in technical escalations for a Fortune 500 technology company. Had a ticket come in from our L2 team for an older gentleman, who had recently purchased a fully kitted out Ruggedized laptop for personal use… I’m talking about something the military, police, fire departments would use. For the sake of this story we will call him Lou.

After reviewing the case details, I give Lou a call to help get the machine setup. His primary concern is that he wants to get GPS up and running, and none of the other technicians could get it working.

This laptop comes fitted with both a WWAN module and dedicated GPS module, so I configured the GNSS drivers, set the correct baud rate, and explained that if Lou wanted to use the dedicated GPS module, it would require line of sight. So he would either need get close to a window or step outside to establish connection. I go through the steps of testing in PuTTy and confirming we are getting a good data stream, then validate that everything is working as expected by checking the native software specific to the GPS. We can see a fix in the satellites, and LAT/LONG data is populating correctly.

Assuming we have wrapped up the issue, I asked Lou if he needed anything else, and he responded with “well you still haven’t gotten the GPS to work”.

Confused by this, I asked Lou what he meant, as the GPS coordinate data was clearly coming in and working with all apps we tested. Lou responded with “I still cannot see the live view, the whole reason I bought this computer was so I could see a live satellite view of my neighborhood and keep watch”.

Now I understood. Lou has purchased a $6000 specialized laptop, expecting that he would have some clandestine spy capability to keep overwatch on his neighborhood.

I then had the unfortunate task of crushing an old man’s dreams of having the most advanced neighborhood watch in North America…

TLDR: Grandpa thought that having a GPS module meant he could utilize satellites to keep watch over his neighborhood in real time.

So I was operating the in-person IT helpdesk at Megacorp HQ today. Among many... 'fine' examples of competence, this one stood out. Newbie comes in, starts asking newbie questions. So far, so good, noobies do be like that. Then, (conversation paraphrased)

"Okay, next, I want to request a locker for myself. How do I do that?"

"*shrugs* I dunno. That does not sound like something IT would be in charge of."

"...I thought this is a general helpdesk for everything though?"

"No, this is an IT helpdesk for IT problems that require personal contact. Hardware handover, phone help, that sort of thing."

"Oh, I see. I would like to keep my laptop in a locker, so how do I get one?"

​

There was not even a hint of humor, he honestly seemed to think he pulled out an ace, and was surprised that it did not work.

Currently in virtual training from home but I'm responding to emails depending on the amount of work required.

User sends the usual email requesting a feature they already have and suggesting a solution that defies every sensible security convention known to the modern society.

I respond with: 'before I go digging too far into this issue are you doing this blah blah blah? If not here's a doc that will help. Otherwise you will need to submit a ticket so we can document the process."

The document is a PDF of how to send as a shared mailbox. Page 1, outlook desktop with 2 sentences of what to do and 1 pic. Page 2, outlook web with 2 pics and 1 sentence each of what to do.

So here we are at the end of a 3 email thread where so far I have included screenshots of the document with "just so you know, you have to open a new email first then everything else should be fine." Or so it thought ... It seems putting your hope in these users is as pointless as their doctorates and certifications.

I think I will leave this one to simmer a little... See if anything happens on its own. Perhaps they'll get an email a couple weeks from now asking if they still have that issue. Or truthfully, maybe not, there was no ticket after all.

The main endpoint security system for the MSP I work for is basically a host-IDS powered by machine learning. Let's call this system "MIDS".

In October someone suspects that there are workstations without MIDS and set up SNMP alerts. Over the course of the day we find probably 300 machines (out of 2k) that don't have MIDS. Since my long term interests are in security, I volunteer to fix this.

Our RMM lets me upload files (such as the MIDS installer) and run shellcode (such as the install command) without bothering the user.

Keep that workflow in mind, it will be important.

After about a week I've gotten most of them taken care of, but there are some that the install process fails on. Through some digging I realize that these have some of the services and some of the registry keys, but not all of them.

So I email the vendor. They explain that it looks like these are failed installs or failed updates. Why would this happen? I will ask months later and the answer is basically 🤷

MIDS isn't in the control panel, but the vendor shows me how to uninstall it:

• Option 1: The server's web console. But, if the install is broken, it probably isn't talking with the server, right?
• Option 2: A shell command that requires an uninstall password. The command may not work, possibly because the password hash on the endpoint is corrupted
• Option 3: Go into the advanced boot menu, delete the services, delete some stuff from C:\Program Files, delete some stuff from C:\ProgramData, reboot, delete some registry keys (huge pain to delete that many keys from the command line), reboot, and now you can install it. Except, sometimes there's no command line option in the advanced boot menu, and sometimes when you navigate to the C: drive it just doesn't believe that there are any directories. Then you... have to reimage? I haven't tried to figure an alternative yet.

There's also the forced update tool, but I've never gotten it to work, so I'm not going to count it.

Earlier this year I was doing something on a server and realized it had a failed MIDS install. In a rage I spend a day going through the list of computers in the MIDS web console and the RMM and find another TWO HUNDRED devices that have some kind of problem.

Turns out, the monitoring we setup was only on workstations, not servers. And was based on the presence of a particular file that is deleted then remade during the update process. So if there's a failed upgrade, the alert is triggered. But if the upgrade just never starts (or fails really early or really late in the process), no alert trigger.

Also, the name in the MIDS server is the name the machine had when MIDS was installed, not its current name. And machines that have had MIDS uninstalled are still in the server. This is a big part of why it took me 6 hours to audit 2k machines.

When I'm down to under a hundred problem machines, one of our customer's starts having work-stoppage system slowness. Yes, it was because of having MIDS and Defender running at the same time. No, we didn't disable Defender with GPO. Yes, ownership was mad at me. No, it could not possibly be all my fault.

Recently, now down to ~50 issue machines, the owner realizes a VM host is like three years out of date. He asks me if there's a way to get alerts about this so we don't go three fucking years with a vm host having basically no security. Well, actually, he sent a furious message in Teams about why this happened in the first place, then asked about monitoring after I explained it.

The vendor's answer seems to be "lol, no, we don't have monitoring for that."

But, I happen to know already that there's a log file that updates every five minutes when it checks in with the server. And it includes the current version. Which means we all get to hope I can figure out enough about SNMP to query this file on atleast our servers, because if not, I think my boss is going to have a stroke.

Also there are 12 VM's that need to have uninstall option 3 done. Can you go into advanced boot menu in a Hyper-V VM? Not sure. Hope so.

I work for my local MSP and I encountered the biggest cluster I have ever come across and I had to share.

I got a call about a down server from a company who was not one of our clients. I was expecting it to be a pretty easy call and boy was I mistaken. The further into the on-site call I got, the worse it would get.

The server was actually a “server” (a 10 year old desktop with windows server 2019 installed on it)

Windows would not boot. I tried to repair the install and was unable to fix it. Then I checked out DISKPART and noticed they also had a Windows software raid. One of the drives had died and the raid was degraded as well.

I got a hold of the backups. Not only did it use a backup software we had never heard of, the backups were being done by an employee with no backup or server experience. They would just plug a USB drive into the server and unplug and bring it home at the end of the day. It was only doing file level backups and after waiting an hour for the encryption password that no one had I finally got access to it.

The only backup was from August of 2022 and it the software was unable to scan and restore any of the data in it.

So, reinstalled the server from scratch. While that was happening, I managed to extract the CRM backups off the operating system drive but the last backup was January 8th. Their CRM is for customer management, financials, and inventory. The person doing the backups had a backup of the CRM from yesterday but he stored it on the raid.

Now they are moving to Azure in 2 months and they are decommissioning the “server” at that time. Being Active Directory has been blown away, I had to remove all the clients from the failed domain. My only saving grace was that everyone had Domain Administrator credentials. EVERYONE…

So now I have a fresh server 2019 install but a broken RAID5. I had to wait 12 hours to scan and map the broken raid to then write the array to a new empty drive.

All the companies data was on this software raid. All of it. They have no working backup.

On top of all this, the IT person that was running this web of hell before he was fired had network switches in the ceiling tiles and was a rats nest of wire which could not be traced and they ended up having us rewire the entire building as well.

Needless to say I made lots of overtime this week.

EDIT: managed to recover all their data. Pretty sure the company would have gone under if I wasn’t able to.

This is going to be a short story, I just recently applied to a new job that will be managing a support center and their service management platform. It lead me to dig up stats. I used to be a single person IT support department. Because of my very demanding job, I had setup zendesk to keep track of all request and had setup an automation to close tickets and send a survey. Survey was simple tumps up or down. Optionally user could write a note.

I was reading thru thousands of these and most were really simple, "thanks!" or "you're awesome" etc. However some would take the time to praise my efforts. It was really good to go back and read these. Until...

It was such a simple ticket, printer not working. I responded to it within 2 hours. It was fixed within 5 minutes. Tray has been resized and needed to be adjusted. Cleared the queue and sent a test print. I sent the user a follow up that it had been taken care of and to let me know if issue continued. I also added notes to ticket that user had successfully printed multiple documents based on logs and printer page counts. 5 days later ticket closed, survey sent. 6 days later thumbs down "MY PRINTER WORKS BUT WHY IS MY COMPUTER SLOW!"

Dashboard changed from 100% satisfaction to 99.98%...

Why does this still make me so mad when I think about it.

I have written several stories about the time I worked for a company that created network management software for Un?x computers back in the mid-to-late 80s and early 90s. I just remembered another one that happened during that time.

At the time, I was responsible for porting our software to various Un?x platforms as needed by the clients to whom we sold our software. We generally planned on the porting to take 2 weeks, but many times, I could get the migration done in about 3 days, depending on the issues found and how "standard" that flavor of the OS was.

This one time, I was in the process of porting the software to a small computer that had been manufactured by a certain telephone company, whose name consisted of 3 letters & a symbol between the second and third letters. The hardware had been hand-delivered by two software engineers who worked for that company, because they had some meetings with our management and Lead Developer for some project. They were going to be in our offices for a week, and the plan was for me to get the porting done and then they would take the computer back to the East Coast with them when they left.

The first day went great. I unboxed the computer, turned it on, and was going through the compilation process.

On the second day, I had to reboot the computer for something, and all of a sudden, it made 3 soft beeps, paused for some number of seconds (perhaps 10-15 seconds), and made 3 soft beeps again. The monitor showed that it had started the boot sequence, and, if I recall correctly, it may have shown a logo that looked like a Death Star, but other than that, there was nothing on the monitor that indicated what the problem was.

I checked to make sure I didn't have a boot disk or tape in the relevant drives and tried again.

Beep ... beep ... beep ... (pause) ... beep ... beep ... beep. Very, very softly.

I was going to ask the two engineers, but they were in a meeting with our management, so I took a chance and called the support line at their company. The support tech was friendly, and assured me we could resolve the issue quickly. He asked me to tell him exactly what was on the screen.

Unfortunately, this is where things started going south in a hurry.

First off, I was in a fairly large space, and the one phone in the space was on the wall about 20 feet away from the computer. The phone had a short cord, and the way the computer was positioned, I could not see the monitor from where the phone was.

I immediately thought, "Let's move the computer closer to the phone", but, double unfortunately, there was literally no room on any of the tables near the phone.

"So, let's just move one of these other computers out of the way!". Sadly, those computers were critical to certain business functionality and could not be moved without being cleanly shut down by the Lead Developer, who was in a meeting.

I joked with the support tech that we were going to have to triage the situation "remotely". I reminded him that the system was making three soft beeps.

What he said next floored me: "Are they long beeps or short beeps?"

Ummmmmmm ...

"Compared to what?", I asked.

"What do you mean 'compared to what?'? Are the beeps long or short?"

I said, "I'm not trying to be obtuse, but work with me here. I am a ham radio operator and am fluent in Morse Code. I know how to interpret dashes compared to dots. But the thing is, dots and dashes can only be interpreted correctly when heard in close proximity to each other. I literally don't know if these are long or short beeps. I would call them 'medium' beeps. Here, can you hear the beeps?"

Again, sadly, due to the noise of the fans in all the computers in the space, he could not hear the soft beeps coming from the computer.

Silence. Then, "Can you hum them for me?"

Now, I'm pretty good at singing, having sung in choirs and small ensembles in high school, college, and at church, so I did my best to hum my rendition of the "Three Soft Beeps". I'm not sure if it is because I wasn't as good at mimicking sounds as I thought I was, but the technician said, "Hmmm ... you're right, I can't tell if those are long or soft, either."

About that time, one of the engineers from that company walked out of the meeting room to go to the restroom. I stopped him and asked, "Hey, do you know how to interpret these beeps?"

He listened for a few cycles and said, "No, but I have the direct number of my buddy back in New Jersey, and he was one of the principal designers of that hardware. Let me give him a call."

And, surprisingly, he actually did know what was wrong with the computer. I think it may have been a board that needed to be reseated, I don't actually remember. Whatever it was I had to do, worked, and I was back to porting our software.

But he did let the hardware designer guy know that having beeps of only a single duration probably wasn't the most operator-friendly choice.

Before restarting my career in IT, I was very experienced in IT but I was working as a dishwasher in a very unique restaurant environment.

Under the first management I worked under, I could comfortably call myself the most comfortable team member. washing dishes with headphones in and only paying heed when I am yelled at for certain pieces. Which didn't bother me as it's a very high stress environment for the cooks and yade yada.

However. As management left to open their own restaurant to rescue their business after COVID, that's when issues started for me...

New management came. And under the pressure of the new management, the fact that the old team had almost entirely left and the fact that the environment changed from one where it's been basically the same team for a year with minor changes to getting brand new set of staff every 3 weeks. The only real constant was me. Expanding me from dishwasher to front of house greeting staff to dishwasher AND front of house greeting staff to basically being a one man army at the job.

"This is a tales of tech support subreddit, what's the story of a dishwasher got to do here?"

Well, if my duties hadn't expanded to contain managing some of the technical sides of business, I wouldn't be here. Would I?

I had the regular thing pop up here and there that benefited from my work experience, ticket printers needing some network troubleshooting "Surprisingly resilient printers might I add." Our POS system needing updates, upgrades and long calls with support to get it working. To the computer where business was conducted. It was all slowly but surely becoming my duty.

I feel the need to mention some of the politics of this place. There were 3 owners, one was the old manager who left to start his own business and was slowly beginning to distance himself. One who was a hard cheapskate who didn't treat his employees all that well and didn't like me, and one was a hard cheapskate who didn't treat his employees well at all but for some reason inexplicably liked me.

We were under the management of owner 3 right now. With owner 2 livid that I was touching ANY of the back of business things. But I frankly didn't care. I like doing IT stuff, I know it might be freaky to some of y'all but I liked fixing the office printer whenever it didn't work all that well.

Here, however, begins the controversy.

one day. I was asked by my boss, owner 3 to change the menu prices on the restaurant's website as we were now increasing prices.

I told him this should be easy. Just edit a couple of lines of HTML. I went to the website to check it out on my phone. In my many years of working here I was aware of the website but not that it was much more than a promotional page, the menu and it's prices and that it was http, yes. It lacked an SSL certificate.

I click on the website from my phone to check it out, I flick away the SSL certificate warning as if it were nothing. "It doesn't matter that this site is insecure. We don't have anything important on it anyways!"

However, a weird tab caught my eye on top of the screen. Right next to the menu....

"Gift cards"...........

Huh... is this.... What is this page?

I click on it, and in horror I see before my eyes.... A form......

This form requests the following information.

First name, last name, first and last name of receiver.... email address..... home address......... Birthday????????? PHONE NUMBER?!?!?!?!! WHO WROTE ALL THIS!!!!

Ignoring all this... And ignoring the questions of my manager... I click on the "order now" button and it takes me to a paypal pay page.....

"THIS PAGE DOESN'T HAVE AN SSL CERTIFICATE" I shout panicked.

The owner, taken aback replies "what?"

I explain to him that this page basically compromises the information of any user who inputs anything into the fields in the page would basically be broadcasting it to the entire half of the internet that it has to go through until it reaches the server hosting our website. Frankly he does not care despite my insistence... I tell him that we can go to the website management page their provider gives them. He tells me to do that he has to get the login information from owner #2.

He tells owner #2 we're trying to change the menu prices. He gives his login information to owner #3, owner #3 logs me in from the restaurant's imac. I change a few lines of HTM frm like 41/43 to 49/51. Huge price hike. I beautify some of the inconsistencies in the formatting of the pricing and I send it online.

At this point began the begging and pleading to "purchase" an ssl certificate from the domain provider. A shady host where you have to pay 50$ yearly for their most "basic" SSL cert.

He wouldn't do so, and he definitely wouldn't change hosts especially since that host is who he bought the domain from. So now we're at a bit of a pickle.

However, the save came completely unrelated to me. As finally, it was owner #1's paypal and personal Email that was linked to the gift card page. And he decided to focus more on his new restaurant and ignore the old one! And when they asked me if I could somehow change it... I said that, of course..... I can not! I have no knowledge on how to do so! Not only do I have no experience actually designing pages. But that changing both the Email where all this information went AND the paypal page where the money would be sent would be incredibly hard for me and it could get messed up! I don't want to be held responsible!

Instead, I suggested using a third party gift card processing app that can integrate to our POS system instead of just being an excel database of codes sent from one of the owner's Emails that we aren't even able to cross reference in a timely manner! Just get a solution that reads QR codes and confirms them directly to our app and subtracts it that way so it's tracked!

We'd have to pay a subscription fee as well, ofc. But they liked this idea so much that they went ahead and bought it before I even knew lol.

All that was left to do is actually implement it. Being the lazybones I am I wrote some HTML embed in some text that said "Click here for gift cards". Beautified the page a little more and even got the footer to stick to the bottom of the screen like some sort of fancy website! And in recognition of this achievement I deleted the name of the "web designer" whom was commissioned for this website who used one of those other HTML template pages from the mid 2000s to "write" the whole website. Which left it's name in the html code as comments...

And my solution worked!

Well... At this point I'd hardly call myself the "dishwasher" for that place. I was practically running the show and covering for almost everyone who cold quit on us until one day I decided to quit myself over an altercation with owner #2.

In the end, a few years later, I went to the website again to check it out and to my surprise very little has changed from when I changed it. In a way, I'm glad my fingerprint on it is still there. The gift card coupon page embed... Still very happy about it. And decided to post it here.

I posted this elsewhere and figured this would be a good place to post it too.

So this happened a good like 7-8 years ago. I used to work on a help desk. We had recent switch a major software to a new one. They didn't want to train us on that system so they paid for a third party deak to deal with those calls.

When you called it was phone tree that basically said to press 1 for the new software or press 2 for any other issues. If you pressed 2 and had an issues for line one we HAD to transfer you. We had no training, access and the contract stipulated it as well. So no way to help you in any way. things are different now but that was the deal at that point.

So this user calls us, I get her information and proceeds to immediately tell me that she is having an issue with this system. I try to interrupt her polite and she cuts me off and proceeds to yell into the phone " HOW DARE YOU! YOU DO NOT INTERRUPT ME THAT IS RUDE AND YOU WILL LET ME FINISH!

I said nothing and let her talk. She proceeds to talk for a full 5 minutes (I timed here) giving me the full run down of her issues in excruciating detail. She finally finished. I let there be a pause where she goes "hello?" Both to annoy her a bit and to make sure she is actually done.

I put on my AAA+++ customer service voice and ask if she is done as I didn't want to interrupt her again. I then explained sweetly and politely "I'm sorry I was trying not to waste your time early when I tried to interrupt. This is the wrong line for this system, and I was trying to get you to the correct line so they can help you. I need to transfer you." Just oozing all the "I was just trying to help you" I could.

There was no sound coming from the phone but I could feel the red hot rage and embarrassment coming through the phone as I transferred her to the correct line.

I used to work for a large Seattle based software company in the mid 90s. We had decent machines, xeon workstations IIRC.

One morning I had to power cycle my main dev box, and when it started up the CPU fan went into thermal runaway due to a motherboard issue.

As it passed about 10,000 rpm I hit the power button and called up of local PC repair folks. The tech listened to my description and asked how fast the fan was running, and I said pretty fast.

"Can you repro it for me?"

Sure, I can do that. As the fan approached the limits of CPU fan integrity I could barely hear the shouted voice over my headset..

Turn it off. Turn it off!!!!

Three hours later I had a need board and everything was once again calm. Tech and I had a nice laugh about it....

I do 80% of helpdesk stuff for our service. For reasons galore, our boss doesn't really seem to value the concept of helpdesk as much, not even when a big chunk of tickets are from paying clients from important places, but I digress. We have weekly team meetings, but I have been absent from the past two because I've been delegated elsewhere in that time slot. I reviewed the meeting summaries that were sent out afterwards and saw nothing of note, sans one bullet point at the very top:

• "Locate password for $email_address at $external_host" (which has a big chunk of our users)

Worth noting, we don't use this inbox for receiving mail. To my knowledge at the time, this was just the address/username we needed to login as an admin at $external_host. So of course I knew the password because I've been using it for helpdesk stuff, and it was given to me just months ago by two other members of the team, so I assumed that was meant as a reminder for them to give it to the boss as well.

Cue Monday afternoon:

Boss: "Do you have the password for $email_address?"

Me: "Yeah, of course."

Boss: "They gave it to you already?"

Me: "... yes? Months ago?"

Boss: "What months ago? For this account?"

Me: "Yes."

Boss: "This password?" reads out the password

Me: frantically looking through password manager "No? That's not the password for this account?"

Boss: "No, this is the password, they gave it to me today."

Me: "Who gave you the password?"

Boss: "The $external_host guys, I asked them for it 'cause we couldn't find it."

Me: "Oh, okay, so you reset the password?"

Boss: "Yes I guess, they gave us a new password."

Me: "Okay, got it. Good thing you told me, otherwise I would have lost access to helpdesk stuff at $external_host. That could have been pretty bad!"

I considered myself lucky to have gotten the info before I needed the account again, saved the new password and called it a day after our meeting ended because it was already late and Mondays on helpdesk are hell reincarnate sometimes.

Tuesday, I get a single ticket about email notifications from our service not working. Rather common occurence, as users can set up their notifications to be sent via different clients and addresses, and there's a billion settings on their side we're not responsible for. So I log in, check their settings, can't see anything wrong, whelp that's an easy escalation to the server guys to poke at our email stuff a bit, just to make sure it's all good on our end.

We had an unrelated bug crisis later that same day, so by the time I finally signed off for the day, that email notification ticket was the last thing on my mind. Until this morning, at least.

Lead dev (currently on vacation, mind you): "Logs show no email notifs have been sent on $external_host since Monday, not a single one. Can we check this ASAP?"

Boss: [some completely confused babble about me sending email notifications on $external_host via my own email account and if I changed anything after he gave me that new password]

Lead dev (frustration palpatable in the email): "This has nothing to do with her account, this is about $email_address."

Boss: "Yes this is why I am asking her if she changed anything, the password I gave her worked but it was different than the password she had before."

Lead dev: "... we changed the password for $email_address?"

Me: "Yeah, Boss had it changed on Monday, what's going on?"

Lead dev: "Please get it over to me. We use $email_address and that password for the email notification settings at $external_host. Nothing is being sent because the new password wasn't updated in the settings."

We got it all sorted out in a minute after that. How nice of our boss to foster a true sense of cameraderie by not informing any of us in due time!

Anyway, now I'm stuck with a boatload of tickets about yesterday's outage on email notifications, asking what happened and why. Anyone got corporate speak for sorry, our boss is a toddler playing with passwords at his own discretion?

Update a few days later: turns out the boss was so particularly unconcerned about changing the password because he didn't think he changed the password. He thought saying "we forgot the password, please give us the password" meant the $external_host guys would just look up our current password and give it to him. In plain text. Where do I cry and/or scream my lungs out?

Not to bore you guys with my entire life story - but super quick Cliffs Notes gives background:

Both of my parents passed when I was young. 9 and 12. I'm now 38.

I've always loved electronics since I could hold something in my hand, and wanted to know why, how they work etc. It lead me to get into computers and tech support etc later in life.

My brain literally mashed together the current circumstances of me being a computer tech and life I currently have with my past all in one. Woke up feeling - emotion. Don't like it. Don't usually happen.

In my dream, I was on a call with my Dad as tech support on the phone with him trying to fix our NES ( Nintendo ) with him in our living room as 5 yr old me could be heard playing in the background. Even heard my mother speaking. It seemed so real, as if it actually HAPPENED. I even remember slightly one time my father on the phone with tech support trying to figure out an issue with the Nintendo.

It was freaky guys. So realistic it had me wake up bawling my eyes out. Have not really thought about my folks for a loooooong time, so it kinda come out of left field.

Thanks for letting me write that out.

It's cheaper than attempting therapy ( which I couldn't afford anyway )

User: So I have a computer my ex-girlfriend is giving back
      to me and I can't get in  
Me:   Okay we have a procedure for that. If you have a copy 
      of the receipt you fill out that form and wait to get  
      approved and then you can erase it  
User: I was looking at it and it says it may get denied  
Me:   Yes technically it could get denied but this is the 
      only methodology we have available  
User: So even though I have a receipt with a serial number
      it could get denied  
Me:   Wait.. have you been denied on this  
User: No, but it says I could be  
User: I spend thousands of dollars on this computer and 
      you're telling me I could be denied even though I  
      have everything to prove it's mine  
Me:   Technically yes but if it's a new computer and the 
      receipt has the serial number that doesn't seem
      likely to me
User: And if it gets denied I'm SOL  
Me:   Yes if it got denied there's nothing else we could do
      to get you into the computer but again that seems  
      unlikely in this case  
User: This is the last [Brand] computer I ever buy  

***User disconnects***  

Why yes I DID simplify the (what must have been five?) times the user had us going in a loop where I reconfirmed that we had one way to resolve this and he requested confirmation that he COULD be rejected.

Checking the case notes he had already called about this 2 hours prior to this call. And was told as much then.

This was in the dinosaur days of the 90's, and happened to a friend of mine. His company captured check images for several banks, using a high speed check reader/sorter. At the time, these were high-tech and high dollar, and since they were essential equipment, came with 24/7 maintenance.

One night they called for support. Told the tech "think there's something wrong with the sorter. Smells like hot electrical."

It was 1am on a winter night, and the tech didn't want to leave his cozy bed and drive an hour to investigate the issue. So he told them "call me back when you see flames" and hung up.

An hour later, his phone rings again. "Ok, we have flames. Are you going to come now?"

Of course, the sorter was now a boat anchor, and the manufacturer/maintenance company sent a bunch of guys in suits to argue how they were in no way liable. Unfortunately for them, friend's company recorded all their phone calls, and simply played back the tech telling them to "run it until they see flames".

Friend's company got a brand new $250k sorter out of it. I think the tech guy got an extended period of unemployment.

I saw a comment thread on r/sysadmin about ISP technicians sneaking into demarc spaces and it reminded me of this story from a couple years ago.

At my previous company (Company A), they owned Building 1 and Building 2. At some point there was some weird restructuring and moving of buildings. At the end of this, Company A built a property Building 3, and sold Building 2 and half of Building 1 to another company, B. They were all 3 in a row, so it was a little weird that we were split by them. There was an underground fiber line ran somewhere between our 2 buildings (1 and 3), and there was an entire infrastructure set up between their 2 sites above ground (2 and 3). There was also an enclosed walkway that connected them.

Company B was making some facilities changes and with this new system, they needed to make some other changes to continue to meet building codes. The enclosed walkway (and anything in it) needed to be completely demolished to separate the 2 buildings, and allow emergency egress from all sides of their Building 2.

They politely communicated the details to us, as good neighbors do, and our facilities team sent out details related to parking and site access. The construction was completed, everything seemed fine. Some time passed.

Our annual elevator certification was scheduled and part of this included checking that the emergency phone line worked. You know how if you get trapped in an elevator, you should be able to press the emergency button, or lift a phone, and immediately get transferred to the local emergency services? That phone. The inspection people tested the phone and... nothing. Line was dead. We were given a few weeks to make the necessary changes and get it back up to code, and they would reschedule.

My boss tried to get in touch with a certain major national ISP whose name is 3 letters with a symbol. We had the absolute worst luck with them. Every time we needed support, our provided account numbers couldn't be found. Eventually they would just open a new account for us. Our billing team was convinced they never applied an overpayment correctly but they couldn't get anything communicated because of all the issues with the account. A year before this story, they accidentally canceled one of our other phone lines and gave away the number, so we didn't even know which phone number was supposed to be assigned to this elevator. Our account manager was no longer with the company. The replacement person who was listed on their voicemail message didn't even work on the AM team anymore. We eventually got a response from some VP who was able to provide us with our new AM, who scheduled an on-site visit to troubleshoot and repair the phone line.

I was the on-site tech for the day and got the phone call that the ISP tech would be there in about 5 minutes. I normally worked from Building 3, so I headed over to Building 1 to meet him. 10 minutes later, he called me and said that he's in the lobby of Building 2. I told him that we no longer worked from Building 2 (as of about 15 years prior) but he insists that's where the billing address was. I told him it certainly wasn't, and that the elevator was in Building 1. I was standing inside the Comms room and there was a very clearly labeled box that had "Property of (provider) do not touch" stamped on it.

The technician eventually believed me and walked to Building 1. He measured all of the lines and there was no voltage going to any of them, which implied nothing was hooked up. The line had a faded paper tag written in pencil hanging from it. It listed a phone number that matched the previous number that had been accidentally disconnected, not our current one reflecting our most updated billing statements. He continued to insist that the demarc was listed as Building 2 and that he needed to measure it from there. I told him that I had no access to that building, or knowledge of where the Comms room would be, or ability to get him access either. He said he would go check the main box somewhere else and left. I sat in the Comms room with my laptop and continued to work other tickets while this guy was gone for about 25 minutes. He came back and told me that he was able to confirm in that other equipment box that our line went to the same point as the other company's line and was DEFINITELY in Building 2, with an install date roughly a year prior.

That timeframe was peak Covid times, when our team wasn't on-site, and about 2 months before I'd started. working there. It was also around the time when the ISP had stupidly gotten rid of our line and changed our phone number. Some confused technician had been given the wrong address because of a billing address clerical error because of the complete disaster that was our account history with this provider. He'd wandered into Building 2, and activated our line on the wrong building. Because the 2 buildings were connected, the line went through the overground walkway, directly to the elevator. When Company B did their construction project, their IT department had no knowledge of the elevator phone line, because it wasn't their account. We had no knowledge they'd done this improperly, because we'd told them the issue was with Building 1. When the construction was done, they took down the infrastructure because to their knowledge, it was legacy and not live.

The technician packed up his stuff and told me since he was a repair tech, he wasn't able to help. I needed to call our Account Manager back and request an installation so they could send an installation tech instead. I hate 3 letter 1 symbol provider.

From: me
To: Brian, Jeff
Subj: Ticket 20240221.015
Date: 2/27

Hey guys,

Ref above ticket, the new hire is starting tomorrow 2/28 instead of 3/4. Please set up cubicle 168A for them by end of day.

(Reply from Jeff, verbal: "Zug-zug!")
-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-

From: my boss
To: me
Subj: New Hire (name) desk not set up
Date: 2/28

(me),

I was here at 7 am and the cube for (name) was not set up. Going forward, make sure to check Brian and Jeff's work on your way out when you assign them to work together.

(Thought from me: "The fuck? They're screwups and they feed off each other but to completely not do the work?")
-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-

From: grandboss
To: my boss, me, my team
Subj: FWD: Junk alert!
Date: 2/28

Hi team,

Please make sure this doesn't happen again. It's a really bad look:

From: Bill
To: grandboss
Subj: Junk Alert!
Date: 2/27

Hiya (grandboss),

I just wanted to let y'all know that there was some leftover IT stuff in cube 168A that I saw as I was leaving today. I went ahead and let Jim in Facilities know to clear it out of there for you, since we got a new hire starting Monday and they might be sitting there. Take care!

YiC, Bill

To set the scene:

I worked as a netadmin for a mismanaged MSP in a smaller college town who had been contracted by a local scatterbrained apartment complex to manage their IT needs in their main office area.

The tale:

This particular apartment complex was having some construction done and had let us know that some additional cabling was going to be run (by the construction crew, not by their MSP) in the new area and that once that was done, we would need to come in, patch these runs into our switch, and then test. After a few months of work, our sales team had received the call that construction was finished and that they were ready for us to come do our part. I was tasked with the job and sent out onsite.

When I arrived, I headed to the area where construction had been completed and assessed the situation. The first thing I noticed was that construction had actually not been completed - there was still some drywall and painting work to be done and all the new drops they had run were just hanging out of the wall and hadn't been terminated into jackplates.

Since terminating those cable runs wasn't part of what they called me out to do, I headed to the data closet figuring I could just patch in those runs into our switch and could test once the construction crews had terminated them at the other end. In the closet, I saw the new cable runs coming out of the ceiling next to our rack but instead of them just hanging loose, ready to be terminated, they were actually already terminated into RJ45 heads and connected to a switch sitting on a metal folding chair next to the rack. This was odd, since we had installed switches in their rack (which is where we were going to patch these in) and the one sitting on the chair was made by a different vendor than we used. I took pictures and sent them to our sales team.

I went and grabbed the lady managing the apartments and showed her the situation. I showed her the cables hanging out of the wall in the new construction area and let her know they weren't complete. I told her we could terminate those, but that was part of the construction team's job and she probably didn't want to pay us to do a job she was already paying another crew to do. She agreed. I took her to the closet and showed her the switch in the folding chair where the new runs were terminated. I let her know I had no idea where it came from and that she would need to have the team that installed it remove it before I could do anything else.

She was also confused and grabbed someone from the construction team who apparently knew what was going on and brought him to the closet. She asked him why they had brought in a switch and he replied that it was part of their contract with the new construction. I let them know this wasn't part of our contract (or our network) as their MSP and we couldn't manage it or do any work on it, especially considering we already had managed equipment in their rack that all data drops would need to go to and that this extra switch would need to be removed. As far as the contract with the construction company that included a new switch, I let them know they'd need to get all that figured out before I did any work.

The apartment manager lady said she would get with the construction manager and figure out what was going on and let us know. As I was leaving, I noticed a box sitting in the middle of the data closet that had some APs sitting on top of it that were the same type of APs that we had installed for them. I shrugged as I left, thinking that was a problem for another day.

As anyone might have guessed, it was a problem for another day as I was called out not long after that to figure out what was going on with the wifi. Come to find out, not only had the construction team brought in this switch, but they had also pulled down all of the APs we had installed and installed new ones of the same vendor as the chair switch they had brought in and had plugged them into their switch, still separate from our network.

Once again, I had to let the apartment lady manager know that the construction crews had installed some new equipment that we couldn't manage and that they would need to remove them and reinstall our original equipment before we could do any work. She was again nonplussed.

Clearly, this project had been mismanaged and/or not communicated well from the start. And on top of having to have these crews backtrack and undo a bunch of work that they had done, they also had to pay for me to be called onsite twice to not actually do any work.

Fairly easy work for me but a huge pain in the butt for the customer.

At least this time, it wasn't a blunder caused by mismanagement from the company I worked for, who was prone to mismanagement.

C'est la vie.

​

​

Location:
Whirlwind Computing^(TM)
A nondescript MSP that can blow the competition away

Cast:
$Me - Hero extraordinaire and the debonair questionnaire
$Stevarino - Good with walls of fire
$Murphy - Has laws written in stone, do not attempt to taunt ^(a second time)
$CRM - Doing it's job by asking for lots of logins
$Customer - Confused

Customer hires our MSP dream team to upgrade their AD, web server, firewall, etc.

All goes well +/- a few niggling issues that are pancaked by the IT Hammer^(TM) (patent pend.)

Then Murphy calls!

Hey guys, our CRM sessions are dying every few minutes while trying to process orders, it's a massively inconvenient process to sell high-strength stainless steel hollow tube sections!*

We try a quick tricks and put some nangling pins in for observation/let the MSP team work on the issue as it was completely random when it happened.

A week later with no improvement, Customer wants to use the IT Hammer on the CRM.

We advise not to and in the interim, there's tech discussions and the occasional shootings of trouble trying to narrow it down.

A Zoom session is organized and I start asking questions about the CRM interactions and kick computer off the domain to see how it reacts and other sorted madness...

We stumble upon the odd thing of the off-domain PC trying to hairpin the connection instead of going local to the CRM platform.

Wait... Steveo, why is this PC trying to hairpin?

$S - Ummm... Aha! That's why! It's trying to reach the public side of the CRM

That'll really screw up their ordering when the computer is constantly flip-flopping between the private and public paths

One refresh DHCP on all affected computers and a plan to reboot the firewall tomorrow

The end result was DNS being too helpful and the failover/backup/spare DNS was answering first in some queries and Murphy was obliging by obliterating the customer's connection to the CRM

The wall of fire was also being cheeky and it would destroy https session tokens which only made the problem that much weirder to narrow down

End result, with a split-horizon system yes, it is always DNS or make sure hairpin is working at the very least to ward off Murphy's tricks.

^(*Semi-obscure joke - Stalatube)

​

​

At a hospital provider I worked at, the emergency department was having some building works performed on it, so a few demountable were hired to house some of the doctors whilst their offices were being ripped apart by the builders.

The demountables were a fair distance away, so a fibre was run, a standard network switch model we use was installed and 2 Wireless Access Points installed to provide WiFi coverage.

2 things to note here is the standard network switches we use had fans in them (which were relatively quiet after they booted), and we usually didn't lock racks in non-public areas.

The fan was too noisy for one of the doctors (let's call him Dr. X), so he accesses the rack and unplugs the switch. This takes out the WiFi in the demountables (as the Wireless Access Points gets both power and data through the switch) and generates an alert that a switch was offline.

So you would think Dr. X would contact Helpdesk, or at least someone in the IT department? No. He immediately walks down and sees the executive response for IT and has a winge about how crap IT is, this is unacceptable etc.

As the exec then makes contact with IT herself, I've already been dispatched to see why the switch is down, and quickly discover that the power has been unplugged.

I ask if anyone there knows why this was unplugged.

"Oh yeh, Dr. X said that was making noise and it was annoying him, so he unplugged it"

"Ahh, so that device is what supplies your WiFi, so unplugging it takes out your WiFi"

"Yep, that makes sense, especially considering the WiFi went down as soon as he unplugged it"

I plug the switch back in, check everything comes back up and lock the rack.

It's communicated back to the executive that because Dr. X unplugged the switch, this is why WiFi stopped working and this is all understood.

Without this story getting too long, as this doctor was complaining about the fan noise so much, we needed to acquire a fanless, high temperature model of switch (that cost substantially more than the standard model) plus very expensive overnight shipping for the 2 weeks that he was in these demountables.

This story is probably only tangentially related to tech support, but it is related to clueless users, so here it is.

Back in the 1990s when I was a programmer, my group was responsible for a business-letter-writing package. At the time, the editor control was "home-grown" and had rudimentary bolding, italics, and underlining. However, we had been asked by several clients to hook it up to Microsoft Word.

One of the developers on my team got the information for Microsoft Word and filled out the requisition slip and sent it to Purchasing. Under the "Reason for Purchase" box, he put, "Integrate MS Word with our software per client's request". About two weeks later, he gets an email stating that the purchase had been rejected.

He thought maybe it was because he was relatively new to the company, so he asked me to submit the requisition. I did, and even included the reason.

A week later, I also received a rejection notice, so I called the Purchasing office. When I asked why the request was being rejected, I was told, "Our company standard is Lotus Notes. Since you already have Lotus Notes installed on your computer, there is no need for you to have Microsoft Word."

I told the Purchasing agent that this wasn't for use on our day-to-day tasks, but it was so we could integrate Microsoft Word into our business-letter-writing package, as requested by the client.

For some reason, this simply did not compute, and I was told, once again, that our company standard was Lotus Notes, and we could not get Microsoft Word.

I went to my boss, and we both went to HIS boss and told him the tale. The Boss-squared got on the phone and called the head of the Purchasing department and rained down fire and brimstone, telling them that because their purchasing agent could not understand the difference between "using something for my work" and "making sure something works for our client", we were now three weeks behind.

By the following Monday, we had a brand-new set of floppy disks for MS Word.

​

We put a sign up on the office door recently to say please don't knock unless you've submitted a ticket.

User knocks, looking confused and looking around.

Asks, "Where do I get a ticket from?"

As in, they thought this was a butcher's or Argos in the 90s and needed to take a paper ticket and wait.

Then to top it off the issue was "I can't get into my emails on my laptop or phone" so I just followed them as it was going to be easier.

They hadn't even attempted to open outlook on the laptop & their phone just had the big black bar in the outlook app that said sign in.

Like. Come on. I swear some users go out of their way to be so bad at IT. Not even attempting anything beforehand!

Me: *After dealing with a horrible user on Friday who's given us no end of grief on the service desk*

-Weekend Passes-

Me: *Coming back in with a ticket in my queue for a leaver with a note from my manager saying "You're going to love doing this one."*

Colleague: "Why are grinning?"

Me: *with the look of a happy gremlin on my face reading that specific users leaver ticket*

Me: "Oh, you know <LEAVERS NAME>?"

Colleague: "Yeah, what did they do now? hear they pushed you so much you almost slammed the phone down on them last week."

Me: "Their mailbox just got promoted to a "shared mailbox""

Both of us: *Proceeds to cheer and hug each other as we'll never have to deal with them again*

For context, the user got fired after their manager heard the call recordings of the abuse that the user gave us since they started and also because they could not use a computer and even restarting it for her was like asking them to move the Earth, including avoiding calls and just being terrible at their job.

It's the small things in the service team that bring us together. :3
I hope this gave you all a shot of that serotonin we all desperately crave after a long week. xD

This story involves me, the client and the vendor. And its been going on for a year.

Recently an issue got escalated to me and my team is hoping I can solve this ongoing issue for the past year. After a quick verbal review by a colleague, I get the ticket and look at the notes from the past few days. There are other tickets on this issue but I'm just focusing on the recent one.

The issue is a piece of equpment installed on the manufacturing floor has not been connecting correctly. They are unable to connect to it with the management software to control the device. Its one fo those typical WinXP embeded software devices. I've found that 9 times out of 10, its a vendor configuratoin issue. The device has been put on a subnet that is for the shop area. And this is where the vendor says the problem lies. They want to be put on the same subnet as the software. This issue has been going on for a year, because of long timeframes when the vendor does bother to respond.

After reviewing it I reach out to the client and CC the vendor asking for some missing information. The vendor politely responded quickly. I then ask client if we can setup a time so I can go through the software with them and understand how it functions so I can better analyze the network problem. We agree to a time and we connect then to go through it. Its fairly straight forward configuration setting. It shows 3 fo the 4 machines are reachable but that 4th one installed last year is not. I run some tests to check the communications and I see it pings but the port it wants to connect is not open. I advise the client I am going to report my findings back to the vendor regarding the port not open on the device. We both agree the test shows an issue with the vendor device.

Before I do that, I run some port scans from another device to compare a functioning device compared to the non-functioning device. And as I suspected, the non-functioning device has some ports open but not the port the software is trying to connect. This clearly show a configuration issue with the vendor device.

I put this all together and send this to the vendor stating we have found no network problems to cause this issue. A reasonable vendor would see the logic in my investigation and go back to investigate their device. But this is not a reasonable vendor. No their response was to say they want the device on the same subnet as the software... again. I hold back the urge to respond in some unprofessional manner and go make a cup of tea instead.

I know I can change the switch settings and it can give me leverage for the future. So I change the port config and put it on the same lan. I send him a set of IP addresses he can configure into the device, then I wait. later I see the vendor send info to the client on configuring the static IP addresses. When I check again if the new addresses are alive, I find they are and run my same scans. And unsurprisingly the same ports are open but the one port the software needs open is not.

The story isn't over. I'm now waiting for the vendor to try to tell us its a network problem still, at which point I'm going to tell him we will not waste any more time until the correct port is open. After all, I gave him the professional curtesy to change the lan for him, so I expect him to give the same in return by opening the port. This is going to be a fun responce if they do.

Back in the early 10s I worked tech support for a large Swedish ISP. Like most people who did that I've got tons of stories, and I thought I'd share a short one, and then a long one.

The first one is pretty straight forward: Customer calls in, internet was down this morning when he woke up and still isn't running. I do the standard troubleshooting and I can see we've got no link up to his final connection point. I ask him to go over to his fibre converter (after explaining what it is) and ask what lights are on, etc. Then he hits me with this:

Customer: "Hey, there's some kind of gelatinous slime coming out of this box here."

*beat*

Customer: "Is it supposed to do that?"

I cannot tell you how close I came to telling him that yes, that was perfectly normal, that was merely the excess coolant dump.

The second story is, by popular opinion amongst my friends, the funniest story from my days in tech support, though it doesn't actually involve much tech:

Customer calls in and their landline isn't working. This customer speaks with a *thick* german accent, like a comically clichéd one (although a clichéd german accent in Swedish, which incidentally made her sound quite a lot like our queen, who is German). She's got Voice over IP, and in those days we had a very common issue where our brand of routers would sometimes just drop the entire config for a customer's VoIP. Sometimes this was resolved simply by resetting the router and sometimes you had to enter all the VoIP data manually again. Buuut since we were used to doing this pretty much five or six times a day neither method really took any time at all and had an extremely high success rate, so pretty chill calls to be getting overall.

*sidenote: I was later told this was in fact at least in part our own doing, as there were two ways of getting our routers to accept firmware updates. The first, and correct way of doing it, was to simply reset the router, forcing it to check for updates. The second, wrong (but slightly more 'exciting') way of doing it, was to select the latest firmware from a dropdown list in a part of our UI. However this second method had two issues:

• Number one, the list included ALL firmware releases and did not filter depending on which model of router the customer had. If customer had a late model, no problem, but an older model might not be able to handle the latest firmware.
• Number two, this in fact 'hard-selected' that firmware for this router, meaning it would not ever check for firmware updates ever again unless manually instructed to.

Everyone was told to only ever use the correct method but infallably agents would discover the 'cool' dropdown-list and use it. Then they would tell the colleagues about their cool, more optimal way of updating the firmware and it would spread. Personally I don't think this can be the sole culprit, we just had so many of these calls for about a year there must have been a different cause, but we might have made it worse.

Anyway, after a very short look-over and some fast typing I resolve the issue, the lady thanks me profusely and then asks me for my last name. I give it to her and her voice suddenly perks up. Then the following exchange takes place:

Customer: "Did you say [surname]?"

Me: "That's right."

Customer: "That's a german name!"

Me: "That's right."

Customer: "Are you from Germany?"

Me: "Haha, well no, my family is of german descent but we emigrated to Sweden in the late 1700s."

Customer: "Aaah, I see. Well, you know, these days a lot of people would not admit they are from Germany, but that thing that happened, it was not our fault."

Me: "*stunned silence* ...okay was there anything else I could do for you if not have a nice day goodbye."

​

I was going to reply to a comment in another post, but I didn't want to hijack the original thread or the comment, so I thought I would post a story.

In case you want to get a little of the background, the original story was “It’s because it doesn’t have a mainframe, isn’t it?”, posted by u/thebarcodelad. The comment that triggered this memory was made by u/knottabiggins:

I still can't get over (20 years later) the one who, after I fixed the problem, asked me what was wrong. This user was about as computer ILliterate as they come. So I told her.

I said, "Oh, the ODN conduits got stuck in the Feinberg oscillating framistat mode."
Her response? "Oh. Okay, that makes sense."
I couldn't believe she actually had the gall to pretend to understand that Star Trek doubletalk! She would have believed anything I told her, as long as it sounded "techie."

With the background out of the way, here's my story.

Back in the mid 1990s, I was working for a company that provided software to various financial corporations. One of our clients was an insurance company and was located on the 11th floor of our building (we were on the 8th floor). Our main client contact person was a woman named "Sue".

Now, Sue, my boss, and I all lived within half a mile of each other and for several years, carpooled from our home in suburbia to the building downtown. During these daily commutes, Sue would mention issues the office was having with the software, or she would suggest improvements. My boss was driving, so I took notes and when we got to work, I would add her suggestions to our backlog and figure out the best way to incorporate her suggestions to our software.

(This was literally one of the best vendor/client relationships that I have ever been in. The symbiotic nature of a client making suggestions and the vendor providing those features based on a daily 60-90 minute commute was very effective!)

The three of us were really good friends.

Anyway, sometimes Sue would ask, "When I was doing such-and-such today, the system threw an error. Any idea why it happened?"

I would always tell her, "I'll have to look at the log file, but I'm pretty sure it isn't an actual problem in our software. It's most likely a random, cosmic event in the universe ... probably a neutrino flipping a bit in the memory core. I'll look at the log file tomorrow and confirm if that is what happened."

Sue would laugh and then we would talk about the local sports teams or some other stuff.

Anyway, one day, I was installing some software in their server room. Sue was in the room with me, watching the installation process and being there in case I needed credentials for a section of the network to which I didn't have access. While we were waiting for the software to be installed, she asked me about a problem that one of her colleagues had run into the previous day.

She said, "Oh, by the way, Laurie had a problem yesterday that I think may have been another random, cosmic event in the universe. Tell me again what those are."

I launched into, "Well, approximately 9 minutes before she encountered the problem, a neutrino was formed in the center of the sun. It was expelled at the speed of light, hurtling toward earth. Nine minutes later, it entered our atmosphere, went through the building -- because neutrinos are incredibly tiny, don't you see? -- and because they are so small, went through everything ... walls, floors, tables, computers, everything ... except, for some reason, it hit happened to hit one of the bits in the main memory chip for the CPU. That bit was originally a '1' and got flipped to be a '0'; or, it could have been a '0', and it got flipped to a '1'. Either way, as it continued on its journey through the center of the earth and back out into the universe, that one, single bit flip caused the CPU to stop in its processing and, not knowing what else to do (because these kind of events are so rare) made it throw an Exception. When we are done here, I'll take a look at the log file and figure out if I can confirm whether that is what happened."

Sue laughed and said, "I thought it was something like that!"

I finished the installation, told her to email me the log file from the other user, and went back to my desk downstairs.

About 5 minutes later, my boss asked me to come into his office. I got there, and he told me to shut the door.

He said, "I just got an email from [Some-other-manager] that I think you need to hear."

Apparently, there were two people from my company in the client's server room doing something on their AS/400. They had heard me telling Sue the story about the neutrino, but they didn't know that Sue and I were good friends. The email included phrases similar to the following:

• Unprofessional behavior;
• Outright lying to the client;
• Should be fired or severely reprimanded; and,
• The client was actually buying his unmitigated BS!

After my heart stopped pounding and my boss started laughing, he then sent the other manager an email of explanation. In that email, he explained the relationship between me, him (my boss), and Sue, and that not only were we good friends, but she absolutely knew I was simply embellishing a story I had told her many times in our daily commutes.

There was no love lost between our group and the other group, and, to be honest, my boss had actually told me that the reason our group was held in such high regard by our clients was because we totally shined whereas the other group was primarily known for screwing things up. The other manager was a not-very-friendly person, and ruled his petty little fiefdom with an iron fist.

Unfortunately for me, about 2-3 years later, there was a reorganization looming, and I realized that my next step would be to move over to the other manager's group.

I took that as another sign of a random, cosmic event in the universe, and started looking for greener pastures. I left the company shortly thereafter.

And about six months after I left, I heard that the other manager was fired because he had made repeated, improper suggestions toward his female administrative assistant.

​

This morning I had a teacher come to me and complain their keyboard doesn't work.

As normal you show the IT guy the problem and it's not there. I give him a loan laptop anyway. I go upstairs to get a coffee, as I'm approaching my office I see the loan laptop with a note reading; keyboard still doesn't work. I'm thinking wow that was quick.. since it was working when we exchanged laptops.

I take his original laptop to him in his class room with it working normally, then head over to the teachers office. Guess what I find?...

There is a stack of paper on the external keyboard holding down the "control" key. The external keyboard is plugged into his docking station. And I now know why it's not working. Before the laptop plugs into the dock the keyboard is sending the "control key press down" signal to the laptop, user unplugs laptop and the laptop still thinks the control key is being held down, hence keyboard "doesn't work"

Later on in the day I go see him again and explain what happened.

This happened just earlier today.

We use largely cloud-based systems. We’d recently run an update on our remote desktop software and rolled it out to all users after 2 weeks of testing. There were teething issues, but overall it was a 10% failure rate. Could’ve been worse. This update causes display issues and other minor inconveniences.

Get a call from a lady who’s always been problematic. Stuff like shouting at us when she can’t plug her firm laptop into her AIO home PC and using that as a second screen, not charging her laptop and phoning us asking why it doesn’t turn on, you know the drill. Picnic.

So, she calls the helpdesk. I pick it up. Connect to her session, get through a bunch of diagnostic questions and stuff. Figure out the issue.

Me: “Sorry, unfortunately you’ll need to pop down to IT and get a replacement laptop. This could be fixed if I were there in the office with you, but unfortunately I won’t be in for another 2 weeks.”

ID10T: “Oh i see. Is there no way you could do it for me now? I have [urgent task] to get on with that I put off for a week.”

Me: “Sorry, unfortunately not. I’d need to do it in person.”

ID10T: “Oh ok right, that’s because it doesn’t have a mainframe, isn’t it?”

Me (absolutely fucking dumbfounded): “Um, uh, sure? Pretty much. Just go down and grab a replacement from [colleague].”

ID10T: “Awesome, great! Will do!”

Queue Cue 10 minutes of silence after the phone goes dead. Completely dumbfounded. I’ve actually not heard anyone say something as stupid as that before.