/r/talesfromtechsupport

Photograph via //r/talesfromtechsupport

Welcome to Tales From Tech Support, the subreddit where we post stories about helping someone with a tech issue.

> FILTER POSTS BY LENGTH <

All Short Medium Long Epic None

> WELCOME TO TFTS! <

Hi, Everybody!

TFTS is where we post our amazing Tales From Tech Support, including but not limited to:

  • Incredible Feats of Networking Heroics;

  • Tech Troubleshooting Under the Direst of Circumstances;

  • Unsolvable Problems Cracked by Sheer Genius and/or Pure Luck;

  • Moral Support after Having Dealt with Difficult Clients;

  • And of course, Stupid User Stories!

There's a bit of a lull in the queue just now, so kick back, grab a cold one from the secret tech fridge behind the server rack, and share your best tales among friends here at TFTS!

> TELL US YOUR TALE! <

Rule Ø : Your post must be a written story about tech support. Please do your best to make it interesting, readable and concise. One post per 24 hrs please.

Rule ∞ : Don't be a jerk. Just don't. We get enough jerks at work. If you can't participate without being one, we don't want you here.

In particular, Reddit Truth Police will be banned without warning. That will happen.

> MODERATION INFO <

  • Posts and comments that do not meet these guidelines may be moderated.

  • Breaking Rule 3 or Rule 7 may result in an instant ban from the sub.

  • Visit the TFTS Wiki Page for more details about our moderation policies.

  • Please feel free to message the moderators with any questions.

  • DO NOT USE CHAT OR PM.

> COPYRIGHT POLICY <

> SPAM FILTER ALERT <

If your post does not appear, or vanishes after posting, it may have been flagged by Ye Olde Spamme Fyltre.

Do not repost the same thing again, that just makes it angrier.

Send the mods a quick message instead.

You may also review our lists of Things That Aren't Tech Support and Officially Retired Topics, just in case it wasn't the spam filter after all...

> SISTER SUBREDDITS <

> TALE SUBREDDITS <

> THE BEST OF TFTS <

> TFTS ESSENTIAL LINKS <

> BYE EVERYBODY! <

TFTS Top Tales - The Complete TFTS Archives

/r/talesfromtechsupport

770,315 Subscribers

337

But it's going to blow up!

Around about 2008, I'm working tech support for a large health insurance company. I get a call from a distraught young claims worker telling me that she needs a new PC. Hers is making a horrible noise and won't boot up, and she swears there was smoke coming from it before she turned it back off.

So we go back and forth, I'm just trying to get her to turn on the PC for a second so that I can hear the noise over the phone and diagnose for the site tech. She adamantly REFUSES to turn it back on because she doesn't want to risk a fire. I'm thinking; horrible noise and smoke, she's probably got a bad case or power supply fan, or maybe the bearings in a hard drive. So finally after she calms down a bit, I talk her into turning on the PC 'just for a sec' so that I can possibly help, or at least give my diagnosis to the floor tech.

She hits the power and there is no strange noise for a few seconds... Then it starts. It was a cacophony of beeps, BEEDABEEDABEEDALEEEP! etc...

I know that noise. I know exactly what's causing it... (Well, best guess anyway with a bit of experience thrown in)

Me: Mam, where is your purse?

Her: Excuse me, but what?! My purse?

Me: Your purse; exactly where is it at this moment?

Her: What? What the hell are you talking about?! MY PURSE?! WHAT? I'm calling tech support to get a new computer and you are playing stupid games? What is wrong with you?

Her also: Pause

Her: Click! She hung up...

Yes, her purse was sitting on her keyboard. I can't possibly verify that in anyway, but the hangup and no callback tells me I was spot on...

I have many tech support stories I could share, but that one was the funniest to me, and has gotten me the most laughs over the years.

The one that made me the maddest:

Got fired one time after I accidentally embarrassed a CEO. Drove 3 hours to plug his PC in after the cleaning crew caught his power supply plug with the vacuum cleaner. He swore up and down that he had followed the troubleshooting steps that I was dictating to him. I walked in, saw the plug from the door, plugged it in, turned it on and said "There ya go sir, have a good day" And I left.

He called my manager and told him that I no longer have a job, 'or else'. I really don't miss that place.

38 Comments
2024/06/08
13:10 UTC

221

The curse of being a Youngin’

Hello, welcome.

This is more of a rambling complaint post.

Just as a bit of a preface, I’m in my early 20s. Worked in tech for a bit over 2 years. No degree, only official qualification in IT is a GCSE (UK exam for 15-16 year olds). Working on qualifications like CompTIA Net+, Sec+, ITIL, Linux, etc, and done a whole load of IT stuff at home like a NAS, gaming rig, and fixing shit, but whatever. Job title’s Junior Technical Support Engineer. Lot of words for a kid who hangs out on the phones.

Couple months ago, I walk into the server room in the office. It sounds louder than normal, one of the racks sounds a bit aggressive and unhappy. No real way to describe it other than that, just that it didn’t sound right. Autism helps me out there, I can just feel the unfamiliarity of the sound in my bones.

So, I walk out, carrying the bits I went in there for. Walk up to my manager.

Me: “Hey Man(ager), one of the racks in there sounds kinda loud and aggressive, can we open it up and have a look?”

Man: “Loud and aggressive? What do you mean?”

Me: “I dunno, just… not quite right. Sounds like it might be coming from a UPS.”

Man: “No idea what you mean, but whatever. I’ll take a quick peek.”

Following a 5 minute audio and temp examination with ears and hands, we determine it’s definitely the UPS and it’s definitely hot. Outlet air’s nice and toasty. Check the stats on our online portal, temps are a little high but nothing major.

Me: “It’s possibly a short circuit or a battery issue, should we open it up?”

Man: “No it’s probably just a broken fan or something working too hard. We’ll log a ticket and get a tech to come out in the next few days.”

Cool, weird analysis, but not my problem. Server infrastructure isn’t my thing, I just pick up the phone for people who can’t right-click.

Next day, get in early, and Manager’s already there. Not good.

Me: “Hey Man, why you in so early? You don’t start for another hour, and I thought you were WFH today?”

Man: “Hey, yeah we got a call from Maintenance at 5am with a notification the rack’s running too hot so I had to come in. I think it’s the A/C, the repair guy’s been working on building A/C all week. The air distributor should usually move up and down, but it’s just staying still.”

Me: “I don’t think the air distributor thing usually moves, and even if it did it shouldn’t make any difference. I’m still pretty sure it’s a short circuit or a battery. I really think we should open it up and check it out.”

Man: “Nah it’ll be fine, we’ll wait for the tech.”

Me (with contempt): “Yes boss.”

Doesn’t sit right with me, but I’m not authorised to open stuff up, or override my manager. I drop it.

Rack gets louder through the day, temps keep rising, A/C works harder than normal, room gets louder.

Me: “Sure you don’t wanna check it before we leave? Just in case?”

Man: “Yeah it’ll be fine.”

I wake up the next day. It’s a day off for me, nice little bit of annual leave just before my birthday. Spend a couple hours chilling, then realise i left my work phone on.

As I go to switch it off, I notice a text in the groupchat.

“””
Hey all,
I was called in at 5am today by Maintenance as one of the batteries in the rack 4 UPS leaked overnight.
It’s now running on dirty power until fixed.
X system is not operational, we’re dealing with it. [Colleagues 1 and 2] can you deal with phones, [Colleagues 3 and 4] join my meeting later, [Colleague 5] don’t forget to contact [Supplier].
“””

I’m a junior, not a moron. Please listen to me sometimes. I promise I’m not as dumb as I may seem.

14 Comments
2024/06/07
17:08 UTC

515

"I'm looking at it right now, and there's no screen!"

First time poster, but I used to work in a call center for a cell provider. Many times, people would call in with hardware issues that weren't really our problem to fix since we really only dealt with the cell service, but today was a slow day and I was happy, so I felt like helping this guy out. (mistake #1) He was an older guy and was not familiar with technology, but "that's fine, I've worked with people that don't know technology really well before" I wish I was exaggerating, but I've literally had to describe to people what the power button is on their phone.

Anyway, this guy bought a new phone with us and activated the day before, so he was asking for help turning his phone on and setting it up. Now, I worked in phone support and did not have the phone with me, so I had to walk him through these steps verbally. I could see that, based on his IMEI, he had a Samsung Galaxy A14. I walked him through inserting the SIM card, then asked him to hold the power button to turn the phone on.

When he asked where the power button was, I gave him the answer I've, unfortunately, had to give to more people than I would have ever believed before working TS. "If you're looking at the screen, it will be on the right side of the phone, you will see one big button which is the volume button, and one smaller one. That one is the power button."

What I wasn't ready for was his response. "My phone doesn't have a screen." "You have the Samsung Galaxy A14, right?" I replied, even though I knew he did, because the steps to insert the SIM were all followed perfectly, but I wanted to make sure we weren't working with a flip phone and had the wrong IMEI. When he told me he had that Samsung smartphone, I told him it did have a screen and the whole frontside of it was the screen. That was when he yelled back at me, "You can see the phone I have, right? (I had mentioned that we had the IMEI and that was how I knew what phone he had) I'm looking at it right now, and it has no screen!"

I really wasn't sure how else to describe the screen to this guy over the phone. I almost decided to just have him go to a neighbor or tech support place to just do it for him, because helping him over the phone was clearly not working, I'm not sure if it really was that he was that oblivious to what a phone screen was, or if he didn't want to follow instructions and wanted me to press our "magical make-it-work button." I decided to just have him try to describe what he was looking at on his phone right now, and he said, "Well, it has three circles and then it's completely flat."

Honestly, the call was silent for way too long as I tried to process the fact that this guy was looking at the back of his phone the entire time and decided to yell at me instead of turning his phone around because he could not see the screen. After getting him to turn it on, I just told him to follow the on-screen prompts to finish set-up of his phone and ended the call. I knew there were tech-illiterate people, and I tend to be fairly patient with them, but yelling at me that there's no screen on a smartphone while you're staring at the cameras? really?

43 Comments
2024/06/07
12:33 UTC

329

The Dumpster Fire of a Teams Meeting

This is just a couple of years ago. I work with the Help Desk team currently but I have a lot of experience in team leadership, administration, information security, development, and project management. So I am normally a liason between Help Desk and other teams providing advice and guidance. This is for a large fulfillment and logistics company.

A project comes in which is to build a brand new centralized reporting tool. This is to replace the loads of PowerBI, Excel, and Access DBs that exist on the network that use ODBC connections to connect to SQL databases. There is no standard at this time.

The Database Team has built out databases that are replicated from the Production databases called Reporting Databases. No applications depend on these Reporting Databases and there isn't much of a delay between the two.

The problem begins when end users that work on the warehouse floor ask developers for the password to the Production Databases to do this reporting in Excel. The devs think nothing of it. The problem is the account has administrative privileges so it could both READ and WRITE data. And now a regular Joe with a handheld scanner picking clothes for an order has god rights to these databases. Then their management creates a spreadsheet that lists all of the passwords in plain text in sharepoint.

Then they build these Excel reports that query every 5 minutes... on multiple machines, across the enterprise. This CRIPPLES the databases. So they want this centralized Reporting tool.

Now I'm aware of the use of these accounts. I spoke with the Database Team and they thanked me for telling them. They didn't know the full extent of the problem and neither did I at the time. They encouraged me and the rest of the Help Desk team to push users into running queries against the Reporting Databases. This however was difficult to enforce.

Okay now you have the background. Now here is the dumpster. The meeting begins. The Project Management Team, Reporting Team, the Fulfillment Teams, Help Desk Team, and Database Team. One of the heads of Fulfillment shares screen and begins talking about these reports.

The screen share shows some of the queries and it immediately pulls the attention of the Database Team.

Why the Production Databases? How did you get access? What accounts are you using?

Then here comes the flames...

The moment that Team realized that EVERYONE knew the administrator passwords, the inferno began.

Everyone sat quiet while the Database Manager was berating the Fulfillment Teams. My Manager and I both are having a good chuckle to the side. I step away to STRAIGHT UP POP POPCORN.

I come back to the meeting. This guy is seething.

He is asking questions such as...

How did you get these accounts? Who approved this? These passwords are in plain text for all to SEE?! You mean to tell me anyone can just... DROP A TABLE?!

Information Security Team gets pulled into the call. The Fulfillment Team Managers and Leads were stuttering as they could not begin to answer the questions. This manager was on a rampage. I could HEAR the veins popping in his forehead through his voice, accusing this team of causing a potential security breach.

He accused them of causing all of the outages such as application slowness, random disconnects, and data completely missing. That they were either doing this deliberately or accidentally out of ignorance.

After he was done, you could hear a pin drop.

His last words, "I'm revoking all access. This project is dead."

He then disconnected and took a week long leave.

Just typing this out has gotten me hyped up again.

TLDR;

Database Team becomes aware that users have obtained administrative passwords to the databases and the Database manager lights into offending teams before revoking all access.

59 Comments
2024/06/06
01:08 UTC

998

“I’m not an idiot and don’t need to be treated like one”

I have a customer that is about an hour away from us. They are a small office 3-4 people. Not much equipment there, a switch, firewall and AP. One day the battery back up died and everything went down. I was texting with the user trying to figure out what was happening. They have a power strip that was plugged into the battery that was housing most of the plugs, I eventually asked her to bypass the battery and just plug the strip into the wall. Still wouldn’t work…asked to send me a picture of everything. The next part is the actual exchange we had:

ME: “It could take a minute for the network to come back up.” “Are there lights on the equipment? “

EMPLOYEE: sends picture of equipment “What equipment” “No lights on on anything. Nothings working”

ME: “It looks like the power strip is plugged into itself, make sure it’s plugged into the wall outlet”

EMPLOYEE:”OK I’m not an idiot and I don’t need to be treated like one. The strip is plugged in to an extension cord that’s plugged in to the wall so it can reach everything worked yesterday including the strip so it’s not plugged into itself it’s plugged in where it’s always been plugged in. We’re probably you guys plugged it into.”

ME:”I’m certainly not treating you like an idiot? From the picture it just looked like it was. Are your monitors plugged into the power strip? Wondering if that thing is dead”

After a few more fruitless back-and-forths I decide to drive the hour out there and take a look. I needed to get a new battery out there anyway. Was there for a whole 30 seconds before discovering that it was INDEED plugged into itself. They were down for a couple hours when it was avoidable simply by taking the time to actually look at what they had done 🤦🏼‍♂️. I told her that it was plugged into itself and she literally said “oh” and nothing else. On the bright side, haven’t heard from her since then and it’s been over a year now.

96 Comments
2024/06/05
20:20 UTC

193

(lack of) UPDATE: "Can't be arsed to open a ticket for their "work stopping" issue"

#include Original_Post

Because someone asked if the excrement had hit the rotary air mover yet:

It's been a week since the original email now. There still hasn't been a ticket opened, as far as I can tell. I've searched the ticketing system for recent tickets opened by the original email author, and it appears nothing related to this request has been opened in my team's queue. It's possible something got opened in another queue, but if so I have not seen it.

There also haven't been any shit-stirring emails to me or to my group's shared email. There have been more "oh, add me too!" emails, but it does not appear that the email author has made a stink about it yet. The evidence suggests that about a dozen people can't perform this task without this access, but they aren't yelling at me, so I count that as a win for the moment.

28 Comments
2024/06/05
16:04 UTC

1,012

"I'm sorry, are you a technician or not?"

Nothing annoys me more than people who are rude to you when you're there to help them. Well, except for people who are rude to you when you are there to help them AND the problem is completely their own ineptitude and lack of common sense.

Today we got a message from a user saying Outlook wouldn't open. I remote on and I can see Outlook open on the task bar. I change monitors and on the 3rd screen, I can see Outlook prompting to select a profile. I assume she just wasn't sure what to do here so I chose the default profile and set it to always use this one to avoid the pop-up from happening again.

I briefly explain the issue to the user but she insists it still isn't opening and gives a fairly snotty response saying she's been unable to work since 2pm (it's 2:45 at this point). I tell her I had it open before I left but I can connect again to check. I connect and it's sat right there, as open as an Outlook application can possibly be. I ask her how many screens she uses - 3 is the answer. I tell that it is open on one of those screens and ask if there is another problem? She says no. I then say sorry, I don't know what the problem is. I then get the response "Sorry, are you a technician or not?". This ROYALLY pissed me off.

I connect AGAIN, screenshot the window showing Outlook is open and send it to the user. She insists she can't see it. I go to the display settings and show her that it is on screen 3. She says it isn't showing anything on that screen. I ask her if she can see the notepad I have opened on the screen. She says no. I ask her if the monitor is connected, she says yes. I ask her if it is turned on, she says no. I ask her to turn it on, she does and says she can see Outlook now.

The fucking audacity of some people to be rude to and criticise people for helping when they lack the basic brain power to do such rudimentary tasks astounds me. She's now my 2nd least favourite user.

EDIT - the 1st spot for least favourite user was a similar story, except the issue was with a 3rd party mail provider and when I tried to explain that it's not something we can help with he used the phrase "Do you not know what you're doing?". That level of rudeness is hard to beat, though I've had some close ones.

192 Comments
2024/06/05
15:01 UTC

539

We Don't Need no Stinkin' Version Numbers!!! (Part 2)

Link to Part 1.

Recap: I took over support for an application that had not been regularly supported for about two years. One of the first things I ran into was that the application did not display its version number anywhere. Thinking that this would be an easy thing to resolve, I found out it was anything but easy.

= = = = =

I talked to my boss about adding the version number to a the Help menu. He thought it was a wonderful idea and I was happy I had a very simple improvement to the codebase to start out my newly-assigned support position for our software. Within a few minutes, I had compiled the program and tested it. The functionality worked great and the version number was easy to find.

I had a question about the software repository and had to go ask one of the developers for the flagship product. Big mistake.

I explained what I had done with the version number, and he immediately told me I wasn't allowed to do that. Not "shouldn't do it" or that it "wasn't a good idea", but "We have orders why we can't display the version number."

Ummm. Orders? Like, from management?

The developer called over the team lead and told him about the grievous sin I had just committed. (Actually, since I had a question about the repo, I hadn't actually committed the source file yet, but you know, semantics.) The team leader scowled and reiterated, "Absolutely not!"

I asked why, but was told to stop meddling and go back and do my job. Something in his tone of voice indicated that I was almost persona non grata with that team. There was a real hostility toward me, more than just a "developer from one group digging through another group's code". I would later find out that the entire team had been somewhat embarrassed that they hadn't been able to get Version 2.0 out the door for so long, and they were ticked off that I had been tasked with cleaning up their mess.

I walked back to my desk and stopped by my boss' office to talk to him about the issue, but he was on the phone. When I opened up my email, I found two emails from two different clients, each reporting a new problem with the flagship product. I immediately replied to the technical admin for those clients that I needed the version number file date/timestamp from the application directory for the .EXE and .DLL files. I had to explain how to open the command prompt, how to change directories to the desired directory, and how to get a file listing into a text file, with a reminder to send me the resulting text file.

And then I made a note to check back with the tech admins in two or three days to see if they had gotten around to getting that information for me.

I had no sooner sent the responses when my boss called me into his office.

"BobArrgh," he said, "I just got off the phone with so-and-so. I hate to say this, but you are not allowed to update the code to display the version number. However, I've never heard anything so ridiculous, so I'm going to investigate a little more. For now, just sit tight and do what you can for the bugs."

It took him about 4 days to finally get back to me. He had had the temerity to go up the chain (I have to admit right here that he was probably the best programmer/manager I have ever had the pleasure to work under) and he found out the ugly truth.

Someone about two or three layers up in upper management felt that having a version number in the software was an indicator to the world that the software had bugs.

I stared at him as if he had bugs crawling out of his ears. I took a deep breath to gather my thoughts.

"So, we can't show the version number because people will think that our software isn't perfect? Let's look at that. The end users at the client offices know that our software crashes during the day, and they raise it with their tech admin and their client reps. Our clients know our software has bugs because the tech admins and client reps have written several nasty emails to their management and our account folks, raising the level of awareness that the bugs in our software are starting to affect their work. Our account folks have escalated the issues to upper management. Our management knows our software has bugs because that's the very reason I was assigned to fix this crap, right? So, by prohibiting the version number from being shown, the only folks it is being hidden from is from your Support Team. You know, the only people in the company who are responsible for triaging the issue with the client and finding the solution. Do I have that correct?"

He asked me to write it all up in an email and vowed to fight the good fight.

The good news is that once we had pointed out the fallacies with the underlying logic -- and by showing that it was literally slowing down our ability to respond with actual code fixes in a timely fashion because we had to wait for the client to getting around sending us the file listings -- we were finally allowed to display the version number in the software.

The bad news is that it took my boss 3 weeks to convince the powers-that-be that this was a good thing and would instill confidence in our clients.

How would it instill confidence? Because, as I had pointed out, everybody who touched our software knew it had bugs. But now, they could look and see that version 1.23.4 had had several repeatable issues and now we were on 1.45.7 and those problems were no longer occurring. And, hey, look! Version 1.50.2 is really, really solid!

So, finally, it all ended wonderfully. My developer and I killed all the bugs and did the Bug Stomp Boogie, and even made a simple improvement that led to a 20-30% increase in productivity and workflow efficiency. We regained the confidence of our clients.

Version 2.0 got released about 6 months later.

And, surprise, surprise ... in one of the top navigation menu options, there was the version number for all to see.

39 Comments
2024/06/04
21:01 UTC

304

We Don't Need no Stinkin' Version Numbers!!! (Part 1)

Back in the mid 1990s, I was a programmer for a financial services company. We had a flagship product that provided workflow management for our clients. Letters -- real letters, written by real people on real pieces of paper -- would be opened by the mailroom staff, scanned, indexed (Name, Account Number, etc.), and then the system would queue the task up to be handled by the processing staff. The processing staff would read the letter and then invoke one or more auxiliary applications to fulfill the request, such as initiating the Buy or Sell orders, updating account information such as Address Changes. The last step in the process was to send out a confirmation letter indicating that the request had been completed, or to ask for more information.

I had just led a project wherein I had redesigned the letter-writing program, making it capable of interfacing with a different mainframe ecosystem for different financial industries, and the team had moved on to work on the Next Big Thing. Our redesigned application was very successful and our clients were really happy with the new application.

As the rollout was nearing completion and support calls were diminishing, my boss was approached by his boss with a pretty serious problem. The development team on our flagship product was starting to lose credibility. The development team had been working on "Version 2.0", and they were perpetually "Just about ready to release it, any day now, maybe two weeks, four weeks, tops!" Sadly, that had been the status for about two years.

Bugs that cropped up in Version 1 were simply added to the equivalent of a "backlog", which was essentially a document that had a growing list of issues. The development team had to be entirely focused on developing Version 2.0, and could not be spared to address any of the bugs in Version 1 that were starting to hamper our clients' workflows. Remember, Version 2.0 was just a few days away from being done, maybe two weeks; four weeks if they run into any issues.

Occasionally, an issue would be so critical that one of the devs was side-jacked to fix the issue and to send out the updated .EXE and .DLL file to the client. Not "clients" ... just the one client who was being affected by that particular issue.

(Side note: My boss had a list of observations called "Things That Just Are", and the situation with our flagship product caused this addition to that list: You cannot do New Development and Product Support Development with the same team at the same time.)

Managing this DLL Hell was insane, since every client was essentially running their own custom application.

To try to tame this unwieldy beast, I was asked to prioritize the backlog and start fixing the bugs. At the time, I only had a passing familiarity with the flagship product. I knew what it did and how it did it, but up to that time, I had been working on the auxiliary application. I didn't have a lot of hands-on experience with the big beast.

I had developed a good relationship with the Technical Administrator for one of our clients who was just down the street, so I went over to get a fresh look at their operation and the problems they were having. So I showed up and he was going over some of the issues when he got a call from the floor: My application just died and there's a weird message on the screen. His response was, "Don't touch it; we're going to come take a look."

So we head over to the person's desk and I sit down and take over the mouse. I made a note of the error message, closed the dialog, and started poking around, muttering to myself as I poked: Ah, a "Help" menu in the menu bar ... let's see ... is there an "About" option? Hmm, what about in the Settings? Hmm, where do they hide the version number???

Being unsuccessful in finding the version number, I finally resorted to going to the application directory and got a listing of the .EXE and .DLL files. At least, that way, I would know the relative vintage of the software. Once I had my listing, I restarted the application so the worker could get back to processing her queue. Of course, the problem didn't happen while I was standing there. Eventually, I left to go back to my office, determined to find out what was wrong with the software, what needed to be fixed, and, oh by the way, where do we hide the darn version number?

Little did I know that I was about to descend into a world driven by logic that almost makes the claims made by sovereign citizens sound sane.

(Stay tuned for Part 2, coming shortly.)

Updated: Link to Part 2.

25 Comments
2024/06/04
14:24 UTC

259

Another brick in the wall

Cast of characters:

$Me: (Soon to be ex-)medio(c)r(e) sysadmin, PFY without the P, or Y. Mild streaks of BOFH.
$Company: A magical place that pays me to convert above-average quality coffee into configuration files.
$User: Narrative device. They are legion and largely interchangeable in this story.

It's a relatively cold Wednesday morning for this late in spring. We have just finished moving into a new location, and the ride has been bumpy to say the least. Nothing that by itself would warrant a post here, besides this one.

I'm opening shop at 8AM, and like any overworked and underpaid mook, want to start my day with a cup of life-saving bean juice. I've just recently signed my conventional termination, but still intend on mooching on every last drop of coffee I'm legally entitled to as part of $Company. Unfortunately, the mighty Font of Ink-Black Dark Thiccquid had other plans and proceeded to experience a mechanical failure (I believe something somehow jammed in the water path).

I don't think much of it. Too early and not caffeinated enough to get angy. I just grab a post-it from a nearby office, write "Out of order (water nozzle jammed)", pull out the backup grounds-and-filter coffee pot from retirement, and set a pot brewing. Still bleary-eyed, I go on to do my rounds and turn on whatever needs to be, letting the coffee pot work its thermodynamics-based magic process.

About 20 minutes later I come downstairs to reap the fruits of the machine's work, and run into $User, who had removed the out-of-order post-it note from the other machine and is staring confusingly at it while it fails to push any liquid through its nozzle. I fscking wonder why.

I calmly explain what's happening with the machine, put the post-it back in place, instruct $User to use the coffee pot instead, and just in case leave the broken machine unplugged. Should have unplugged it in the first place, but eh. $User proceeds to remove the almost empty coffee pot, put their mug under the drip feeder mechanism thingy, and watch a couple runaway droplets make their way into it before wondering aloud why it's not dispensing coffee. Goddamn I feel old.

One further explanation on the operating principles of the coffee pot later, I'm back at my desk, one steaming cup of coffee in hand, and get on with my day. Fast forward a couple hours later when I feel a renewed bean juice craving. Guess what's up downstairs.

It's a different $User, who also removed the notice on the broken coffee machine, went through the trouble of replugging it, and is just as dumbfounded when it isn't dispensing death-staving potions. Once again, I explain the situation, add some vague coffee pot operation instructions, and pour myself a (largely cold by now) cup while I'm here. $User proceeds to ask me what I plan to do about it.

Sarcasm got the better of me and I jokingly said that the coffee maker doesn't have enough networking capabilities to fall under my contractual dominion.

So anyways, the new coffee maker is now on our wi-fi for some godforsaken reason. FML.

42 Comments
2024/06/03
13:11 UTC

192

Customer doesn't understand the difference between a HDD and a SSD

Okay so this is my first story here, and english is not my first language so please keep that in mind.
Also this is probably going to be one of many storys over the next years as I am working basically in the tech support industry.
I work for a fairly old company that has been building and configuring servers for customers all over europe and even a few across the globe in china/usa and the likes.
One of our products is a standard Server Chassis with Hardware tailored to the customers needs with motherboard, CPU and NIC and any add on cards the customer needs.
This customer bought a fairly standard server with not really any special motherboard or crazy CPU and just a Network Card and a Storage Controller as an Add On Card.
We shipped it out to the customer about a year or two ago. All working fine with a few SSDs installed in the 12 SSD/HDD Slots. Operating System already installed.
Now there were still a few empty Slots left so we keep the trays for installing additional 3,5 inch Hard Drives in the Chassis as per company policy.
Now onto the not so experienced customer whom had bought an aditional Hard Drive, yes you heard that right an HDD which although not much are still slower than SSDs. Now as I said before we installed SSDs as per the customers orders (all Items are put in an invoice which is sent to the customer for approval and final Go for the order). So the customer knew full well what they were getting.
Now a couple weeks ago our Support Team gets a Ticket stating that one of ther drives in the server is substantially slower than the others and they can't work like this and what not.
So after about 20 Emails back and forth and the email chain growing and growing with pauses in between because of vacations and sickness on their side.
We finally get to the point where my colleague has no more idea and decides, you know what there's nothing wrong with the server but the IT guy from their company will not believe or accept that. So we go ahead and build a complete copy of their server, they swap the SSDs and HDD and send us back the "faulty" server.
After some time and the email chain growing to about 70 Emails, which is a hassle to read through in order to find out what exactly the problem was in the first place we get the server. I am tasked with testing it by basically letting it run under full load and testing several Drive Slots at the same time to see if anything is broken or has any errors.
And by full load I mean full load, that server was running for several days under 100% CPU, 100% RAM and maximum Performance each Disk could achieve. And no issues, no errors or any sudden reboots or slower speeds even under highly unlikely 100% CPU load.
So now we have spent our time and effort fixing a non existent problem. Afterwards i then found out through my colleague that the customer installed the HDD and that they had a talk about the slower speeds of HDD and that its likely the problem the IT guy from the other company was reporting. But he was to shy or embarrased to admit via email or on the phone with my colleague.
Needed context for this story: The server we are talking about cost the them about 10.000€ plus about 5000€ in Service Fees for our Support and advance replacement if needed.
We on the other hand could buy these parts at about 7.500€ so we still made money but it's way less whenever something has to be replaced, because we can't sell these parts as new. We can only keep them in our stock, mark them as service and send them out once somebody needs a warranty replacement, or an advance replacement.

28 Comments
2024/06/02
16:32 UTC

105

Time-travelling sys-admins powers...?

No ma'am, it doesn't work that way...

In another episode of the Way Back machine, we travel back to my baby geek era of the early 2000s. In this chapter, I'm responsible for the miscellaneous collection of things all new tech support folks inherit including troubleshooting our fancy new payroll system. We've recently upgraded from green-screens to a browser-based system that is proceeding to absolutely baffle our payroll team with its "high-tech" interface.

The morning begins far too early, as they all do, and I can hear my phone ringing as I'm plodding toward my desk with my first cup of coffee. Caller.ID shows that its our Payoll Director, so I know its going to be a long day.

I put on my best cheerful Customer Service voice, and ask her how I can help this morning. She explains she's having trouble getting 'the payroll report' to balance. This is a payroll system, they're ALL payroll reports... so we go back and forth until I finally tell her to fax me the page that's wrong with the errors circled and I'll figure it out.

(Yes, we had email, but getting her to scan and attach a greenbar report was not going to happen at 9am. She understood the fax machine.)

Shortly after, I hear the fax machine chatter to life and head over to see what tales it has for me today. At this point in our story, its pretty ordinary. The printout has what I need to find the report, and the error doesn't take too long to find and fix for her. Triumphant, I call her back and explain the issue I fixed and let her know it should balance on her next run.

I go back to my coffee, content with my day so far.

Within about 15min, my phone rings again and I see its her calling. When I answer, I'm met with indignant yelling about how the report still doesn't balance, and I didn't do what I said, and <whargarble...blah>. The minute she said it still didn't balance, my spidey senses started tingling but I hadn't quite figured out why just yet. Eventually, she ends her rant with the breathless exclamation that 'It didn't change ANYTHING!!'

<insert comic strip light-bulb meme here>

What set off my bullshit-detector was the timing. Remember this is the pre Wifi-era, even an on-prem browser platform took awhile to do things. That set of reports took a good 30-45 min to fully execute. It hasn't BEEN that long. There's no way she's re-run them.

With some trepidation, I gently asked her to help me understand what she was seeing. What was she looking at that told her the numbers hadn't updated? (She does technically make sure I get paid on time, so I don't want to be too rude.)

20+ years later, I can still almost hear her yell echoing in my ears.

"MY PRINTOUT DIDN'T CHANGE!"

When I told her the system error was fixed, she re-opened the exact same copy of the report she had already. Without actually telling it to recalculate. When I pointed out the timestamp was from before I told her it was fixed, she said she didn't care and that I should have updated it anyway.

No, ma'am. I can't rewind time for you. You're going to actually have to run the report again. Yes, really. No, I can't run it for you.

11 Comments
2024/06/01
12:33 UTC

237

Internship, Dota and old school data mining

This was years ago. I was in my final stretch of the bachelor's degree and had to do an internship for a semester to graduate. I looked around but in the end I decided to just do it at the local internet Cafe I was hanging out at. I knew the owner, I had spent a considerable amount of time there so it seemed like the easiest path.

The owner, M., had two guys for IT support but they were working remotely. They had the PCs almost automated, loading a prepared image when needed or on each boot and they would come down every three months for some checks. So I thought I could learn from them, they always seemed cool. And he needed someone a little more technically capable than the current employees. I would do the usual, make coffee and tend the registry but also help customers or repair hardware when needed so that he wouldn't have to ship it to the pair of IT guys.

Everything was peachy. Besides the usual creeps and some GPUs that were failing (managed to "fix" some of them by putting them in the oven, having found an article about it) it was excellent. I worked nights, which was quiet and just counted the days to the end of the internship so I could go and find a real job.

At some point, about two months in my internship, things started happening. It was at a time where there was no reconnecting to an online game, not most of them anyway. And Dota was popular. Like the actual warcraft 3 map. So customers were rightly pissed when connections started dropping like flies. They would play and then nothing, network would drop them.

The IT guys immediately said we need to change the switch in the server rack room (more like a rack closet). But that was expensive and not a guaranteed solution. So the boss stonewalled until customers threatened to leave and go elsewhere. He tasked me to find one online which I got from ebay for half the price. It was shipped, received and the guys guided me on the phone on installing it.

For a few days it was okay until the issue returned. And I had limited experience so I did my best. Went online, tried several things but nothing. It went on for a couple weeks more until I had the idea to do some data mining. Nothing much, but I just started writing down details about when the disconnections happened. Soon it became apparent to me that it only happened when there were more than a number of customers in the shop. About 20 or so but it wasn't an exact number. I did some research and found a setting on the network card for each computer (they were loading an image, remember?), related to high stress or something. I can't recall the name or where it was, but essentially when the network reached a certain bandwidth it shut down the Lan port. It was just a bloody check box.

The IT guys fixed it, repaired the images as well with the new setting and we all went on to playing Dota like nobody's business. But I still use it in some interviews when asked how I handle problems.

(Excuse the formatting and grammar. I wrote this while waiting for a train on my phone.)

16 Comments
2024/06/01
11:35 UTC

304

A Unix core by any other name

Back in the 1900s, I worked tech support for a software company in northern California. Being on the West Coast, there was always one lucky person who's opened early for the Eastern Seaboard. That would be this Monkey.

One early Tuesday, I opened the phones and my first call of the day was in a panic. "MY SOFTWARE STOPPED WORKING!". (As you and I all know, things just didn't "stop working". We can figure out how to fix whatever you did. But you have to be honest with me.)

We started with the basics. Kill and restart the license daemon. "Okay yes. The Write program is working.". Great. Check the other two in your bundle please? "IT'S NO GOOD! I CAN'T GET INTO THE DRAW OPERATION!"

Okay. It could a be a corrupt file. Let's recreate your license and I'll email it. Lucky me also got to explain to a systems administrator how to kill a process. I did not ask how long they had been doing the work.

My new friend moved the old license to .OLD and placed the new file in its slot. Restart the license daemon and let's try those programs again. "OMG THE WRITING PROGRAM ISN'T WORKING!"

Okay. I've already asked twice, but I ask again, "Have you made any change to the system before this stopped working?" I get heaving silence on the other end for a moment. The SysAd finally days, "I'm done with you." and hangs up.

Less than 10 minutes later, another tech calls me. My friend escalated her issue to her manager. Okay. I'm a professional. I review the call (just in case), and pick up the tirade from a woman who is pretty angry that I would DARE question her people.

"Yes ma'am. I'm happy to try and resolve this. If you can tell what changes were made to the system, we can--"

"We haven't made any f--ing changes to the f--ing system!"

Ok. So we can try removing and reinstalling--"NO! This is not our f--ing problem! I want someone to fix this now!" So time to bring in my boss.

I have no idea what happened on the call. My boss came a few minutes (30) later and said, "We're flying Head of QA out. I can't find anything wrong from how you've handled it, so we're going in person."

Wednesday morning, I open the phones. Less than 40 seconds later the head of QA is on the with me. "Did you suggest reinstall?" Yes--well, I tried to. "Did you ask what they changed?" Yes--they said nothing changed on the system.

I sent her a new license file. Restart the LM and .... Yup--paint program isn't working this time. QA says, "Let me dig around. I'm call you back."

Sometime just before my 1030 lunch, QA called back. She is obviously trying not to laugh. "So. I'm reinstalling the entire program. Apparently, Monday night they rebuilt the core and removed a bunch of hardware initialization they weren't using."

"But! They said they hadn't made any changes to the system!"

"Well--obviously the core isn't part of the system!"

41 Comments
2024/06/01
02:22 UTC

714

Can't be arsed to open a ticket for their "work stopping" issue

My team works from a ticketing queue, supporting a subset of the systems at my company. We don't accept work requests via phone call, email, IMs, walkups... if you need to engage us, you open a ticket, via any of a number of means which are unimportant for this story. Of course, we also have a group inbox, but again, no ticket, no work.

Two days ago, we got an email to the group inbox: "Hey we can't save files. [screenshot of access denied error message] Can you please grant this list of IDs access." No mention of where these files are located, and my team isn't in the business of granting access to anything. But, if we did have a ticket with some info about where these files were, we could route it to the right people.

So, in a slack moment, I reply: "You haven't included any information at all about the host or the path where these files are located, so I don't have any way to put you in touch with the right team. If you put that info in a ticket to my queue, I can put it in front of the right people, but as-is, I don't have enough information to help you at all."

The client sends an email with the hostname and file share path, so I reply again: "Great, thank you. Now please put all that into a ticket, as I mentioned, and I'll assign it to the team that can help you."

Crickets.

What they want is deep into some other group's territory, and I would be stepping all over that group's toes and breaking all kinds of protocol if I did what they asked. It's possible (unlikely, but possible) that I could even lose my job if I did what they say the need. I know the group that can help. I asked, twice, for a ticket so I could help them. They just can't be arsed to spend two minutes creating a ticket.

I'm convinced they just want to say "we asked, but nobody would help us."

74 Comments
2024/05/31
16:13 UTC

81

Skee Ball Repair extravaganza! Part 2

This is the machine from Part 1.

I got everything on this Skee Ball machine working, other than the actual ball release mechanism. This particular one had a seesaw type thing that was as long as 9 balls, so if there were too many balls in the machine, still only 9 would come out. The solenoid was about 4 feet away from the end of the seesaw. The solenoid was supposed to pull the seesaw in the right direction to "tip" it to release the balls, then the solenoid would turn off and the seesaw would "tip" back to the resting position. But, the bar or lever linking the solenoid to the seesaw was long gone.

The solenoid was controlled by a triac, and ran directly on 110VAC from the wall, which was an odd choice in itself: usually, arcade machines have a lower voltage between 9 to 48 volts for motors and such, but there are exceptions, and this was one of them. The wires to the solenoid were of questionable integrity, so I got a regular power cord (because of the 110V) and wired it to the solenoid. While I was at it, I went ahead and tested the solenoid directly. It worked, except it buzzed really loudly. I wondered if that would go away when it was actually pulling something, or if there were rubber grommets missing (or worn out) where it was mounted.

I connected it to the control board (triple checking the schematic before I dared to apply power) and then plugged it in. I pushed the "start" button and I could hear the solenoid buzz. And, it shut off in a few seconds. So, the logic was controlling it properly. Unplug the machine.

At that point, with my measurements, I got some bar stock from the hardware store, and cut it down to size with a power saw. Then, I clamped it in a vice in such a way that I could hammer the end into a hook shape. I also hammered the other end into a hook shape as well.

I had also gotten some carabiners to connect my "hook bar" to the relevant parts, so it could be easily removed if something else had to be replaced. I hooked the bar in place, then I worked the whole contraption back and forth by hand to see if it worked smoothly. It moved okay by hand. Will it work?

I crawled back out from under the machine, plugged it in and pressed start. No response. What the fsck?

I looked at the schematic. The solenoid and the coin switch (now a start button) literally couldn't be further apart, electrically speaking. They don't run on the same voltage or anything. So, how could tampering with the solenoid prevent the start button from working?

Unplugging the machine and retracing my steps, I saw what the problem was. I had knocked the connector loose while I was down there. I put it back, got out from under the machine again, and plugged it in.

Voila! It works! The solenoid was still pretty loud, though. I asked the clients if that bugged them, they said no. Therefore, I left that alone.

Doing some research revealed that those solenoids get "loud" like that when they are wearing out. Not sure how or why that happens, but it does. Clients paid me for the work I had done, and I left for the final time.

Except, that wasn't the final time. I got a call about a month later, game won't start. Screen still lit up, but pressing the button didn't do squat. Drat. Well, I went to the hardware store and got another button. A doorbell button should do the trick if it was the actual button that broke.

The button itself was fine, 0 ohms when pressed, open when not pressed. After fumbling with the wiring a bit, the game worked again. I tried and tried to find the intermittent connection, but just couldn't find it. Finally, I said fsck it, and ran a brand new pair of wires all the way from the motherboard to the button.

I finally got the machine to work. Gave it a couple gentle knocks to make sure there were no more intermittents. I had my clients come play it, and it worked perfectly.

For my client's sake, I drew on the schematic what color wire I used to bypass the old wiring, in case they needed to fix it themselves or someone else worked on it in the future. And, I didn't charge for this trip because it hadn't held up as long as we had all hoped for.

TL;DR: Had to make my own mechanical part for a skee ball machine's ball return. Then intermittent wiring reared it's ugly head and tried to make me look like a fool, so I took a short cut (or shotgun approach?) and bypassed the original wiring (safely, of course!)

13 Comments
2024/05/31
01:45 UTC

84

Skee Ball repair extravaganza! Part 1

Not a computer or network device, but both myself and others have posted arcade stories here before, so here goes.

This is the same Skee Ball machine I was working on where I almost fried it by shorting power to ground, when I misread the schematic.

To start with, I made sure the machine was unplugged, even though it was "dark" already. Indeed it was.

First thing was to get the coin slot wires reattached and see if the circuit was still being made when the original coin switch was tripped. However, I had to make sure the wires to the bottom half were still connected to the right things and there were no shorts present. Accessing it was more like working on a car: I had to physically get under the machine. Once down there, I found the connector bearing the coin switch. The connector pins leading to that switch, tested open just as they should have. Speaker was close to 8 ohms. There were no shorts or abnormally low resistance readings, what more could you ask for?

Well, the connectors were all mashed up, but I was still able to get the pins to go into the correct sockets with no chance of shorts. Were there any chance of a short I would have spliced those wires separately, for testing, or added electrical tape in creative places to prevent shorts.

I plugged it in, still booted. I put a quarter in it, no dice. The coin mech was hosed, but we didn't actually need it to work: a button tapped to the coin switch will work. As a test, while the thing was already open to get my quarter back out anyway, I pressed down the coin switch. The game played it's beepy welcome tune.

The balls didn't come out automatically because the mech for that was broken. A rope had been tied to it by the previous owners to release the balls by hand. I jerked the rope all around but the balls didn't come out. Darn. After unplugging the machine, I discovered that what was left of the mech that the rope was tied to, was frozen solid. So I freed it up with WD-40 (well away from any wires) and put 3 in 1 oil on pivot points (they were all metal parts, built to last!) and indeed, that part of the mech now worked. So, I could at least get the balls out by hand now. But, whatever used to connect the solenoid to the ball release mech was completely missing. Probably broken many years ago, and fell out while moving the machine? Who knows.

Once I got the balls to come out, I confirmed there was a scoring issue. That was simple: broken switches. The "target board" just sat atop the ball guides and could be lifted up. Thankfully, they were just standard micro switches I could easily get.

Next was the display. These older Skee Ball machines actually used incandescent bulbs for each segment of the display, so they of course eventually burn out. Other things could have gone wrong as well, but it's better to try the simple stuff first. So, I tried to remove one of the working bulbs, but I couldn't, because some idiot soldered the bulb directly in! Yep, I guess the socket broke and the previous owner removed the socket and soldered the new bulb in place! Guess he didn't know where to find a socket. They were thankfully still made, though. Well, at least they didn't do the same for all of them. I used a working bulb to test the sockets (along with the logic that actually turns the lights on) and all the "dead" segments just needed new bulbs. The client didn't bother having me replace the soldered-in bulbs with sockets because those bulbs still worked.

I could get the bulbs at auto parts stores, but they were expensive there compared to an arcade supply shop. So, the client, being patient to save money, went with the latter. I found an arcade supply shop that also sold those micro switches, and ordered those. One thing they didn't have was the same type of connector the Skee Ball machine had, but those were available on another site that had 'free' shipping at a reasonable price (they're usually called 'molex' connectors, if you care)

New parts arrived and then, after double checking the machine was unplugged, I brought the schematic and a pencil with me, along with my other tools, when I crawled under the machine. I had a headlight on me for seeing things down there.

The people who built this machine, didn't use the same wire colors between the top half and the bottom half. They just changed color at the connector for no good reason. Thank goodness I had the schematic. Worse yet, they reused wire colors not only on the same connector, but on the same ROW of that connector. Yikes! Even with a schematic, that was a monumentally stupid design. I had to wonder how many machines got miswired at the factory the first time around and had to have parts replaced as a result? Or, why the miswiring of countless machines didn't lead to a design change with different wire colors?

Nevertheless, this has to be done. So, I cut off one wire at a time from the old connector, crimped on a new pin, and put it in the same hole of the new plug, repeating the same process on the other side. That way, there was no chance of mixing them up. Then I put a little check mark on the schematic next to the wires I'd already done.

After it was all said and done I came back out from underneath. Time for power! Everything related to that connector worked.

This machine apparently was in a carnival or some similar place where prizes were paid out by hand rather than tickets: the machine had no ticket mech, but a stobe light on top and a key switch. When you got above the set score, the strobe light would come on and keep strobing until the key switch was turned, then the game was ready to play again. The key switch was replaced with a button by the previous owner, but the original switch was left inside the cabinet.

With a button already there, I wanted to see if putting that in parallel with the coin switch would work. I found that it worked perfectly fine, whether the game went into "win" mode or not. Had that not worked, I would have rigged the coin return button to somehow trip the coin switch to avoid drilling holes in the machine. (Drilling holes for extra buttons and switches is something arcade collectors frown upon, and it might reduce the value of the machine if my clients ever wanted to sell it)

After replacing the bulbs for the display, I had nothing further to do but make some kind of lever for the solenoid to release the balls. I took measurements and pictures and then left, with the machine "almost" working.

Stay tuned for the series finale, where I get the solenoid working and releasing balls like it should.

11 Comments
2024/05/30
17:48 UTC

355

In-house tech support: Where the procedures are made up and the proper channels don't matter

Part of my work is wrangling helpdesk for a product which my department develops, and which many other departments within our organization also use (alongside other commercial users). As you'd expect, there's a healthy degree of collegial support around the whole thing: everyone knows someone who's good with the product, and they'll ask them basic questions here and there. However, for more complicated stuff, helpdesk still needs to get involved, and from our perspective, the work really is no different whether we're helping a coworker or John Doe we've never met before.

The helpdesk operates during specific hours, and there's currently 3 people switching shifts on it. Evidently, the last thing we want in this setup is for people to get in the habit of emailing those of us who work there at our personal work addresses instead of the @helpdesk address. It messes with our hours, it makes work harder to keep track of, and it can result in support requests sitting untouched for weeks if one of us is on vacation when they email us. So we do our best to be pretty strict about this, and people are usually understanding of being redirected to the @helpdesk address - for the most part.

Today, however, was not the most part. It's my day to work the desk, and I've just come back from vacation so you can bet your ass I'm logging out of helpdesk accounts on the dot and getting back to the piles of work waiting for me on other projects as soon as I can. Or at least that was the plan, until Outlook ever so generously slid a "$product question" email notification on my screen.

Guess I didn't log out of the account? Ah, nope, someone just emailed my personal account. I don't know this person, don't know where they even got my info from, but okay: the desk is closed for today and my coworker is on it tomorrow, so I'll just redirect this person and then they're someone else's problem.

So I open and read their email:

"I have a few very specific questions, I would like to know if you can call me at this number when you have the time."

A few very specific questions is never a good sign, nor is thinking that those can be resolved via phone instead of email or the ticket form. Someone else's problem indeed!

"Hello. Please use the @helpdesk email or the user support form on the $product website for questions about $product. Thank you and best regards."

Pretty clear, right? Nope.

"Hi, my coworker said I could contact you. I think it's easier if you call because this is too long to write. If you are not the right person for tech support please let me know who is."

A lil bit presumptuous to just completely gloss over what I sent, but okay.

"Hello. The helpdesk is open between $hours. Several people work on the helpdesk, so we need requests to come in throught the proper channels so that we can reliably offer support without disruptions. I will let your coworker know about this as well. Tomorrow, my coworker will be available for you on the address I provided below during $hours, and you may also reach the helpdesk on this phone number."

There is no reply ... for an entire 15 minutes.

"Okay, then I will do it this way." proceeds to list their very specific questions

They didn't move to the @helpdesk address. Didn't even CC it in. I looked at their email signature footer and by some miracle, it says they have a PhD. In what, procedure avoidance?

I feel bad for my coworker tomorrow, but they ain't getting a reply back to this email from me on my personal account, that's for sure.

44 Comments
2024/05/30
14:14 UTC

975

12MB email signatures (Why is Outlook running so slowly?!)

I work for an MSP. We have some customers (including this one) who cannot afford downtime due to the nature of their business. They used to run on a self-hosted email server which was dying a slow death, so whenever there was even a slight blip in their emails going down or running slowly, our phones would light up like a Christmas tree.

We receive several calls one day to say that *everybody's* emails are running slowly and they are finding it impossible to work. Every email takes 10+ seconds to open and it is impeding their workflow.

I connect to a machine and test it out for myself, see the exact issue several times over, then notice that the issue doesn't occur on every email, only the ones with their signature in it. I also notice that their email signatures have changed slightly since I last spoke with them. I send a test email to myself for further analysis, at which point I determine (as you may have surmised from the title of my post) that their new email signatures are 12MB in size.

Their email signatures are a single image, no text. This has always been the case, but now they had a new design, thanks to a new member of their marketing department, who must surely have some expertise in Photoshop and should know that making an email signature 9000x14000px is ridiculous, right?

Of course not. So, the marketing department create a humongous template, pass it onto the office administrator who doesn't know any better, then task her with creating 100+ signatures for the entire business, including an instruction sheet on how to change your email signature. Cue every member of the company complaining about Outlook slowing to a crawl.

I explain the issue to the office admin who is handing all of these email signatures out, suggest that she speaks with the user in marketing who created the template, then distributes new (smaller) email signatures to everybody again. I even offer a few ideas on the most efficient way to go about this, but I never receive a response. I do, however, see users' emails begin to speed up over the course of the next week or so.

The strange part now is that every email signature seems to be slightly different. Slightly different resolutions, even some looking somewhat blurry. Eventually, User1 out in the field calls our office, saying he's having problems attaching his new email signature. I connect, ask him to show me where the file is, and he points to a PDF on his desktop, saying that he can't find the option to attach it.

I explain that "You can't attach a PDF, you need the image file. I suggest you speak with [office administrator] and ask her to send you this again in the right format." User1 says no problem, will do, I disconnect and we end the call. User1 then emails me + the office administrator, requesting the signature in an image format. Office administrator replies "That's the correct format, just follow the instructions attached."

It turns out that their apparent workaround to the 9000x14000px issue is the following:

  1. Recreate the email signature in Photoshop
  2. Not reduce the resolution of the signature at all
  3. Print them as a PDF, still in 9000x14000
  4. Send the PDF to the relevant user with their signature in it
  5. Advise the user to open the PDF, open Snipping Tool, and take a screenshot of the signature in the PDF
  6. Save the screenshot, then use that as your email signature

This explains why the signatures were all different sizes and of different quality. I tried again to advise that this wasn't an efficient way to manage their signatures, but was met with silence in response. Eventually, the users changed their signatures using their internally-advised "method" leaving them all with mismatched email signatures.

At least Outlook was running better again for everybody.

152 Comments
2024/05/29
00:44 UTC

342

user needed help logging into 2016 outlook

So 2016 outlook has lots of issues. One issue is when MFA is enabled it will not neccessarily tell you that lack of MFA setup is the reason you get a broken login prompt.

I explained to a user that the reason they cant login is because we need to setup her MFA. She scoffs, doesnt want to do it, but says, ok w/e.

I have her login to outlook.com after giving a brief explanation of why it is necessary to have MFA, and that this process can actually make it easier for her to login if she uses it properly. I get her to the screen with the QR code FINALLY after she forgot her password, entered a new one, forgot it immediately, and then I set one manually and left it up so we could copy/paste later if we needed. I get to the point where it is time to install the MFA app. Keep in mind she is dragging the mouse away from me every two seconds to continue working while we are troubleshooting making this take ten times longer than it needs to.

"please go ahead and unlock your phone and look for the MS Auth on your phones app store."

"i have to install an app on my phone too? I'm not doing that, they don't pay me for my phone at work. This is too much. I just won't read my emails until they fix this."

Me in a sickly sweet voice, "Ok! Well let me know!"

i explained ad nauseum that she cant get around it. MFA is mandatory. 40 minute call to setup a 5 minute MFA and she decided not to do it when we finally get to the end.

Next user calls in, same company, same issue, except the user is in their 70s instead of 50s.

They follow my instructions, they dont try to go ahead, they dont do anything i dont tell them to do. We logged her in because she knew her password. We scanned the QR code. shes in. took me about 2 minutes after i remoted in, but honestly she didnt need me to remote in and she was competant enough I could have sent two screenshots and a short explanation and she would have been perfectly fine. I don't understand why some people have to fight so hard to do what their tech support is telling them must be done.

"

101 Comments
2024/05/28
20:14 UTC

722

The moment I learned paperwork doesn‘t mean much in the real world

I don‘t work in tech support but I thought this could be an interesting litte story for you guys.

This happened when I still was in school, our Computers had a program on them that would reset all changes made after logout, so we had to save all our work on the schools server. For some of my classmates this was somehow already impossible to understand but this is just background info. The point is that this program needed a license that was paid by the city and they just paid when the old license ran out what causes that for a few months of the year (I live in Germany and as you might know cities and IT are no friends) every pc shut down after 20 minutes and deleted all unsaved work.

Everyone just accepted this and occasionally lost their work, so I made a simple three line powershell script that would prevent the pc from shutting down. This was the first time the license ran out, the second time the school hired a „Microsoft trained datacenter expert“ that tried to solve the problem, while we were waiting for the new license. My teacher knew that I made this script and told another teacher working with the expert in the second room. In the middle of our lesson they asked for the guy who made the script because they needed help. I was confused that they needed the help of a student, but Ok. So I switched rooms and this was when I realized, that the title „trained expert“ seems to mean nothing. It went something like this:

$expert: „You are the guy who made the script, right? We cannot recreate it, could you explain it to us?“

Wait, I shouldn‘t be the expert in this room, but I will give my best

$me: „Ok, I show you what I made, its just a loop, that breaks the three second shutdown, that the program starts after twenty minutes, by spamming shutdown -c every second.“

$expert: „Thats so simple I wouldn‘t have thought of this! We want this on a Thumbdrive and start it on every PC at the beginning of the lessons so that the students don‘t see the code and have no window with the execution to close by accident.“

$me: „Can‘t you just put it on the reset image on the server?“

$expert: „That does not work, we can not update the images this simple.“

So I tried something on the PC they were working on and the moment I opened the powershell IDE the proclaimed expert asked me what powershell is and it took me a second to understand that he was not joking. When I tried to execute the script, the test PC blocked execution from external drives and after some testing I found out that powershell files were blocked by windows default security policy an those machines, but not batch scripts. I was not able to change the policy by script, so I told them, it would take me a few days to come up with a solution to bypass windows security and left.

The same evening I had a working script that would create a powershell file on the system and execute it hidden, the script was still not that complicated, but when the „expert“ saw it, he did not understand anything of it.

After this I understood that an expert on paper can still be incapable of real world tasks as I already read many times in this sub.

TLDR; Microsoft trained expert, didn‘t know Powershell and windows built-in security so he had to ask a highschooler for help.

93 Comments
2024/05/27
15:59 UTC

706

Encyclopædia Moronica: R is for Reconfiguration

It's a Monday.

Due to a series of coincidences and random occurrences that run somewhere between "a run of incredibly bad luck" and "a witch has cursed you", I have been on call now for approximately four months, while my neighbouring region has been effectively unmanned for almost the same duration. I haven't exactly been covering two regions constantly, but there have been multiple occasions where I have been asked to leave my area to work in the unmanned adjacent one; including still receiving calls about issues requiring urgent and immediate attention in my own region while I am multiple hours distant.

This doth not please the technician.

On this particular Monday, however, I checked my queue to find only a single, regular, normal job. Yes, technically it was for a piece of equipment that had been down for several days before it was reported to the customer's internal help desk, and yes, that help desk had spent several days "investigating via remote connection" before escalating it to my Help Desk, who then sat on it because it was received on a Saturday, until I saw it on Monday morning, so the total time of equipment unavailability was rapidly nearing weeks.
I read the job description, and immediately, there were several likely options forming; equipment failure, disconnection (accidental or otherwise), software misconfiguration, just to name a few. Some of them fell outside my particular remit; but I would at least be able to investigate and potentially narrow down the fault to determine just who's responsibility it was to rectify it.

Drop the spares I might need in the back, insert keys in ignition, rev engine, and race off to the customer site while maintaining 100% compliance with and respect of all speed limits, local laws, and traffic regulations. It was quite literally the worst time of day for traffic, but I persisted nonetheless. It is what they pay me for, after all.


Finally, a scant forty minutes after I first received the job, I was on site.

ME: Hey Manager (MAN)! I'm here about the {equipment} being offline?

MAN: Oh yeah, that's been down for at least a week.

ME: Really? I only just got the job this morn-

MAN: Yeah, I forgot to actually call it in to the Help Desk.

ME: Oh! Yeah, that'll slow down the process a bit... I'm here now, so let me have a look and see what I can figure out, OK?

MAN: Sounds good! BOSS is around too, I think maybe he was having a look at it on Friday?

ME: Ok, I'll let you both know what I find out.

So, customer is aware I'm on site and why. Time to earn my pay; time to put the brain to use and figure out what's going on here.

The screen is blank and unresponsive. Cool, it hasn't just gone to sleep and no one knew how to wake it up... this time.
Power indicator is out. Pressing the power switch has no effect. Okay, no power to unit. One step further back the power chain.
UPS is turned diagonally in order to fit inside the tiny cupboard. Turn it forward and... no power indicator here, either. Press the power button, screen comes up for a second but immediately turns off again. In that second, the battery charge indicator showed zero charge remaining.

BOSS: Hey G! Any luck?

ME: I'm making progress, or at least, I think I'm making progress. MAN said you looked at it last week, what do you remember?

BOSS: On Friday, it turned on when I pressed the button, but then it would turn off again. Saturday, it wouldn't even turn on any more.

ME: That tracks. Let me check something...

While we'd been talking, I had visually traced the black cable that snaked from the rear of the cabinet to a hard-wired connection point on the wall. Immediately adjacent to that connection was a key in a lock.

ME: What are the odds...

I twisted the key to the right. Immediately, I heard a small click from the UPS cupboard, followed by a quiet beep.
I pressed the power key on the UPS. Immediately, the VOLTAGE OUT indicator showed line voltage was being provided, and the BATTERY CHARGING indicator came on.
I hit the power button on the equipment itself; the screen flickered to life at once and began to show the normal start up process.

BOSS: What did you do?

ME: I turned that key about an eighth of a turn to the right. I'm guessing it got bumped and the UPS kept the equipment online as long as possible. When you tried turning it back on, it only worked until the UPS battery got too low.

BOSS: Well... damn, that's been down for DAYS. How do we stop that from happening again?

I reached out, and dropped the key into his hand.

ME: Maybe store this somewhere other than in the lock; it doesn't take much of a turn to hit the OFF position.

BOSS: ...yeah, that'll work.


I returned to my vehicle and started the close out paperwork.

Time to site: 30 minutes (2x 15min block)
Time on site: 15 minutes (1x 15min block)
Equipment used: None
Return to base: 30 minutes (2x 15min block)
Description: Investigated report of offline equipment. Discovered UPS not providing power. Confirmed UPS battery discharged. Confirmed insufficient input voltage to UPS. Reconfigured input voltage connection to restore correct voltage. Confirmed UPS battery charging. Confirmed UPS output voltage correct. Confirmed equipment started correctly and online. Informed customer of issue and resolution steps. Observed successful operation of equipment prior to departing site.


That's a whole lot of words to say "I turned it back on at the wall."

61 Comments
2024/05/27
01:12 UTC

1,170

Welp, I'm retired now! Guess I'll delete everything

Hey all

A while back I shared a story about a guy who wanted the admin password so he could install some malware on his work pc. Not too long ago I had to handle his "retirement", so yeah, you guys called that one.

Problem is that I'm still handling a lot of tech debt from the previous guy who handled this client's infrastructure, this was a real piece of work. I'm talking 10yo server OS thats EoL 3 yrs prior to me starting there and nobody will fork out the cash to upgrade it, busted filesystem ACLs from a botched recovery of a ransomware attack several years prior, the client being firmly of the mindset of "it just works so dont touch it"... you know, just normal small msp sysadmin stuff.

Anyway, Client reaches out to me in a panic one monday morning. Turns out our favorite idiot "retired" and nobody thought to tell me to lock down his pc or make sure he didnt do anything stupid... like delete the entire shared folder.

Now about this system - The backups... dont work. I've been sounding the alarm on this for months and everyone just goes "yeah thats a problem, we need to bill the client for a new solution, ill get onto that hmmmmmmm never". Fortunately for this client, a few weeks back I had made a complete copy of his entire share folder to put onto a different drive, but never finished moving shares across nor had I deleted these files (he has 3 raid arrays, but someone in their infinite wisdom thought that the best place to put all company data was on a split partition on the OS array. I hope the guy I replaced got put in an asylum).

Fortunately I was able to recover most of the files for the client but he tried to blame me for this, which I swiftly 180'd with a dozen emails of me hounding my boss and him about how these backups are busted and I need this and this and this to fix them because your server is woefully out of date and everyone expects the latest and greatest software to run on decade old equipment and cost literally nothing.

Anyway, I'm looking for a new job, it's only a matter of time before this place gets hit by a ransomware attack and the client's stubbornness and my boss's eagerness to please is going to result in bus sized tire marks all over my torso.

(Quick aside to anyone who's about to comment about how I need to get my manager or my HR involved - this MSP is tiny. I'm signed on to be I.T. Support but I do everything. I might as well be self-employed. Trust me when I say the answer is to wash my hands clean and move on)

51 Comments
2024/05/25
17:38 UTC

394

Teaching the Unteachable - Outlook Edition

Disclaimer- outlook is awkward I know.

So our favourite New Hire (NH) has been in the job two weeks. My team has realised I am patient and can get her to screenshot instructions as she’s doing the steps.

One of the team “working from home” (WFH) points out the email is on the fritz to our “fearless leader” (FL). Tells her the unassigned emails 100+ in the shared team inbox have been disappearing and she can’t figure out what happened. Supposedly NH has no idea and her queue looks fine. FL comes to me and demands to know what’s wrong and did I touch anything? Nope, I’m not working in outlook today my focus is in a project.

So I go take a look…….

NH should have a max of 10 emails, because she’s too new to handle more than password reset stuff (that’s another story). She has over 100 in her queue. Ok, cool found em. So I unassigned them, easy peasy and informed FL and NH it’s probably someone gave her an old category colour or something and it grabbed their stuff.

As I am standing at NH desk, I watch in horror as she deliberately assigns all of the unassigned emails to herself again. As in, right clicks the “none” category and goes down to categories and assigns herself, then fast clicks the warning telling what she’s doing and ignores the huge loading bar that opens. WTF! Realised oh ok that doesn’t do what you think it does. So I get her to the view - categories-collapse all groups and screen shot it with red circles around each thing. Explain what she did and why it’s bad. Fix it for her. Then I tell WFH, who informed me this was happening yesterday.

2 minutes later, it happened again. Ok now I’m losing my cool. I have shit to do. I go over and explain again “Stop doing that, it’s stuffing up everyone else.” And I shit you not she said “But that’s not what that does”.

big breath

Me “What is it you think it does?” NH “It shows me all my queue” Me “Interesting, since I told you it didn’t and we can see items from 2 years ago. You aren’t allowed to assign things to yourself yet, that’s WFH’s job for the first few months” NH “Ok, but that’s not what I did. I want to test it and see for myself”

I raised my voice at this point to sound probably like your mum when you just talked back or lied about leaving your room a mess. FL had left as soon as I stepped in.

“I watched you do it, we’ve fixed it for you three times. Don’t believe me? Do it again, because WFH hasn’t been able to do anything all day, since you keep screwing with the queue. You’ve been told to stop, you have instructions. Do it again and see how that goes for you.”

I walked away and sat down, inbox queue is fine, no more disappearing.

Remind me again how this woman has more “experience” or “qualifications“ than me. She doesn’t, but the fact she was hired over me speaks volumes about FL.

23 Comments
2024/05/24
23:37 UTC

422

Um, dude, you own a print shop

I manage a tech support team for MIS software designed for the print industry, it's SAAS & hosted "In The Cloud" so it's been work-at-home from day one for our entire crew, pretty nifty. Been doing it for 15 years, they treat me & my crew well and everyone is a self-starter so there's very little actual "managing" for me, mostly support and training for new customers.

Being in the industry every document we produce is a press-ready PDF file, same tool for everything including reports, invoices, packing lists etc as well as press impositions. This has never been a problem until today, when a newish customer who is still in the setup process emailed me directly (not through our ticket system, oh well) to reset his user login after multiple fails... it happens, I did it and responded. His response was:

"One question when I put work orders and packing list, can they output in color?  at this time the documents are coming out as only black-and-white"

When I asked for clarification since they're color pdf's and I confirmed that their color logo is there in the results he responded:

"I see PDF in color but when I print to a color printer, it comes up black-and-white"

Like, I'm happy to support you all day as needed when it comes to setting up and learning how to use our software, but no I'm not about to go digging through your computer's print settings to find out why that's going down...

I sent a polite response to that effect.

45 Comments
2024/05/23
21:26 UTC

486

Training the untrainable

Hi! First time poster on this sub.

So this one is still getting me in the giggles. Backstory, this woman got the job I was going for because she has no prior knowledge of how to do things in my industry. Basically my boss is sick of people in her team telling her "we cant do that because its against compliance law". So she's hiring "yes" people... well its about to bite her. I was asked to train the competition that beat me, ok fair enough I have better things on my career horizon so I'm not gunna be horrible. I started to train her on tiny excel reports.

First thing I notice is she's a hen pecker on the key board, this role requires intermediate skills. Ok fine that's weird, but sure all good. Next thing is the super fast clicking "ok" without reading anything. Yep I've been guilty of that. Third one? Well crap we've been screenshotting the steps all along, but she skips them or doesn't look at them properly and completely screws it.

When she runs a report without me beside her, she ends up generating a blank report. She proudly shows me an empty excel with headings only - she knew the report was supposed to have a list of names and dates for the particular item.

We run it again and I just keep repeating "Slow down and read your notes" getting frustrated by the 7th time I had to say it. I've taught children and adults with and without disabilities... I've never been this annoyed before.

By the time she gets it right, I'd practically hand over hand done it for her.

The next report she runs a few hours later and stuffs it from the get go. So I stopped what I was doing and ran it myself, sent it to her as a screenshot so she could see it, but not claim it as her own. and told her follow your notes until you get the exact same thing.

I still don't know if she'd hoped I would give her my report and say she did it or if she really is that bad. Time will tell. So far the report has not been produced, want to take bets?

72 Comments
2024/05/23
09:20 UTC

351

The curse of the iOS updates and the non-compliant phones.

So. I’ve posted here once before, but I feel this place may become my new home for complaints about user issues.

Basically, I’m 20. Helpdesk support. This is my first “proper job”. Previously supervisor at a large supermarket, and before then ran around carrying plates of food. I went from being yelled at by the drunken and hungry, to the tired football (soccer) mums at school pick-up time, to legal professionals. It’s an improvement, but 2 years here with the same problems from the same people is getting a bit mind-numbing.

Regardless. Onto the story.

We provide iPhones for all the fee-earners. Ok, fine, not a problem.

We also use an MDM. Quite detailed, lot of stats, simple enough. Shows us the iOS versions of mobiles and whatnot. In the IT department, we have to keep up to date with all the Apple CVE entries and mobile updates. This is a regular occurrence, and we’ve done so for the last 8 years (I would’ve barely been a teenager when they implemented this - ha, bet you feel old now).

We have the same process each time. We get an alert when a new iOS update is available, check the CVE, if it’s not too bad we give the users 2 weeks to update, 1 week if it’s not great, and 2 days if it’s awful and could wreck the company. We then send out a mass-email to the mobile phone users telling them to update, what’ll go wrong if they don’t, when they need to update by, and includes all the detailed documentation you need to update a bloody iPhone. Simple enough, my nearly-illiterate little sister could understand it. Day rolls around when we update our mobile compliance policy, consistently notice about 40 of the 150 users haven’t updated their mobiles… fuckantastic.

Remember, this policy’s been in place for almost half my lifetime.

Every. Single. Update. The phone rings off the hook when compliance gets changed. I’m the “new kid” on the block, so the phones are my responsibility.

$Newbie: “Helpdesk, this is [Me]. How can I help?”

$PatientZero: “Help! [Vital app] on my phone stopped working again!!”

$Noob: “Hey $PZero, what’s the exact problem? Are you able to sign in or not? Can you see your emails?”

$PZero: “No! It’s all broken! Nothing works!”

$Noob: “Ok, cool. When did this start?”

$PZero: “About 15 minutes ago!”

(I check the helpdesk. Take a look at the change request ticket. Last updated: …15 minutes ago).

$Noob: “Ok, cool, we did recently make a change that blocked all non-updated phones from working. You were definitely notified about this, and the email outlines all the problems that would happen if you didn’t update. Could you please update your phone with the included guide and try again, then let me know if it’s still a problem?”

$PZero: “I didn’t get the email!”

$Noob: (Flicks through to my email admin console) “Yes, you did, it shows as delivered and archived on my end. Regardless, we need to update your phone.”

$PZero: “I don’t know how to do that. Can you show me?”

$Noob: “$PZero, I’ve gone through this with you 5 or 6 times before. Yes, I can show you, but please keep in mind I can’t do this for you every time. Please come down to our office when you’re free.”

The user then proceeds to show up at the end of the day. Yes, very vital big important broken apps. Repeat this process 15 times, walk the rest of them through it over the phone, internally question yourself multiple times why these people are practicing solicitors selling houses and merging businesses, wonder how they can read a 150 page contract but not a short email with a small how-to doc, internally question their comprehension and literacy skills, then pack up and go home for the day.

Just to repeat it all for the next update.

Thanks, helpdesk. I hope one day to not return to you.

41 Comments
2024/05/23
01:12 UTC

325

The one with the TV remote

Hello again!

The other day I got a call from my dad. Nothing unusual about that, I'm the first call before any troubleshooting has been done, as always. Anyway. He has problems with his TV remote. He's trying to watch <insert important football thing> and the TV will not respond to hos commands. My mom wasn't home so he was basically helpless. I tried som basic troubleshooting over the phone, but quickly concluded this is an on-site problem.

I love my dad, and even though I don't watch football myself, I know this is very important to him. So I jump in the car and arrive in about 15 minutes. Still some game time left. Just to clarify, for no particular reason and for no points in this story: I'm talking about soccer, not American football.

I arrive and get the remote in hand. With him rambling about everything he tried, the new batteries, testing all other remotes he could find in the house, turning things on and off (gw dad!) etc. The lid under the remote is gone, so when I hold it, I feel the batteries on the bottom. They feel weird. So I turn i over, snap out the 2 AAAs and immediately see the error. Oh, you think he put them in the wrong way? Oh no, that would not be a story for this sub, no sir! The two batteries were still in plastic wrapping. He inserted to batteries that were wrapped in plastic like the ones you'd get with a new TV for the remote. I have no idea how he had those, but unwrapping the batteries, inserting them and voila! The TV remote works!

So we surf around to find the correct channel and we just can not find the correct channel. After som prodding we find out that my dad does not even subscribe to the correct channels! We didn't even stand a chance! Wow, what a twist! There can't be more now, right?

Well. We dug around some more about the particular game he wanted to watch. You may not believe this. It didn't even air tonight at all!

Two ways to look at it. I wasted 45 minutes. Or. I visited my dad, had a good time and got a great story. The second is my preferred one!

See you next time! : - )

**TL;DR** No, man. You have to read it all for all the funny plot twists ;)

27 Comments
2024/05/22
21:31 UTC

182

The Point Of Sale Files: The Buttonless Eftpos Machine

Hello again everyone!

It's been a very long time since I posted here. I figured I'd start back up again, for a number of reasons. For one this set of tales (yes implying more than one) are from what I would consider to be one of the worst jobs I've ever held. For a number of reasons I will describe and you will soon see. But also, I figured you lot would enjoy them and we can commiserate together. The second reason is it's been a few years since I left that job, so I figured it was relatively safe to post these tales now. Especially given most of my original coworkers no longer work there. Additionally, the tales from this place may be far more than all of the tales I've posted in the past combined. I'll post these if you guys are interested.

Finally, to also give a tribute to the unsung heroes (as rare as they were) in my time there and say here is a drink on me! You deserve recognition too you brilliant buggers. And I'm glad you lot went onto better things.

Now, on with the tale I say! I'll set the scene.

I had applied for this job out of anger at my previous employer who wasn't treating me very well. So I left after getting this new job. The job itself had a very minor pay bump too. Unfortunately, due to what was a very deceptive hiring process it turns out this new job was anything but advertised. To make matters worse, I'd get stuck here for longer than I intended. Part of it being a naive "I can make it better!" can do attitude I had, and a pandemic that got in the way not too long after I started. Due to my circumstances it meant it was safer for me to stay put given the crazy situation we were all thrown into. It wouldn't be until much later I would find something far better. I'll get into all this another time, but it gives some context as to how I came about this job.

On this day, a hot sunny day in the middle of Australian summer, I was carrying my trusty hot chocolate. Oh I should add. This job is what finally pushed me to coffee, despite the stomach issues it would give me. I walked into my first day on the job. No really, I literally walked into the front door smacking my face. It was locked...I tap on the glass, and see the HR lady that interviewed me walking to the door with a rather apologetic look on her face. She unlocks it and lets me inside, thankfully it's air conditioned. We will call her Laura.

Laura, was a bit taller than me (with heels, no I'm not very tall to begin with), with brown hair accented with highlights, dark brown eyes and a tanned complexion. She seemed to have this eternal smile etched on her face. From what I could gather it was genuine. Not that I'd trust HR as far as I could throw em'.

Laura: Sorry about the door Reaper! Stupid thing always plays up, it never seems to like staying open, or it likes swinging out wildly in a way that scares everyone. We've had 3 broken windows coz of that thing. Anyway, glad to see you made it, I'll take you to your desk to meet with your new manager.

To give an idea of the layout of the building, the foyer has a stairway that leads upstairs to the offices. The building itself was a converted warehouse, with an office upstairs and a warehouse downstairs. I was about to climb the stairs when...

Laura: Reaper it's this way! Down here to the warehouse

Mentally I cursed, I was sold on a desk upstairs when I interviewed. Apparently that's not where I was going to be stationed. I took a mental note of the red flag, hoping it was just a mishap. Maybe, I misunderstood something. Either way I followed Laura to the warehouse.

I'm led through 2 doors, the first had a code. The second had a key lock. As the final door parted I was greeted with a blast of hot air, and the presence of my new boss. He had also interviewed me. He was tall, and built like a rugby player. Easily would have been 6'2 to 6'4 in height, with thinning brown hair up top, sweat glistening from the top of his head, and one of the most luscious brunette beards one could have. Believe me, he should be proud. His name would make you think the man was Russian, he was very quick to correct me that he in fact was not Russian. We will call him Ivan.

Ivan: Good to see you Reaper! Your new desk and laptop are over there. Just login and get yourself setup. Unfortunately we don't have time to train you today, so I'll need to get you on the phone immediately to call some customers to resolve some issues. Any questions I'll be around if needed. Bit of a trial by fire I know, but I'm sure you can handle yourself.

Fuck. I'm getting a nasty feeling this is not at all what I signed up for. I look over to what is to be my new desk in this non-air conditioned warehouse, the desk had a fan mounted on the wall that would oscillate from side to side. Beside this desk was another desk, that had my new coworker sitting with a headset on, talking a customer through setting up something, I'm not really sure.

I lament the fact that not only am I having to talk to customers often again which I was told in the interview I didn't need to do often, but I have no aircon, and basically no training and thrown to the fire. Fucking great. I'm running out of hot chocolate by this point. Taking my last swig, I walk over to the desk. I greet my new coworker. He is very slouched on his chair. Arms outstretched. Black hair and a longish beard. Middle eastern descent at a rough guess, with dark eyes. I estimate he was about my height. He mutes his call and quickly greets me with a thick Australian accent, he's cool as a cucumber, we will call him Yousef.

Yousef: G'day mate. The name's Yousef, I'm the lead tech here. You must be the new tech?

Reaper: Reaper's my name, and yes I'm the new tech here. I briefly shake his hand.
Yousef: Good to see you. Seriously we need all the help we can get at the moment. We are absolutely slammed did Ivan give you any run down?
Reaper: Nope, just said to log in and start calling customers.

Yousef rolls his eyes, as if he's seen this before. He mutters to himself, just loud enough for me to hear.

Yousef: 'Fuck sake Ivan, of course he does that. Well no matter, hop on and search for this ticket into the YogaDesk ticketing system and call the customer. Apparently their EFTPOS machine came with no buttons. Call and find out why.

I log on to my new work laptop and quickly setup, most of it seems to be done for me. Just need to change some passwords and get into YogaDesk. I quickly search for the ticket and read the notes. The laptop itself is an old clunker that barely functions at the best of times. It was slow as all get up. And the underside had duck tape holding the backplate on it. The headset I was given however was brand new, wireless and was actually a decent bit of kit all things considered. I hook it up to the laptops bluetooth and proceed to hit the call button.

Reaper: Hit this is Reaper from POS co, Is Caecilia around? I was calling about an EFTPOS machine that didn't have any buttons?

Caecilia: Thank god you called! Listen, my new POS terminal I received from my EFTPOS provider doesn't have any buttons! I called them but they said to call you instead! Can you Help?!

I'm slightly puzzled here, and quickly check with Yousef, and indeed we don't supply EFTPOS terminals at this time. I press on, intrigued.

Reaper: Well, we don't actually supply EFTPOS terminals to our customers, but we do assist in pairing them up with the software. However, just in case so you aren't without a solution. Can you tell me if you see any buttons at all? An on button? Anything?

Caecilia: I'm telling you, I can't see any buttons! They aren't there!
Reaper: Ok can you tell me the make and model?
Caecilia: It's...um hold on let me check.

Some scratching noises are heard on the phone, the phone crackles back to life again.

Caecilia: It's a SuperEftpos Mark II.

Reaper: One moment let me see if I can look it up.

I go online to look at the EFTPOS machine in question. Sure enough, it has buttons. Lots of them.

Reaper: I looked up the model online, it indeed has many buttons. Are you positive there are no buttons?

Caecilia: I'm telling you there's none! I can't see them, I felt around too just in case. Checked the box. Nothing! Listen, is it possible you guys can send someone out it's urgent I can't take payments, I'm ready to burst into tears.
Reaper: Sure how about I check for you. One moment, I'll pop you on hold and ask.

I place the customer on hold and ask Ivan as Yousef was busy.

Ivan: Yes we absolutely can, however, there's a cost involved and needs to be paid upfront. Here's how you calculate it. And the button is in YogaDesk to send a link to them for payment. Once done, grab the keys for the van and you can go there and sort them out.

I hop back onto the phone and explain this to the customer.

Caecilia: Hmmmm, ok send me the details and I'll pay it. And you'll come out now if I do?

Reaper: Yes that's correct.
Caecilia: Ok, let's do it.

I take the payment from the customer and log the job in the NotJira slow Kanban board with guidance from the others. Before I walk away from my desk. Ivan calls out to me.

Ivan: Before you go Reaper, your new work phone is in your desk drawer. Get that running before you go.

I proceed to pull out the phone from the drawer, it's an older smart phone that had clearly seen better days. But before I can do anything, the screen quite literally falls off into my lap. Clearly a dud.

Reaper: Uh, Ivan. I think this phone is fucked...

Ivan: Takes one look at the phone. Fuck sake. Sorry Reaper, let me get you another phone.

Ivan begins to rummage through his desk drawer, Looking at phones of varying condition, until he finds one red in colour in reasonable condition. He hands it to me with a post it note.

Ivan: That's my phone number. Any issues call me, or call Yousef...in fact maybe call Yousef first, I'm busy. Let me know how you go. Now head off quick.

I proceed to head into the company work van. A slightly dented, older vehicle. Also it has seen better days. I hop into the drivers side with my laptop on the passenger seat. Start the engine, and begin driving to my first onsite.

30 minutes of driving later.

I arrive at the premises, it's a packed out cafe, I begin to make my way so I can speak to one of the staff at the front of the cafe.

Reaper: Hi. I'm the tech from POS co, are you Caecilia?

Greeting Waitress: No I'm not, but I'll get her for you. Hold on.

I wait a moment, and the Greeting Waitress calls out to Caecilia. After another moment, she walks out. She is wearing dark glasses, and approaches the counter. She seems to be scanning in my general direction.

Reaper: Hi, I'm the tech from POS co, you called about an EFTPOS machine with missing buttons?

Caecilia: Yes I did! Come over here and I can show you!

I proceed to walk around the counter, as I get closer, Caecilia places a finger on the EFTPOS machine.

Caecilia: See! No buttons! She removes her finger from the device and appears to look in my direction.

I take one look and immediately spot the problem and things dawned on me.

Reaper: Uh, sorry forgive my ignorance. Are you perhaps vision impaired or blind?

Caecilia: Well...yeah I'm blind. Why do you ask?
Reaper: Erm, not sure how to tell you this. But your EFTPOS Machine is upside down...
Caecilia: I'm sorry what???
Reaper: You're EFTPOS Machine. It's upside down, perhaps it would be better if I show...sorry if I put it the right way around so you can feel what I mean.

I proceed to turn the machine from face down on the counter, where the buttons were. To face up. I then gently get Caecilia's hand and place it on the EFTPOS Machine.

Reaper: Does this feel a bit more like it?

Caecilia: She goes bright red, clearly embarrassed Oh my god! I'm so sorry! I didn't tell you I was blind on the phone did I?
Reaper: No, you didn't. But in fairness...you did say you couldn't see them, and you mentioned feeling around the box. I just didn't twig. Sorry about that.
Caecilia: Not to worry! Sorry for wasting your time. We both start laughing at the absurdity of the situation.
Reaper: It's quite alright, would you like me to pair it to the software?
Caecilia: No that is ok, I can do it. Oh and...grab some cake before you go. It's fresh!

I proceed to watch as Caecilia confidently taps on the buttons of the EFTPOS and on the touch screen. The touch screen has been set to play audibly as things are pressed with accessibility features. Really it is quite impressive to watch, I do also suspect muscle memory plays a part here too. She finishes up the pairing process, and she offers me a free cake from the tray. I take a chocolate cake, she also offers a hot chocolate for the road. I accept. I proceed to take my notes and leave the cafe. I call Ivan from the van.

Ivan: So I just read your notes...the customer was blind?

Reaper: Yep. Couldn't see. And she had no staff at the till, so no one was there to tell her that the EFTPOS was upside down.
Ivan: Bursts out laughing. That's one for the scrapbook, good job for your first onsite. Come back here and give us a hand with these tickets and we can go from there.

I finish the day, exhausted, but happy I was able to help someone today. Unfortunately, things started to go downhill from here. Hindsight really is 20/20.

TL;DR: See no evil, hear no evil, speak no evil. Unfortunately, all of them failed at their jobs today. The blind was really leading the blind...

27 Comments
2024/05/22
16:33 UTC

117

This should bring back some memories...

I mentioned to a friend of mine, in casual conversation, that I liked to take apart old electronics for fun and see what made them tick (sometimes also fixing them) and she mentioned that her VCR no longer worked, that it would either shut down, reject the cassette, or worse, unroll the tape.

Never being able to resist such a challenge, I said I'd take a look at it and see if I could do anything.

This was well into the streaming era, so it would be relatively hard to find another VCR that still worked well. However, based on her description of the problem, it sounded like a belt was worn out.

You might blame the power supply for something "shutting down" on it's own, but VCRs will usually monitor themselves for abnormal conditions, and stop all motors if an error condition exists, in some designs all power is cut to the whole system. Some VCRs will then show an error code on the front panel display or on the TV screen (like "F1" or "E3" similar to an electronic oven or washing machine) It's supposed to prevent further damage to either the tape or the VCR, but unfortunately that doesn't always happen.

Long story short, during normal operation, these two "fingers" pull the tape out of the cassette and wrap it around the video head. The head spins, reading (or recording) in diagonal lines across the tape. This allows much more data to be stored on the same amount of tape than would otherwise be possible. The audio track is along one edge of the tape, and the frame sync signal is along the other edge, both of which are read by the "A/C head". During play (or record) the pinch roller does most of the work of moving the tape. The takeup reel just winds the tape back in aftetwards and doesn't really put much torque on the tape.

First things first, I wanted to confirm the symptoms and see what's happening with the top cover off. So, I unscrewed the top cover and set it aside, taping the screws to it so they didn't get lost. Then, I looked for the "power supply" which turned out to not be a separate board in this design, but a section of the main board. In fact, this design only had one board. Now I know what to avoid touching should I need to remove the board for some reason: The power supply capacitors can hold a charge for who knows how long, so they would need to be discharged or avoided when handling the board. Other than the power supply, though, VCRs are relatively safe to work on compared to other things that have high voltage in multiple places.

I plugged it in and put in an already damaged tape. First time, it was rejected and then the VCR turned off. This one had no front panel display, just a few indicator lights. I put the tape back in. That time, it was accepted and began to play. The takeup reel is turning. If the takeup reel stops while the capstan is still going, the tape will be unrolled and "spill" into the VCR because it has nothing to pull it back into the cassette. However, this doesn't appear to be the case.

However, sometimes, if I try to rewind, it just sits there and then shuts down when it "realizes" it isn't moving. Other times, stopping and starting play resulted in the takeup reel not turning, or nothing turning at all, and then shutdown.

Unfortunately, this wasn't the most serviceable VCR I had worked on: There was no separate "bottom cover" to remove in order to get underneath the movement. The movement would have to be unscrewed from above and then completely removed from the chassis just to get at the stuff underneath, including the belt that drives the reels. That's a relatively common thing to fail, so a better design would have made it more accessible. But, maybe the engineers were hoping the VCR would never fail and thus would never need service.

However, the front panel could be easily unclipped non-destructively, which gave me a better view underneath the movement. Operating the VCR in this manner wasn't a problem because the "buttons" on the front panel were nothing but levers that press the actual buttons on the circuit board, so I could still do everything I needed to, front panel or not. Plus I had the remote and it's sensor was also on the circuit board as well.

I tried an empty cassette shell instead of my damaged tape, which had the top cut out, and I grabbed the takeup reel gently and found that it didn't stop easily. This told me the belt was good. If the belt is old and starting to slip, this will show it. Another tell-tale sign of a bad belt is if your VCR still plays, but has trouble rewinding or fast forwarding.

I went ahead and let my damaged tape play all the way through in the background while doing other things, glancing at the TV and the VCR (with no cover) now and then to make sure it was still behaving. It worked fine the entire 2 hours. Then, when it tried to rewind, it instead started to UNWIND the tape all over the movement. Oops. I turned it off. When I turned it back on to try and rewind this mess, it tried to retract to the home position, but seemed confused as to where it was. It went back and forth a few times then overshot the "cassette in, tape unloaded" position and went to full eject instead, damaging the tape. That's why you keep a test tape around for, well, testing VCRs that need repairs.

The fact that it played for 2 hours without a hitch but then got confused when changing from one mode to another is a dead giveaway to another common problem. Anyone who's worked on VCRs should know what the mode switch does, and what happens when it's dirty or bad.

The mode switch will be somehow actuated by the movement, in order to tell the CPU what position the movement is in, so it knows which way to turn the motor. The CPU knows approximately how long it "should" take to change positions, and, if it takes too long, skips a step or whatever, the CPU will abort and either just shut down or try to retract everything to the home position. In some designs this also will show an error code on the front panel display.

On every VCR I had worked on up to that point, a "main loader gear" with the cams, levers and other thingies attached, sat beside or atop the mode switch and actuated it while turning. I had already found this gear almost immediately upon opening the covers. So, after I pulled the plug, I started marking things with a magic marker, in case they slipped out of position in relation to each other, then proceeded to unscrew the movement from the circuit board/bottom of the chassis.

I lifted the movement partially from the circuit board, exposing the mode switch. Thankfully, the shaft was keyed so that it couldn't be "mismatched" with the main loader gear. And, none of the gears tried to jump off track when I lifted the movement. This mode switch was a rotary one that resembled a gear, but nothing actually meshed with the teeth of it: it was turned by the shaft going into the main loader gear.

Now, replacing it would be a pain since I'd have to completely remove the movement and the circuit board in order to solder it. And, finding an exact replacement would no doubt have been difficult as well. But, what else could I do?

Contact cleaner to the rescue. On low voltage contacts and potentiometers, sometimes, you can get them to work reliably again by spraying contact cleaner into the openings of the switch in question. If the switch is a standard type that can be easily obtained it's probably best to replace it. But, if it's a "weird" one like this, you've got nothing to lose.

I slightly pried open the gap between the switch body and actuator, and sprayed contact cleaner into it. Then I gently turned it back and forth a few times, and then I turned it back to fit in the loading gear shaft. Could it be this simple?

The answer is, yes. I exercised every "mode" on the VCR with the test tape, and no more shutting down, no more tape salad. While it was open, I cleaned everything EXCEPT the video head drum with a Q-tip. (Don't clean the video head drums with a Q-tip: the fibers will get caught or snag on the heads themselves, possibly cracking them in the process.) For the video head drum, I just used one of those head cleaning tapes. I then watched one more movie on it that I hadn't watched in a long time, before I brought it back to my friend. For all intents and purposes, it seemed to work like it was brand new.

She was quite pleased that it was repairable, and I got a free pizza out of the deal. Since I didn't use any parts that I had to pay for, that was a good deal for me.

TL;DR: My friend's VCR became intermittent and it turned out to be the mode switch.

A note about contact cleaner: It should only be used on LOW VOLTAGE switches. A voltage or current level high enough to cause a spark, can ignite contact cleaner! So, it should never be used on, say, a car's ignition relay or a high voltage switch of any kind. Only use contact cleaner on "small signal" switches and controls.

8 Comments
2024/05/22
15:24 UTC

Back To Top