/r/talesfromtechsupport

Photograph via //r/talesfromtechsupport

Welcome to Tales From Tech Support, the subreddit where we post stories about helping someone with a tech issue.

> FILTER POSTS BY LENGTH <

All Short Medium Long Epic None

> WELCOME TO TFTS! <

Hi, Everybody!

TFTS is where we post our amazing Tales From Tech Support, including but not limited to:

  • Incredible Feats of Networking Heroics;

  • Tech Troubleshooting Under the Direst of Circumstances;

  • Unsolvable Problems Cracked by Sheer Genius and/or Pure Luck;

  • Moral Support after Having Dealt with Difficult Clients;

  • And of course, Stupid User Stories!

There's a bit of a lull in the queue just now, so kick back, grab a cold one from the secret tech fridge behind the server rack, and share your best tales among friends here at TFTS!

> TELL US YOUR TALE! <

Rule Ø : Your post must be a written story about tech support. Please do your best to make it interesting, readable and concise. One post per 24 hrs please.

Rule ∞ : Don't be a jerk. Just don't. We get enough jerks at work. If you can't participate without being one, we don't want you here.

In particular, Reddit Truth Police will be banned without warning. That will happen.

> MODERATION INFO <

  • Posts and comments that do not meet these guidelines may be moderated.

  • Breaking Rule 3 or Rule 7 may result in an instant ban from the sub.

  • Visit the TFTS Wiki Page for more details about our moderation policies.

  • Please feel free to message the moderators with any questions.

  • DO NOT USE CHAT OR PM.

> COPYRIGHT POLICY <

> SPAM FILTER ALERT <

If your post does not appear, or vanishes after posting, it may have been flagged by Ye Olde Spamme Fyltre.

Do not repost the same thing again, that just makes it angrier.

Send the mods a quick message instead.

You may also review our lists of Things That Aren't Tech Support and Officially Retired Topics, just in case it wasn't the spam filter after all...

> SISTER SUBREDDITS <

> TALE SUBREDDITS <

> THE BEST OF TFTS <

> TFTS ESSENTIAL LINKS <

> BYE EVERYBODY! <

TFTS Top Tales - The Complete TFTS Archives

/r/talesfromtechsupport

770,583 Subscribers

440

User with a non-issue that was 'fixed' created an actual issue.

We had a ticket come in today from a user who said their OneDrive and SharePoint syncing wasn't working. We remoted on and what the 'issue' turned out to be was that the status of the files were mostly set to online-only with the cloud icon.

"The guy sat next to me has green ticks though, why don't I have those?" I tried to explain the reason and that it wasn't an issue at all but he was having none if it. He wanted those green ticks on EVERYTHING. So I right-clicked the SharePoint library of ~250GB and made it available on his device.

After many hours of syncing, it was finally done and he had his precious green ticks. He phoned back to complain his device was running incredibly slowly. He had a 256GB drive which was now completely full.

36 Comments
2024/07/24
16:27 UTC

212

Computer Overheating?

I had an old lady B come in the shop today complaining that her laptop is overheating. She thinks it is full of cat hair and dust. So I pop the case open and it’s clean as a whistle. I turn the machine on and the fan spins up, appears to be working just fine.

I asked her why she thinks it is overheating. She says “Well this red light keeps coming on that says heat” I try to clarify but she couldn’t really elaborate and simply said “It’s not there right now”

So I hang on to her computer for a while, run some updates, a virus scan, what not. After a while, the windows news and interest taskbar widget changes from showing the local temperature (it’s been hot) to a red rectangle reading “Heat Advisory”

11 Comments
2024/07/23
02:44 UTC

102

Tales from an IT expert

Heya, Lennoth here.
I've spent the last 3 years at an IT service house. From customer support, over network management, client implementation- and training, to server integration and building full networks ground-up, I've seen a lot of IT. Most of the time while interacting with the customer. During this time, I've experienced a LOT of weird things, which I want to share with you.

  1. Not a single incident, but a common theme when interacting with customers via e-mail, or phone. I'm not sure how support is handled in other companies, so I'll just tell how it worked at the one I've been working at: We have support hotlines and support mailboxes. If a customer needs something to be fixed, they call the hotline, or write an e-mail to our automated support mailbox. Their ticket then shows up in all 1st- AND 2nd-level support employees. In general, an available 1st-level technician assigns the ticket to themselves and begins working on it, most of the time by calling back and asking a few questions about the problem. Due to how this work, customers may get an other technician for every ticket, or even multiple technicians at the same ticket, in some cases. This system ensures that new tickets are always worked at, as soon as ONE technician is available. But customers LOVE to have a favorite technician. As soon as they're contacted by a technician, some customers save this one technicians e-mail adress and/or phone number. Their issue is resolved and they're happy about the technicians work. But then the problem begins. The next time they want to open a ticket, they call this one technician who's contact information they've saved. Of course, this one specific technician may be unavailable, sick, at an apointment with another customer, not even working roght now, or even left the company. This is even worse with mails. The customer writes an e-mail and is waiting for a respond in our promised 8-hour respondtime. But by trying to reach a technician, they're bypassing all ticket-systems. So, if the technician isn't available, no one else knows about the customers issue. And making things worse, if the technician IS available and starts working on this issue with no active ticket, the customer is pretty much receiving free support from us. Big no-go. Because of this, we've introduced a zero-tolerance policy with those cases. First-contact is to be made via the official support hotlines. Support mails received in techicians mailboxes are forwarded to support, causing additional processing time. And no matter how often we try to explain this thing to our customers, they still love to have favorite technicians.
  2. Most of our customers are medical facilities of some sort, mainly rest homes for old people. And during my 3 years work, I've been at A LOT of them.This one time, I was working in a rest home for old people, replacing their out-of-date fire alarm system. Most of the time when we do work like this, the places aren't closed, so we naturally come into contact with the nurses and their residents. At this facility, the nurses and other staff of the place where extremely friendly to us. They offered us a room in the basement to store our stuff, another room with couches and furnished like your "old people" livingroom for breaks and even allowed us to get to their canteen and get food for free, at lunchtime. We got the same food the residents got and usually took our meals to the living room that was provided to us. One day, as I was standing in line to get my food, one of their residents approached me, with an expression somewhere between "please, help me!" and "where am I?". I have some experience with dementia and alzheimer and could tell that this guy had something in that spectrum, just from the look he gave me. As he came into reach, he grabed my arm with a strength you'd NOT expect from a man of his age and began to hastily tell me to bring him to his car. He kept going, saying that he was told to eat his meal, after which someone picks him up and get him home. For a moment, I was just as confused as he was, given I was CLEARLY wearing my work pants and even the jacket with the name of our company. Then I remembered some stories my sister told me, who's coincidentaly working as a nurse for dementia-patients. I kept calm and put my plate away, turning to the old man and ... made my biggest mistake of that day. I tried to explain to him that I'm just a technician and that he should get one of the nurses. Of course, he was to far away to understand what I'm saying and kept asking me about the car that's suposed to get him and that I should bring him there. We kept going back and forth like this for a moment, until another resident, an old lady with all her mental capacity intact, approached us and handled the situation much better than I did. She began asking him about the meal he mentioned, tkaing his hand and leading him back to where he came from. I didn't see how their situation ended, but from how she managed him, I guess it was much better then my experience.
  3. THIS GUY. Yes, my fellow IT engineers. I'm talking about THAT GUY. This one customer ... He's an aged man, somewhere between his late 50s, to mid 60s. He's the head of some industrial company he built himself, which was going extremely well for some time. But stagnation in both technical interest, and modernicing their systems is slowly degenerating their company for years. He's noticing losses in productivity, but is calling his employees to be the reason for this. After A LOT of arguing, he's hiring your IT company to help him build a more modern, stable and secure system. Which is easier said than done, given he has ONE server, which provides all critical infrastructure for his company. And this server has no backups. And it runs on a 12 year old OS, with no manufacturer support. And NO firewall runs on it, because of this. Despite this, he's the most relaxed man, regarding his network, while somehow being the most hastily man you've ever seen, if things don't work at the very first try. He's constantly forgetting admin passwords, so he resets them, without informing his IT service provider. He's ordering a state-of-the-art cloud-based network system which would fix all of his problems, just to cancel it last second, because he wants his servers on-premise (in his own house). He's not seeing the writing on the wall, even after his extremely outdated server is running on already borrowed time, with your technicians and IT experts doing whatever they can, to keep it going for just another week. Every week. For two years. He's constantly restarting this server, no matter how often you beg him not to. Because you CAN'T guarantee the server to properly startup, any more. AND he has a favorite technician, always sending his mails to this one guy, no matter how often you try to make him take the official support route.

EDIT: a bit more information for story 1, after reading some comments:

We NEVER give out our private contact data. Een giving out our personal business data is quite unusual. But when working in a 5 story building with the customers own technician running from place to place, people tend to give out their business mobile number, for easy communication. Also, we always give our names to all emails we write, as one does.

That's how information is passed to customers and begins to spread.

Also, we have an online ticket system, where customers can make their own tickets. But only a handful of them actually use it. Most prefer a more personal approach and call us.

But giving you all the benefit of a doubt, ware a quite small and relatively new company, so there's absolutely some stuff that could be done better, on our side

34 Comments
2024/07/23
08:44 UTC

275

It's up now, but is it fixed?

So with the current Crowdstrike debacle, I am sure a lot of you are working extra hard, just as I am. I don't support Windows in my company, but I support a software product that run on Windows servers, so my team and I have a complete crapload of work to do - not in fixing the Crowdstrike issue, but in verifying and doing minor fixes on our software.

Yesterday, we got a ticket from one of our client groups: "Please resolve Crowdstrike issues with these servers: <list of servers.>"

First of all, nobody in tech needs a ticket to do this at all. We're all running around with our hair on fire, fixing things as fast as we can. The ticket is redundant in its mere existence.

Second, the Windows team is working on this, not my team. There's not a damn thing we can do directly. When the Windows team gets the systems in the list repaired, a colleague of mine checks our bit, finds it all healthy, and closes the ticket - "systems are all good now" or something like that.

Today, the client team sent us an email - "Please confirm that Crowdstrike was repaired." I replied, "We're not doing the remediation on that, that's the Windows people. But if it is up, either it was never affected or it has been repaired." They wanted more confirmation - they wanted my team to go through their list of servers and confirm manually that the offending definition file had been removed. I just repeated, "Sorry, you'll have to talk to the Windows team, it's outside my area of support."

Just because my product run on the machine, I don't have end to end support of the machine. I frankly don't have the ability to repair the Crowdstrike issue on these machines, as I don't have permission to access the iLOs and iDRACs on the machines, and I certainly don't have access to the data centers.

61 Comments
2024/07/20
21:08 UTC

241

I need a zoom session

Sometime last year I had a member of staff at the customer company I was supporting, get in touch because they were looking at implementing a feature of our product into one of their designs.

Not quite sure why, as they could have read the documentation, but what the heck: I just pointed them to the documentation, giving the actual link. Job done, case closed.

A few months later, same person gets in touch again, as they are actually getting round to do the work, and they want a zoom session to be guided through it. Now given that I wasn’t overly familiar with this particular feature, but more significantly given the time difference between myself and this customer, I declined and instead pointed them at the reference example provided with our product, and the step-by-step guide that came with the example.

In the meantime, I followed said guide myself with the reference example, to make sure it actually worked. I was able to get the example design working ok. But the customer kept asking for an interactive zoom session.

So then I replied asking which steps he was having trouble with when following the example. No clear response to that one, except that they really wanted me to show them.

Eventually I relented, found a time slot to suit us both (without me having to be in work outside of my core hours), and I shared my screen on the zoom session, where it became apparent that they hadn’t even bothered trying the example design. I therefore set myself up so that on one side I had the example step-by-step guidance (which had been available to them since the beginning), and on the other side the actual example. I followed the guidance, pointing out each step as I went through them, and got the example design working.

Customer was happy with that, notwithstanding the fact they could have achieved the same thing by themselves (note that this was a senior engineer, not an inexperienced person), without wasting time for both of us.

I guess they really needed a zoom session.

21 Comments
2024/07/20
12:13 UTC

565

“If you told me there was a charge, I wouldn’t have bothered!!!”

TL;DR - A ridiculously obnoxious customer came into the computer shop I worked at once and a classic case of malicious compliance unfolded as he refused to pay to have his computer fixed.

They brought in a computer with a BSOD problem, and apparently completely ignoring the signage everywhere (including on the form he signed to drop it off) stating there would a minimum cost, was furious there was a actually a charge to fix his computer, upon his return.

“You’re a bunch of scam artists - you can’t expect people to see and read everything. I bet you don’t read the terms and conditions all the time either, do you? You should have told me verbally!”

Funny enough though, I had only literally just fixed it (it was a corrupt file which you just renamed and Windows fixed with a scandisk on restart), so the computer was on the bench, and actually still on.

The dude was still furious and continued to loudly declare statements like “This isn’t right! If I’d have known there was a charge, I wouldn’t have bothered.” Although he was really starting to piss me off, I suddenly realised I could be maliciously compliant.

So I calmly told the asshole that I would see what I could do and went into the workshop - straight to his computer, and simply renamed the restored/fixed version and and reverted to the original corrupt file. I then turned the computer off, unplugged it, and brought it out a short while later.

“There’s no way I’m paying for it, though!” He said as soon as I came out with it.

“My apologies for the misunderstanding, here’s your computer back, and there is no charge.” I said smugly, smiling happily.

“That’s damn right there isn’t. But you fixed it, right?”

“Well, it was fixed - but you made it clear you had no intention of paying anything to fix the computer.”

He was about to continue his rant when I just cut him off and continued.

“It’s an honest mistake - you somehow managed to completely miss all the signs trying to making it clear we’re a business, and you simply don’t understand that a business needs to charge people for their services to stay open. So in accordance with your wishes, I’ve reverted my work and I’m giving it back to you in the state you brought it in - which has incurred no charge to you.”

“Yeah, but for this inconvenience - I expect it to be fixed… you’re telling me it’s still broken?”

“Well, yes. As you literally just made clear, quite belligerently, you didn’t want spend any money on this computer to fix it. I’ve literally done, what you wanted.”

This confused him for a moment, and I could see he was about to simply continue his tantrum until he got his way, before I again cut him off.

“Unless you’re the scam artist, and never intended to pay for the repair - this is exactly what you wanted. We have literally no more reason to continue this discussion unless you intend to pay for our services, to have the computer fixed.”

I could see he was still angry, and was probably going to continue to be an asshole, but thankfully the phone rang, and I picked it up.

He then took the computer and cursing under his breath left the store… only to have his Wife drop it off to be repaired, at cost, the next day.

44 Comments
2024/07/19
11:59 UTC

331

"I have to IDENTIFY myself?!"

In the olden days, people apparently took your word for things, and a man's word was as good as his handshake! "Look into my eyes, and tell me I'm lying! I. Am. Johnny."

Well... In these connected, remote, globalised days, with GDPR's and cyber security, we use pass codes. And ID's. And badges. And numbers. Apps. Whatever you're trying to do, you can't just claim to be someone without any kind of plausible proof. If you don't ID, I can't fix your problem, and you can't get specific answers. That's just the way it is.

...

Try telling that to "Henrietta" (we will refer to User as such for obvious, previously stated security reasons).

I get a call from User, who is in a location where there's been a huge thunderstorm. I can see that, because the user number that's been typed in belongs to Henrietta. However, it is clearly a man's voice on the phone, and I doubt his name is Henrietta.

"Hello?!"

He sounds agitated already, and I take a breath and become one with the world - I'm clearly gonna need it.

"IT speaking, you're talking to UnintentionalAss!"

"What's happening with this?? I don't know what's going on, when are we gonna be back online?? I have a service number and all it says is that you're working on it but I want to know when it's fixed!"

I look at the case. We are working on it, but I can't relay any specific information if I can't ID User - especially if he's used someone else's information to contact us.

I'll just ask for identification, I thought to myself, like an idiot.

"Alright, Sir, to get to the next step, I'd like for some identification..."

Simple as that.

"You should get a notification to whomever's phone you're contacting us from, and they can just type in the code, and if they're not available to do that and you have their permission, you can answer a few..."

Not simple as that.

"...security questions..."

At first, there is silence on the other end. Then, Henrietta starts huffing and puffing like the Big Bad Wolf choking on a piggy.

"Identify myself...? IDENTIFY MYSELF?!?! HOW DARE YOU?? I... OH, LOOK NOW - YOU'RE SUPPOSED TO FIX MY PROBLEM, NOT SIT AROUND ON YOUR ASS DOING RIDICULOUS THINGS LIKE 'IDENTIFYING' ME!! OH, I GET SOOO FUCKING.....!!"

And like a big, dark storm cloud in the sky, Henrietta is gone.

I sit back and take a moment.

My question is...... When will people learn..?

69 Comments
2024/07/19
08:15 UTC

389

One needs a Password to log in?!

I need to vent.

The stage: Our blue-collar workshop workers have an AD-Account. They need the password to setup/use the mail on their company phone and their tablets. They also need it to log into some other things related to our network. We have some workstations that can be used by everyone in the workshop to do stuff.

When they receive their phone and tablet they get a one page instruction with the initial password that explains some stuff including that this is the "Windows password" and that it is also used to log into the PCs.

Now some of our guys are to be given notebooks. As they are not to tech savvy, we not only install the machines but also schedule a 1 on 1 session to show them around on the devices, help them customize things and answer any questions that may come up.

I've rolled out five devices this month, reminded everyone to bring their password - and each and every one of them did the surprised Pikachu face when I asked them to log in with their credentials. Each and every one of them was totally dumbstruck that they really needed their password to proceed. Each and every one of them did not know their password and declined my offer to reset it, because it would be to complicated to update it in their other devices.

I am THIS close to hurting somebody.

49 Comments
2024/07/19
07:04 UTC

104

Please take a look

I was doing tech support for a number of years for a specific customer company, and generally customer employees were ok to deal with, and reasonably competent (to various degrees). Except this one person who for some reason held a fairly senior engineering position (I suspect there was some nepotism involved).

So whenever this person hit an issue, they would send an email with a cut and paste (if we were lucky) or a screenshot (if we were unlucky) of the error message, and ask us to “take a look”.

Unfortunately given the complexity of the SW in question, that would not give us any information, and we would need (at least) the full set of log files. Even after explaining to them (by phone, email, PPT presentations, etc) multiple times where the log files were (really not that hard), they kept sending error log message snippets only. Also they would complain about response times, notwithstanding the fact that we use CRM SW that the customer has access to, so they can raise issues (and attach log files), so that anybody supporting this customer can pick up the issue (and so that we can search through previous issues to see if the problem is not new).

So eventually (after jumping through many hoops) we were given access to their network remotely.

So now please go take a look (when the next rise issue arises), but still by sending emails (rather than filling issues). Now I would have happily gone and taken a look, except that the issue occurred in a secure cluster within the customer network which is firewalled from the rest of the network, and we don’t have access.

This goes on for a while, and finally after jumping through more hoops (several issues later) we get access to the secure cluster.

So again, when the next issue arises, please go take a look. Except that the directory where all the relevant information is (Linux environment), is only accessible to users in a certain group, and guess what: our user IDs are NOT part of the relevant group.

So in order for us to have the level of access required, several IT people, engineers, directors, legal team persons, had to be involved, lots of paperwork required, because this person refused to upload log files.

And we still could not just “go take a look”.

14 Comments
2024/07/19
06:36 UTC

108

I switched to the new version of the SW, and now it keeps failing

I used to do tech support for a product that involved a piece of SW that has to run a complex flow with multiple interlinked steps. The way this works is that the user launches an umbrella control job, and this in turn spawns multiple interlinked jobs to the compute grid in the correct sequence. If any of the spawned jobs fail, the whole thing fails.

So one day I get one of the engineers at customer company complaining that they have switched to the new version of our SW, and the flow now always fails to complete.

As usual I ask for the log files, and I quickly determine that some of the spawned jobs are being terminated (as if by Ctrl-C, this is a Linux environment all the way). Now I know this customer has a compute grid (managed by LSF), where a job spawned to the grid cannot exceed the job requirements (e.g. memory usage, number of CPUs, runtime, etc) specified by the grid submit command (bsub in this case, as they are using LSF), otherwise the job in question will get killed automatically by the grid engine (no exceptions). So obviously first thing (as there didn’t seem any other cause for the job failing), I ask the user to check their grid submission parameters. After some back and forth they telling me that these are NOT the problem, that they have increased all the resources requirements and the flow keeps failing (roughly at the same point), so it MUST be a problem with our SW.

Not quite convinced of this, I ask them to send me the job status from the compute grid for the jobs that are being terminated. Now I know this is not easy to do, as they have no automated way to do this. Nonetheless I insist because according to the logs there is nothing that explains the behaviour other than external intervention. This dialog takes a while, and goes through different persons at the customer company. Then silence for a while, then I get a terse message that the problem is solved. I therefore enquire as to how it was solved.

It turns out that the umbrella control job (which was itself being run on the compute grid), was being submitted (by the customer’s script) to the grid with a very short runtime requirement. As soon as the runtime limit was hit, the grid engine would kill it. This would in turn trigger the control job sending a termination signal to all the other jobs it had spawned to the compute grid, hence the failure of the whole flow to complete. Once they fixed that on their side, the problem obviously disappeared. And they were so sure our SW was the issue (and getting pretty annoyed at me for not providing a fix).

8 Comments
2024/07/19
04:21 UTC

309

The iPhone Girl

Using a throwaway because I'm a weirdo. I really don't like to talk poorly about people that don't really "get" tech stuff, so I rarely tell stories about the users at my organization. This one, however, I feel like is truly one for the ages. This was a few years ago now, and I still think about it every once in a while. For reference, I was a lowly help desk guy with only a few years total of experience.

On this particular day, I had left for lunch just like any other. About 45 minutes later and I come back to see that my desk phone had 6 missed calls. I sighed, fearing the worst. 6 calls in 45 minutes seems pretty urgent. To my relief, after checking who called, it wasn't my supervisor or director. It was simply one of our office workers. However, I still wondered what could possibly have been so urgent that they tried calling me, and not just making a ticket like everyone else.

I call this person back, let's name her Sarah. Sarah, sounding really exasperated, says, "It's easier if I just come to your office and show you". A bit bewildered at this point, I agree. Sarah immediately comes over and shoves a pair of headphones in my face. "My headphones are broken." she says shortly and plainly. Honestly, I'm more annoyed at this point than anything, as it starts to dawn on me that these six missed calls were simply over a pair of headphones. Still, I do my IT thing and ask her first off, "What's exactly the problem with them?" I take them in my hands and don't really notice any readily apparent physical defects or whatever. The look that Sarah gave me next was one I don't think I'll ever forget. She looked at me like I had randomly sprouted a second head, like she couldn't believe that I wasn't seeing whatever was glaringly obvious to her. "You don't see the problem here?" She said, with a patronizing tone. That look still plastered onto her face. My stunned silence was enough of an answer for her as she continued by saying, "The cut wires. Don't you see that the wires are broken!?" At this point, I inspected the wires more closely. I still wasn't really getting it. They looked like every pair of ultra cheap Y-splitter headphones where the individual wire runs to each earpiece and comes together in the middle.

I was over the guessing game at this point, so I asked her what specifically was not working correctly with them. Audio cutting out? Maybe a short in one of the wires? Maybe it's... She interrupts my line of questioning and flatly says, "I don't know". More confused and growing more impatient by the second, I shoot back and say "What do you mean? You called me multiple times in order for me to help you resolve a problem with these things, right?" She again directs my attention to the cables and says, "Look! They're split!" It's then that it dawns on me that she thinks the problem is the Y-split design. I tell her this isn't inherently a problem. In fact, many headphones are manufactured this way. I am explaining this all, when I realize that I can't believe I'm having to have this conversation to begin with. Sarah isn't really old, nor is she really young. If I had to guess, I'd say she was in her early 30s: well within the range for seeing all sorts of split headphones designs. Hell, I'd eat a hat if she hadn't seen a million kids walking around her high school with those first generation earbuds that came with the iPod.

I asked if that was the only "problem" with them that she was aware of. She nodded. She hadn't even tested to see if they were broken before proceeding to call me 6 times in a 45 minute timespan. After her response, my annoyance must have been written all over my face. She chuckled as she walked away saying "Sorry! I'm an iPhone girl!"

35 Comments
2024/07/18
21:02 UTC

409

My password should be good for 10 years.

TL;DR, frustrating situation with a frustrating vendor involving multiple domain credentials and his confusion on what credentials to use where.

I work in IT as an Engineer and the primary point of contact for vendor support. I setup remote access to various Process Control and SCADA networks across multiple domains that don't have domain trust so we rely on DMZ jump hosts and VPN devices and Citrix to facilitate the needs.

Rarely do we give vendors enterprise accounts as our VPN devices preclude that need, but today I worked with a vendor that had an enterprise account. For quick reference Domain 1(d1) is enterprise and Domain 2(d2) is Process control domain.

The initial issue was the the vendors d2 password had expired, so I reset the password and emailed him. About 2 hours later he emailed back saying he couldn't log into the DMZ jump host (to bridge between d1 and d2) and sent a screenshot that didn't make sense. I asked him to call me to work through it; it quickly became apparent that this vendor is confidently incorrect on nearly every aspect.

Immediately he tells me that they had difficulty in the past with his password so we set him up with a special circumstance password that won't need changed for 10 years and that my coworkers know about it. Well, there are only 2 of us and I built the GPOs for D2, so I know that's incorrect - compounded by the fact that his initial issue was his d2 password expired and that domain is only 8 years old and his account is only 2 years old. He was adamant to the point that I shared my screen and showed him he was, in fact, wrong.

I ask him to walk me through what he's doing.

  1. Log into d1 citrix storefront. (correct)

  2. Launch RDP session from citrix (correct).

  3. Type host name of jump host (correct).

3a. Dropdown "show options" menu

  1. Type "d1\username" (incorrect).

  2. Press connect (correct)

  3. Type d1 password (incorrect)

  4. Error.

Me: Okay, so you're trying to use a d1 domain and username and password to log into d2.

Him: immediately cutting me off - yes.

Me: No.

Him: this is how I've always done it.

Me: That's not possible, you're trying to authenticate a d1 username/password through a d2 domain controller, they don't talk. d1\username doesn't mean anything to this machine.

Him: Then why can I access it from d1 citrix?

Me: Because our firewall is configured to allow enterprise traff... look it doesn't matter. Call it magic, but trust the magician.

We went back and forth for a few minutes for me to finally share my screen, follow his process logging into citrix but he stops me.

Him: You're logging into citrix with your account, that's not going to work.

Me: What do you mean? I don't know your D1 credentials so I have to log in as me. This step is irrelevant, trust the process.

I log into citrix and launch RDP, I type the hostname and press "connect", he stops me again.

Him: You need to click the dropdown and type my user name.

Me: Trust the process.

The RDP login shows d1\myusername

Him: See, it doesn't work like this.

Me: *holding back every bit of frustration and ignoring him*

I click "more choices" and "use a different account", then type his d2\username and password and it connects without issue. The moment the desktop pops up, he says "How did you do that I didn't give you my password?"

Me: I reset your password and sent it to you via email this morning... at your request, I still have that email so I just copy/pasted your password.

Him: But you said you don't have my d1 credentials.

Some more back and forth before I finally was able to walk him through logging in on his machine only for him to rejoice with the fact that the connection failed. I check AD and see that he had a failed password and tell him as much. "No, this is my password, it should good for 10 years."

Me: No. The password I sent you in your email is your password, it's good for the next XXX days. We have never, and will never, alter our password rules to give vendors a password that's good for 10 years. This is your password until XXX date at which point it will expire and we'll need to reset it again. Try logging in again and using ONLY The password I sent you this morning.

Him: *Logs in successfully.* I'm going to call *colleague* when he's back from PTO and get this mess sorted out.

Me: I'm sure he'll love that. For now, you're logged in. Can you access what you need?

Him: Yes.

Me: *click*

I really don't understand how or why he thought he had 3 passwords, one of which didn't expire for 10 years. He's not some schmuck entry level helpdesk guy, he's a systems integrator at a company we've worked with for the last 5-6 years. I've worked with some pretty difficult vendors, but I've never had someone so confidently tell me I'm wrong about something I built and work with daily.

49 Comments
2024/07/18
18:20 UTC

342

"Is there a way to recover deleted emails?"

Have you guys noticed that lots of old people treat email like physical mail? Meaning they read them, then delete them right away? I have 200,000 emails in all of my mailboxes combined, going back about 25 years. But this nice old lady (really a great customer) always deletes her emails as soon as she reads them. I always tell her theres absolutely no need to do that, but she does it anyway.

Today I get an email from her:

Is there a way to find old emails (like from four to six months ago) that have been put in trash and then the trash emptied?

Uhhh yeah there is one way, don't delete them. She is using Time Machine to back up her Macs but given the date range, its unlikely we'd find it. Plus the mail comes in, she reads it and deletes it. The emails may not even be around for an hour, so they may not even make it to the backup.

If only they would listen!

For fun, I just looked back, and my oldest email is a forum post reply notification from macfixit forums from November 16th, 2000.

92 Comments
2024/07/18
07:49 UTC

711

Gotta love the "Make it so this never happens again" people who have unreasonable expectations.

So, a little background I run a small IT repair business in a rural area. A local small business called me a few months back to do a couple small jobs, I fixed their issues and seemed like I had gained a new client. Fast forward to now, I got a call about another company drilling a hole through a wall and shorting electrical wires frying the computer and printer. I looked at both devices and the customer decided that it would be better to replace them than fix them.

I order the replacement units and go to install them. The owner doesn't know anything about their system or how it was set up. They also have multiple emails and don't know what email is used for what accounts and doesn't know the passwords to pretty much anything. I'm fumbling through trying to get this setup like it was before but without being able to boot up the old machine and them not knowing literally anything about how the machine was setup I couldn't really get their stored passwords back. The owner and secretary didn't even know if they were signed into the web browsers to be backing that info up in the first place.

He also thought that he was using iCloud to back up everything on the computer "because that’s how his laptop is setup." Well, turns out his laptop didn't have iCloud, it had OneDrive, and the computer that crashed didn't even have that setup. I tried to explain to him that OneDrive wouldn't be backing up the passwords stored in his web browser anyways. The owner starts getting frustrated with the situation and starts taking it out on me, he says he wants this to never happen again and wants to know if I can make that happen. I tried to gently explain to him why it happened in the first place (because him and no one else knows anything about the computer and I didn’t set it up to know how it was in the first place) and that while yes I could do that, it isn't quite just that simple. He cuts me off and says it's a yes or no question can you make it so this never happens again?

I tried to explain to him that it's not really a yes or no answer and the fact that he has so many emails and accounts spread across all of them that it’s a little more than just a yes answer. That I could help him do it, but it was going to entail a lot of fixing things, and that he would still have the responsibility of knowing what accounts he is using where and what the passwords are otherwise he will be back in the same situation again especially if it’s not me doing the job in the future. He gets pissed and starts telling me that he would expect a professional like myself would be able to do these types of things and make it so a person who doesn’t know anything can do this and that he is going to find someone else to do his computer work from now on.

Gotta love it when a business has no backup plan, doesn’t know anything about any of their accounts or how stuff works at all and then expects you to just be a magic worker and it to just be done in some unrealistic way they want it done. I'm thinking I dodged a bullet because this guy would not have remembered anything 10 seconds after I left and when something happens and he is in the same boat again he would have blamed me.

 

123 Comments
2024/07/17
20:53 UTC

306

Email the copier to email…

Scene: at a client site (small promotions agency) while working for an MSP.

Situation: they’ve received their first multifunction device (copier/scanner/printer/fax).

Cast: the only one who matters is the marketing guy. We’ll call him Joe.

Joe is missing a few screws up top, and doesn’t take suggestions well. This is also early 2000s, long before mailchimp, and email marketing is a far cry from what we now know as easy. Joe is frustrated that his computer gets real slow when it is emailing out their newsletter, complete with giant 4+ megabyte images embedded in it. No wonder many of them are returned as undelivered even though the recipient’s mailbox is otherwise fine.

Joe sees the new MFD, and asks us to set something up for his newsletters. He wants to build his newsletter as a printed file, and build a mailing list using Excel. He wants to put the printed newsletter into the scan/fax feeder, then email the spreadsheet TO the MFD device so it can scan the newsletter and email it out for him.

Um, no.

35 Comments
2024/07/14
16:34 UTC

107

Can't connect to server

Background: We're a small MSP (small company of several dozen employees supporting small/medium businesses. Those who's find it more economically beneficial to buy our support services then hiring a dedicated person)

Customer: Opens a ticket "can't connect to server"

I've given up on hoping customers will know how to "correctly" open a ticket, one with an actual description or at the minimum an error message.

HD: calls the customer

Customer: repeats the exact same description

(those type of customers don't know much about computers or how/what we need in order to solve problem)

HD: instruct customer to connect him to his computer (skipping any lengthy conversation or discussion on how to open a ticket).

Customer is having issue connecting to a terminal server (one of the best guesses for this error description although sometimes it can be to network drives for the remaining few customers who're still using it)

The customer is connecting remotely and the error message mentions that his password has expired. Since he connects remotely via a VPN, changing password remotely can create issues with the computer at logon to it remembering the old password on a restart and causing a host of other issues

HD: extends password expiration (updating a field on the AD called: 'pwdlastset'). Problem solved

33 Comments
2024/07/14
07:27 UTC

589

"The VPN is not working..."

Hello again thought I would share one of the last calls I had this week before my vacation.

Like I've said in all my stories here before, our users are what I would describe as above average in computer literacy. So when calls like this come in it's often one of two things and they are all actual real problems.

Some background. About 6 months ago we migrated away from DirectAccess for remote access to a more robust standalone VPN solution. This migration went great and has been working flawlessly, for most. When issues arise it's often a missing VPN sites in the client or something related to the SAML and MF authentication.

User: Hello, I can't connect from my home. The VPN is not working, I NEED to get my job done now!

Me: Okey, when you click connect, are you promted to configure a VPN site or do you get past that?

User: No! I click on connect and the tiny icon spins for 2 minutes and nothing happens!

We have a remote access web portal for our users aswell. This portal is setup to be able to do limited work from any machine through the web browser, like reporting working hours for example. This site makes use of our IDP. If the users can access this site, they can access the SAML portal and the VPN should work.

Me: Okey. Can you reach "domain.se" in your browser?

We have a discussion back and forth and the user is just getting annoyed with me.

User: No! Nothing is working. This has never been an issue before. This new system is bad and broken...

Me: Do you have a network connect? For example can you reach "newswebsite.se"?

User: NO! What does that have to do with anything!? I need to work, I need my documents and programs!

"What does that have to do with anything?" How about everything? I roll my eyes at myself, should've started there... My users have spoiled me. 99,9% of our userbase knows what a VPN is for and that you will need a network connection for it to work... Apparently I found the 0,1%.

Me: You will have to connect to your home network before you connect the VPN to reach your documents.

The user is basically yelling at me at this point.

User: THAT'S WHAT THE VPN IS FOR! TO CONNECT ME TO THE NETWORK! Why else would I bother with this!?

I then had to explain to the user for quite some time that the VPN does not grant her access to the internet and that it requires a network connection function. In the future I will remember that users are users and I will treat them as such. Now I will take my summer break and wind down from this. Perhaps my faith in my users will be restored over the summer?

75 Comments
2024/07/12
10:10 UTC

122

My email is not in the copier address book.

TL;DR: User said here e-mail wasn't in the copier address book. It was there, but they didn't tap the LDAP button for the full address book.

So long story short, we got new copiers at my work this week. Of course we would expect some people to have issues with getting used to the new interface, especially since we also switched copier brands. Despite this, because we utilise a print server and map printers with GPO to the workstations, everyone was able to continue working fairly seamlessly. Some people kept printing and didn't realise there was a new printer until they went to collect their prints.

However, one of the employees mentioned that their e-mail was not in the copier for use with the scan to e-mail function. We had the copiers setup with LDAP so it could search an address book on our server. However, some people press buttons when they are floundering and just add their address to the local address book. Essentially the interaction goes like this; Open address book, only 5 or so addresses show (manually added), tap button that says "To LDAP Server", Global address book is imported from server. Should be fairly simple for anyone to remember right?

So anyway, I get this complaint that a user's e-mail is not in the address book. The interaction went like this;

User: "My e-mail is not in the copier"
Me: "It's in there, you just need to tap a button to bring up the full address list, I'll show you. [I show them the button and proceed to tap the initial group for their name and their name shows up]
User: "Ok"

a while later...

User: "Can you show me that again"
Me: "Sure." [show them the process again]
User: "Ok, but I don't want that, I NEED my address to be ON the printer. You said it's already on there but you didn't put it ON there like I asked."
Me: "There are 2 address books, one is just a local one on the machine that people have added addresses to manually, and the other is our global address list that has all our addresses on it. Just tap the button and your name will be there."
User: "That local one, just add it there, and add [other user's name] while you're there too."
Me: [giving up] "Sure thing."

I added their e-mails to the local address book despite it making absolutely zero difference. Sometimes, when it's a 2 second fix, you just need to make the technically illiterate happy.

31 Comments
2024/07/11
21:32 UTC

378

I've Waited for 1:30 But It's Still Not Working

This just happened earlier today. The user in question is a student doing a placement at our company and couldn't get into our Citrix platform.

User puts in a helpdesk ticket: "Hi, my Citrix isn't letting me log on. I waited for 1.5 hours thinking it might work again but it still hasn't. I am not sure if it will work in a bit or not but I was told to raise a query."

I picked up the ticket and messaged the user on Teams:

Northman: Hi, can I take a look at your Citrix?

User: Yeah, sure.

Remotes onto user's laptop and sees Citrix already open in the background.

Northman: Citrix was already open. What were you having difficulty signing into?

User: So basically, it's open. I went on File Explorer and it didn't work. So then I tried reopening Citrix, and yeah, it's been showing that for the past hour and a half. I didn't want to raise an IT issue straightaway because I thought it would fix itself, but it didn't. [Colleague's Name] told me to raise an IT ticket because I showed him the issue too.

As I watch them try to sign into Citrix to re-open an already open Citrix desktop, I notice the auto-filled email is one from a university.

Northman: The email you are using isn't a [Company Name] one, so that will never work.

User: Oh my gosh, I'm so sorry. I just realized. It's okay, thank you.

Northman: No problem.

The user was quite embarrassed and apologized for wasting my time, but it was only 2 minutes for me. Nearly 2 hours for them by the time they raised the ticket. I probably picked it up around 20 minutes later, aha.

31 Comments
2024/07/11
19:24 UTC

974

Made a thing that got used to lay me off.

Probably a very common story around these parts.

In my old company, data was processed by a team of eight. Each type of data had a different process and a different length of time to complete. Each member of the team also did each of these tasks at different speeds due to age or level of awakeness.

Big boss insisted on daily reports from team leader about how many tasks were completed each day etc. and my team leader spent an hour and a half every single morning pulling data and compiling the report instead of doing any team leading.

I foolishly pipe up and commit the cardinal sin of 'volunteering my help'. After a bit of explanation, i put together a group accessible excel document, where each team member could input the tasks completed each day on one page and the boss could hit the magical button that caused the macros to do their thing. An hour and half every day became 5 minutes. Team leader was very happy.

Now the layoff part. In order to do the workings out i had to use an average length of time for each type of task, which very literally meant getting eight people to spend a day measuring how long it took them to do each thing. I very specifically stated to team leader that because the output was based on averages, it could ONLY be used as an indicator of progress, not as a formal report by any stretch. The fastest person was way, way faster than the others by some quirk of genetics, so badly skewed what a normal amount of work should look like.

Then came the layoffs. Guess whose tool was used to show whom of the eight was working the slowest, because i spent half my time making useful time saving tools instead of 'proper' work?

Sigh.

76 Comments
2024/07/05
12:51 UTC

420

If you don't save your files, they won't be saved.

A few days ago, a user came to the helpdesk with the issue that their most recent changes to some Word documents had been lost after they rebooted their mac.
I started digging, and found that there was no version history - which is unusual. My org is mostly a Microsoft shop, and by default their org-licensed copy of office should create new files in OneDrive. I took a look at the Save settings in Word and found everything unchecked. It looked something like this, to give you an idea of what I mean.
I hoped maybe they'd managed to save their files to their personal iCloud account^1 since they were using a mac - no dice, they had an iCloud subscription, but they'd disabled syncing to iCloud.
And as I was expecting by this point, they did not have the OneDrive desktop client installed. It wasn't just that they had never signed in - it wasn't installed. Which means it must have been manually be uninstalled, since it's part of the Office suite.
So it seemed like a reasonable case of them following the bad practice of only saving work locally on their machine. That stymied any recovery efforts, but why had the work been lost in the first place?
I asked the user to show me how they normally go about saving documents. They brought up a Word document that they were currently editing.
It was a new file, and had never been saved. They had written about 12 pages of text, and it basically only existed in RAM^2. Apparently they'd had this document open since the aforementioned reboot, which was several days prior.
I thought I was used to this sort of thing by now, but I found myself needing a few seconds to process and mentally press ctrl-s/cmd-s a few times in prayer.
I explained as diplomatically as I could, that because they weren't using any of the auto-save or cloud options, they needed to manually save their work.
Thankfully they were amenable to using OneDrive and I got it set up for them, so even if they learned nothing they might be OK for a while.

EDIT: I almost forgot the weirdest detail - it turned out their actual 'saving' process was to copy-paste the entire document and airdrop from their mac to their phone, to send in an email. I'm still not sure how anything was saved on their laptop to begin with.


^1 During the course of troubleshooting I learned that iCloud's auto-saving features only apply to Pages, not Word - but moot point since they'd turned it off in Settings anyway.
^2 I know that's not quite how it works due to the local autosave and filelocks and whatnot, but for practical intents and purposes... no doubt they would ignore an 'unsaved work' prompt when closing Word too.

69 Comments
2024/07/05
00:49 UTC

1,680

I find it ridiculous you don't carry spare laptop power supplies everywhere

I get a call direct on my work cell phone well before my regular start time. Although it's before my start time, I usually answer the call anyway in case it's because of a site outage, major incident, something like that.

Caller: "Hi speddie23 (low level manager) here. (Other higher level employee) gave me your number. I'm at (small, remote site) and I have forgotten my laptop power supply at home and the battery in my laptop is completely flat. Can you let me know if we have any spare power supplies here"

Me: "No, I don't think we do. They all use desktops there"

Caller: "Ok, well what can I do? I would have to drive all the way home to get my power supply"

Me: "Perhaps there is a spare computer you can use there? Or maybe you can ask around in case someone does have a laptop with a power supply you can borrow"

Caller: "No, I've already asked around and no one has one. Everything I need is on my laptop, I really need this working"

Me: "You might have to drive back home to get your power supply then"

Caller: "I find it ridiculous that you don't have spares. Can you call my manager and let them know that I will be offline for a few hours whilst I" (and they make this next part sound very exacerbated) "go home and get my power supply so I can work"

Me: "No, but seeing as you are on the phone to me now, I'm sure you can call them as you have access to a phone"

172 Comments
2024/07/05
01:23 UTC

635

I will Never Learn that Nothing is So Permanent as a Temporary Solution, But My Boss Has

1) About 15 years ago I got a stack of greenbar paper delivered to my desk every morning. It typically ranged from about 60 pages to 600. I'd have to review each page and look for certain information, which may or may not have been present, and then add it to a list. Then I'd bucket the list by client, type it all up nice and neat, and then send the results over to each client. It took anywhere from an hour to half the day.

I finally managed to convince our management to send me an electronic copy securely via email. That I could parse and almost completely format via excel. The result I could typically bang out in about 15-30 minutes at most. The method was relatively crude, but it got the job done. And it was easy for me to modify if the format of the underlying data changes. I expected someone to come up with something more robust and "official" sometime in the next 18 months or so. But that little sheet was what we used for years, even some years after I left the team and had forgotten all about it. Someone on the team found my name in the sheet and called me asking for help.

2) About 5 years ago I got asked to make a report that would identify gaps in our on-call schedule. Same as before, I built a quick and dirty mechanism using some reports and Excel that would pick out the gaps. And same as before, it was crude, but easy to to understand and easy to modify as needed. And again I figured someone would come along after the fact and build something more robust and official. But that didn't happen. I used that process for years, right up until I left. In fact, I just chatted via Linkedin with the person who took over part of my duties. They are still using that thing.

3) I move to a new company. At my new company we had a need to reconcile some expense accounts against some operational activities. Effectively, we needed a way to verify if we were spending as much as we should have for certain things. We had purchased a solution that was supposed to answer that question, but given some idiosyncrasies in our industry, and some peculiarities in our company, it was accurate maybe half the time. So effectively it was worthless.

And again, I whip something up, and again it is crude, but again it gets the job done. Unfortunately this time my solution requires a fair amount of manual work, but the result is accurate. It takes me about 30 minutes a day, provided the schema of the incoming data doesn't change. This time around I was explicit that this process was a "proof-of-concept." It was far too manual to be considered a permanent solution. We have an entire reporting and analytics department that should be able to whip up something far more robust. Right? In the meantime I'd use what I created to demonstrate what was possible.

But once again it's now almost 2 years after since I first presented this thing. And we're not only still using it, it has been formally deemed as "meeting the need," "a good use of resources," etc etc. In fact, the reporting and analytics department just declared that" implementing a more automated solution is not a good use of development hours." So I guess I'm going to be making this for the foreseeable future. My manager asked me in my last review if I had created documentation for the process, so we can train someone else to take over when I take two weeks of PTO this summer.

4) One of the inputs for the process I described above uses a vendor that we have concluded is wildly overpriced. And once again I made a proof-of-concept replacement that leverages common tools in our company and expertise we already have. We'd save about $150K a year in exchange for someone spending about 2 hours a week on this thing. I was able to mimic the process of our vendor almost precisely. So the result would look quite similar. I could even pretty up the reports if folks have interest.

This time my manager announces its existence and that it's available for use. The department that works with the relevant vendor can use this thing instead. I'd say the chances of adoption are about 50/50. My bosses were excited to potentially save that much money, but enthusiasm waned once they realized my team wouldn't be doing the actual work.

5) Finally, my boss approached me last month and gave me strict marching orders to run any of these little projects past her first before announcing them to the world. She's fine with me making these things, but she wants to make sure my team doesn't get stuck with managing them. She says that we want to be the "Create solutions team," not the "Do-the-work-other-people-ignore team."

Hope everyone who has a "blow off your fingers day" has a good time blowing off their fingers. I just finished mowing the lawn and cutting hedges. Now it's time to vacuum and mop.

96 Comments
2024/07/04
23:24 UTC

936

Was wondering why it's so hot in here

The company I work for has two sites with inhouse datacenters. One site has the datacenter in the back of the IT office with a big glass wall with the server racks visible, lets a bit of noise through. But when rebooting something, you can hear that (fans start spinning 100%).

I was working from the other site, saw a temperature sensor going up in our monitoring tool. Sometimes it just spikes a bit, nothing to worry about. Decided to ignore it for a bit.

Checked the sensor again, it went from 26c to 45c! That can't be good?

I asked in our IT Teams channel who was present at the site, got a response. Called the colleague immediately, I heard the servers whining on the background. Asked him to check the air-conditioning units, indeed 45c! Got a response back, I was wondering why it was so hot in the office. Clearly did not notice the servers whining?!

Called my IT Manager to ask who was the facilities employee on the specific site. Called that person, and got the response yeah don't worry, there is air-conditioning maintenance on the roof, but they are on lunch break right now. They left the roof units off while on lunch break :(

113 Comments
2024/07/03
11:32 UTC

728

User does not realise their monitor need power to work

User calls me for help with her second monitor she hasnt used in a while. She says she checked all the cables and it still does not work.

  • I come over, first thing - check the cables
  • Power cable is not there, only thing connected is HDMI
  • Tell her that the power cable is not connected - "well you see youre missing a cable back here.."
  • Her coworkes responds "See! I told you there should be another cable there!"
  • Coworker2 then says "Oh well I thought the one cable (hdmi) that goes into the little black box (computer) is enough"
  • At this point im just confused how the second lady made it so far in life but alright shes probably not a tech person ..
  • Looked under the table for the cable, found it, plugged it in, everything works
  • "Where did you get that cable? we were looking there and it wasnt there"
  • "No it was right here hanging over the other cables"
  • leave

I feel like I just went through some test of patience.

114 Comments
2024/06/26
13:11 UTC

1,402

User reports that web browser closes when they close the web browser

A user just called me and told me that this website they use for their work keeps closing every couple seconds, and it happens every time they open a pdf file. I remotely connected to their computer to see what was going on. This is what happened:

  • [User]: Opens web browser and goes to the website
  • [User]: Opens pdf file in same browser window
  • Nothing strange happens
  • [User]: Clicks the X at the top right to close the browser
  • [User]: "See, the website keeps closing!"
  • [Me]: "That's because you closed it."
  • [User]: "No, it happens every time I open a pdf!"
  • [Me]: Reopens the website and then opens a pdf file to show [User] that the website she had open does not close when she opens a pdf
  • [Me]: Explains to [User] that the browser was closing because she was closing it by clicking the Close button
166 Comments
2024/06/25
15:38 UTC

297

DNS strikes back

While I'm not tech-support but a systems-engineer, I think it still fits.

This story happened around 3 weeks ago.

I saw an alert for one of our customer domains in the uptime monitoring.
At the same moment, I got message in the support-chat, about that domain not working and colleagues not being able to connect via SSH.

Mind you, this domain is used by customers to consume the content, they create with our software, so it not working is kind of a big deal.

Since that webspace is managed for us by a webhoster, I only hat limit access to it but I tried to debug non the less.

  1. Trying to login via SSH

Server ignored my pubkey and asked me for a password -> weird
Server has different Host-Key than our administration domain -> very weird, possibly an issue on the hosting side

  1. Pinging domain

IP of server looks unfamiliar -> that's when that small voice, in the back of my head, the one you hear when you are about to stumble into a situation that is way worse than it seems, started whispering

  1. Checking the domain DNS

nslookup.io returns the same weird IP -> oh god
Same for the entire zone -> OH FUCK!!!

  1. Whois of the IP and domain

Whois for the domain and IP return a Hosting-Provider in Florida,USA -> not even our fucking continent

At this point, I called my team lead out of his meeting to resolv this Grade-A shitfest.

After digging through multiple stages of DNS providers and hosters, we reached the actual registrar where the domain got bought, more than a decade ago.

Their crew, however, was unwilling/unqualified/unable/un-whatever to help us or even understand that we lost control over the entire dns-zone.

After my TL spend some time and explaining to them, what the issue was, at all, they finally told us, they have no idea, why we lost control of the domain.

Later, my TL set an ultimatum and requesting a statement about the incident. The whole thing got fixed 2 days later.

Now, we received a statement by the registrar, stating that the original registrar, who owns the TLD, apparently shipped a backend update, resulting in a bunch of these kinds of errors.

22 Comments
2024/06/25
12:19 UTC

425

My genius coworkers are at it again

I work for a small MSP with a couple of guys I've known for many years.

one guy is in terms of the organisation my superior, but technically he is a blue arsed fly of a human that is impossible to pin down, made of teflon so nothing sticks, and sometimes a complete idiot...

What he's very good at is concealing his idiocy, riding on the technical coat tails of others and making it seem like he's very up to date. I seem to spend my life clearing up after him.

I have a mantra - we do not assume anything. not for that old joke about it making an ass out of u and me. no because "assume" is a fancy word for I'm guessing, haven't done the research and wanted to use a word that makes me sound a bit more intelligent.

My life working with him is like one of those Tom and Jerry cartoons where spike the angry dog has warned them he will kill them if he wakes up, and then goes sleep walking through various hazardous places like building sites or army ranges, while both Tom and Jerry suffer hideous injuries trying to stop falling anvils, piranhas and electric shocks. Spike wakes up refreshed and we cut to T&J in plaster casts, with black eyes, missing fur and the occasional zap of elecricity sparking from their whiskers.

Todays fun - Datacenter firewall swap out.

Moving from a Meraki firewall to a Unifi UDM SE (i fought hard against this, but all the decision makers saw were prices and contract costs, and ignored the great tech support and how many hours it will save us).

His plan,

He configured the firewall in our office, then i get to take it to the datacenter, Plug WAN2 on the firewall into the LAN on the existing network to being it online so he can configure it the rest...

Only thing is, he was asking me to plug the firewall into a the network it was replacing, which means IP's in the same range on the WAN and the LAN. The little unifi didnt like this.

"but i configured it in our office and everything worked" - yes , our office that's on a completely different subnet....

Why didn't I configure this all myself? because it got him 3 hours of time in the office that he could bill for, I would have had that thing done in 30 minutes....

so we lost half an hour, I couldn't get into the firewall as I had not yet been invited to the console yet, but I got him onto my laptop and got him in locally. i watched and stifled my laughter as he tried to put the public ip in as the subnet mask details, then i put him out of his misery.

then he got horrifically confused. all the servers were not showing online. The firewall was now on the internet, he could see it, and could get it to ping the servers, but they couldn't get online...

If he had actually done his research, he would have seen that the old firewall was not on 192 168 16 1, but on 192. 168 16 252, 30 seconds of work to make that check

I'm writing this from the refectory of the datacentre after checking everything is now OK. I could have left hours ago, but i am having to pick through his work to look for other gotchas (we already have found some missing port forwarding rules)

FML

36 Comments
2024/06/21
09:50 UTC

276

About classing floppy disk

A have a couple of stories that could goes here but a fortuitous encounter with an old schoolmate today remind me of this one. It isn't one of mine but it is the story our software engineering teacher always told to illustrate that, if users can screw something, they will screw it.

For a bit of context, it was the era of the 5.25" floppy disk and my teacher was doing tech support for a PC installer.

One day, my teacher got a call from a compagny where he had made an install a few weeks prior. A panicked secretary explained him that her boss asked her to print somes files but she can't read the floppy disk with them. He tried to solve the issue on the phone but, ultimatly, concluded that her floppy drive was dead and needed a replacement.

My teacher took a new drive and went his way to the client. Once there, he proceeded to check if the floppy drive was really dead by putting in a test floppy disk he had took with him and... It worked. He then observed the secretary operating the floppy drive and, once again, it worked just fine with his test floppy disk. It was as this moment the secretary said "Oh but I have this problem only with those from *this one specific coworker*."

Given this clue, my teacher went see this coworker with the bad floppy disks and ask her to see them. The coworker went to a cabinet and took a binder. The coworker was asked to class the floppy disks so she punched them and put them in the binder.

PS: Sorry for my bad english, I'm not a native speaker.

60 Comments
2024/06/20
19:08 UTC

368

Alzheimer’s VS the Rolling 2FA

I have a funny story from years ago that I still think of every now and then.

My old job was L1 help desk at a mid sized MSP. Many of our clients had a few “retired” partners who still had their own VDI, full access, and worked remotely. I think they mostly responded to emails and just kept a finger on the pulse, but that’s beside the point. These people were always super old and often technically illiterate, making them some of the most difficult customers to support.

We had one guy in particular who was notorious for holding our techs hostage for 30+ minutes, always for something incredibly mundane, made borderline impossible by his tech illiteracy and very apparent signs of dementia. The guy was super nice, and evidently very important at this client (at least, at one point in time). He sometimes had a “helper” present while calling the HD, which made his calls tolerable, but there was a stretch of a few weeks where he was on his own, called almost every day, and it got so bad that he became banned from calling.

It was ALWAYS the same issue. He’d call in, trying to access his VDI but “locked out”. He had a sticky note on his monitors with his 2FA code and passwords, but his memory had declined to the point where he’d frequently forget this, and forget how 2FA even worked. It got so bad towards the end that he would forget why he’d even called or what the tech just said to him. Here’s an example.

C (Customer): I can’t login to my computer.

T (Tech): what seems to be the problem? Your account does not appear to be locked. Are you connected to the VPN?

C: I don’t know

T: Alright, can you click on the lock icon and let me know what it says?

C: it shows the login screen. It won’t let me login.

T: I see, it looks like your 2FA was locked. I just unlocked you. Can you try again?

C: still failed. I don’t remember my password.

T: sir, you need to enter your PIN first. Do you remember your PIN? It should be on a sticky note on your monitor. (This was in all caps on his ticket profile).

C: ok I see it.

T: Ok, now enter that, then open the 2FA app on your phone and enter the code on the screen.

C: what’s the 2FA app?

T: explains, painstakingly, how to find the app

C: takes impossibly long to type in the passcode, so the code rolls over, invalidating his PIN authentication. login denied

T: ok, let’s try again, enter your PIN

C: what’s my PIN?

….He’d need 2FA explained to him over and over, and could never enter the passcode quickly enough for it to still be active by the time he authenticated. We could sometimes get him in eventually, but often not. Sometimes when we got him logged in, he’d admit that he could no longer remember WHY he was logging in in the first place.

I know this sounds far fetched, but I took calls from this guy myself at least a half a dozen times, and listened to even more recordings. It became so frequent, and impossible without his helper, that we had to speak to our contacts at this company and essentially have this customer blacklisted from calling us. I believe he was set up with his own liaison at the company, but I’m not sure. I don’t know what he was even doing at this point for the business but it couldn’t have been much. The poor guy was supposed to be retired, memory failing him, but he was so accustomed to working that he didn’t know what else to do with himself.

56 Comments
2024/06/19
23:59 UTC

Back To Top